• OS: Amazon Linux 2023(Fedora base) / 6.1.77-99.164.amzn2023.x86_64
• DB: Aurora 8.0.mysql_aurora.3.05.2(MySQL 8.0.32 compatible)
• Package version of Zabbix server:
• zabbix-agent2 6.0.31
• zabbix-agent2-plugin-mongodb 6.0.31
• zabbix-agent2-plugin-postgresql 6.0.31
• zabbix-apache-conf 6.0.31
• zabbix-sender 6.0.31
• zabbix-server-mysql 6.0.31
• zabbix-sql-scripts 6.0.31
• zabbix-web 6.0.31
• zabbix-web-deps 6.0.31
• zabbix-web-japanese 6.0.31
• zabbix-web-mysql 6.0.31
• Package version of monitored host:
• zabbix-agent2 6.0.27

Please provide zabbix_server -R diaginfo when issue occurs

Is it difficult to determine the cause without running "zabbix_server -R diaginfo" while the problem is occurring?
We are rolling back from "6.0.31" to "6.0.25" due to the impact on monitoring in production environment and cannot get "zabbix_server -R diaginfo" during the issue.

It's currently unknown why this has happened, in the past it was possible that some items spammed Zabbix server and they could be shown in diaginfo in that case.

Get "zabbix_server -R diaginfo" in development environment Zabbix, upgraded from "6.0.25" to "6.0.31".

Please find attached the diaginfo obtained with Zabbix (6.0.31) in development environment.
I have not observed any "history write cache" exhaustion in development environment of Zabbix.

It will be required to have "zabbix_server -R diaginfo" at the peak level:

 
By running "zabbix_server -R diaginfo," we can see which itemid holds many values in memory at the peak level. This can reveal the item type.

I am looking for a way to determine the cause of the problem from information other than diaginfo.

You mentioned that there were items spamming Zabbix server in the past, if possible please let us know what items were the cause.

Is it difficult to investigate the cause from Zabbix server logs or DB?

We have prepared a verification environment and were able to reproduce the issue. Could you please investigate the cause based on the attached diaginfo from the peak time?

This part might explain it, as it is seen item with itemid 126643 has 1025153 values

Values:1386357 done:1202367 queued:199 processing:15 pending:183776 time:1.026531
Top.values:
  itemid:126643 values:1025153 steps:0
  itemid:56472 values:11404 steps:2
  itemid:56494 values:11404 steps:2
  itemid:56488 values:11400 steps:2
  itemid:56483 values:11400 steps:2
  itemid:56470 values:11398 steps:2
  itemid:132621 values:11395 steps:2
  itemid:132622 values:11392 steps:2
  itemid:56491 values:11391 steps:2
  itemid:56473 values:11390 steps:2
  itemid:132619 values:11388 steps:2
  itemid:56471 values:11388 steps:2
  itemid:56476 values:11388 steps:2
  itemid:132620 values:11387 steps:2
  itemid:56501 values:11387 steps:2
  itemid:56489 values:11384 steps:2
  itemid:56490 values:11381 steps:2
  itemid:126772 values:378 steps:0
  itemid:126677 values:377 steps:0
  itemid:41239 values:377 steps:0
  itemid:69793 values:377 steps:0
  itemid:88884 values:377 steps:0
  itemid:92699 values:377 steps:0
  itemid:91374 values:377 steps:0
  itemid:68404 values:377 steps:0

Thank you for your comment. 
Itemid 126643 is a log monitoring item. As a result of testing under the following conditions, only version 6.0.31 showed a tendency for the history write cache to increase.

Test Conditions

Created a log monitoring item on Linux, writing 30 lines per second to the monitored log file (for 2 hours).

Results

• 6.0.31: The history write cache (% used) increased to about 15%.
• 6.0.25: The history write cache (% used) remained around 0.2%.

Additional Information

This phenomenon was confirmed in Linux log monitoring, but no increase in the history write cache (% used) was observed in Windows log monitoring with version 6.0.31.

Consideration

Based on these results, we suspect a performance degradation specific to version 6.0.31. Could you please share your thoughts on this matter?

We implemented the following two measures:

• Removed log monitoring items with large data volumes
• Upgraded to version 6.0.34 where no performance degradation is observed

We will close this case.