Frankly, I was not able to come up with a good issue summary. Feel free to change it to a more appropriate one
We noticed today Zabbix lagging behind. After a quick investigation we found out that there was always one history syncer busy what slewed down the synchronization as a whole.
A quick investigation on "Triggers top 100" revealed many thousands of status changes for each of round about 15 triggers.
This may happen easily with log or trapper items. In this case however, all were triggers based on web scenarios items. The triggers itself made only use of the count() function and did not create multiple events. By the way, update interval of the web scenarios in question was configured to be once a minute.
So we looked on the history tables of the Zabbix server database. There we found out that for items in questions, occasionally over thousand values for the same second exist. Where it would have been expected to be only one value for the same second and that only every minute.
We haven't further investigated this, but we assume this was not limited to Web scenarios as we found at least one item of different type with similar symptoms (User parameter).
The only thing all seem to have in common was to be monitored by the same (passive) Zabbix proxy.
Zabbix proxy and Zabbix server were on the same version.