Type: Incident report
Affects Version/s: 4.0.7
Fix Version/s: None
Component/s: Server (S)
Environment:Zabbix - 4.0.7
Hosts - 13035
Items - 3431108
Triggers - 387768
NVPS - 7610
I have noticed the history syncers get busy and delay syncing when lots of events arrive and last time this occurred for us was 2700 in 10 minutes due to a large environment change being performed (including internal events ie item not supported).
There is many zabbix server log entries with this entry in there - slow query: 11.164569 sec, "update ids set nextid=nextid+1 where table_name='event_suppress' and field_name='event_suppressid'".`
The behavior is that the history syncers are slow and the cache starts to build up until the above log entries disappear (ie processing caught up) OR lots of hosts come out on their maintenance period (and we have a lot of hosts in maintenance after hours).
The issue is that if the history syncing gets far enough behind other nodata() triggers start to fire causing additional load and compounds the issue.
This never occured in Zabbix 3.4 and we have only just updated tgo Zabbix 4.0 in the last couple of weeks and this has happened several times.