Zabbix Team -
I've modified my zabbix_server.conf file to run the CacheUpdateFrequency every 20 seconds. After doing so, I noticed heavior IOWait and pretty much every process became busier according to the Zabbix server health dashboard and internal zabbix items metrics. I straced a few pids (Pollers and DBSyncers specifically) and a few seconds into the configuration sync I notice a FUTEX_WAIT as the last system call.
Is this acceptable? I have a very large system and had to change the CacheUpdateFrequency back to 60. On average, it's taking about 11-14 seconds to complete the configuration sync, which is great and I think we all wanted this in version 6.2+. But, it seems that there is a lock placed on the configuration cache for a portion of the configuration syncing. I had 15 Pollers running at about 60% utilization, and after adjusting the CacheUpdateFrequency I was able to remain at 67% utlization but with only 8 pollers instead of 15. Every zabbix-server process appears to be locked during the configuration sync.
I've seen a couple of tickets floating around in the last few months discussing lock behavior in the DBSyncers, and I'm wondering if there are any further improvements that can be made here, or if we simply have to deal with the cache locking for the incremental configuration sync benefit. Also, do you have recommendations for system scaling with the incremental configuration sync? I realize every system is different, but some general guidelines might be useful in the manual.
Let me know if I can provide any other diagnostic information.