CentOS 6.2 x64
Percona MySQL 5.6.15
I've run into this issue a few times, but I've never been able to replicate it at will. Here's what appears to cause the issue:
1. My Zabbix proxies will suddenly start getting a massive backlog of data. I really do mean massive. I have 22 proxies and I've seen this issue occur when there are a total of 8+ million values backlogged between them all. This can be a result of various causes (lots of data sent in at once, server was offline for a while, etc.).
2. When #1 occurs, the zabbix server begins accepting all the data (as expected).
3. I can see all of my history syncers inserting data as fast as they can into the database (as expected).
4. After a while (variable time), the database load will suddenly disappear. At any given time I might see one insert query for the DBSyncers, by that's it.
5. After the load has disappeared, I can see that items that are polled/pushed to/from the zabbix server itself are updated.
6. At the same time as #5, I can see that barely any data is being retrieved from the Zabbix proxies at all. The backlog of data keeps growing larger and larger.
7. After waiting for a while without the problem going away, I'll give up and restart the zabbix_server process.
8. After restarting the zabbix_server process, the server immediately begins to process data like crazy from the proxies (like it should have been doing BEFORE the restart).
When this problem is occuring, the load on my database server is almost non-existant. It literally isn't doing anything. The same goes for the Zabbix server. There is nothing to indicate that the server is overly busy. The last time this happened (about 30 minutes ago), my monitoring on the various zabbix processes/caches didn't show anything odd. The history cache usage had grown quite a bit before #4 happened, but that was expected because so much data was coming in from my proxies. At no point did the free space for the history cache reach 0% (at least according to my graphs/data).
I have attached a copy of my zabbix_server.conf and a screenshot of the cache, data gathering processes, and internal processes usage. Notice that my history syncers were maxed out, but as I mentioned earlier, they were not doing anything on the DB itself. The DB server was idle.