-
Incident report
-
Resolution: Unsupported version
-
Minor
-
None
-
1.8.17, 2.0.8
-
Zabbix Server and Frontend 2.0.8, on CentOS 6.4, with MySQL 5.5 on separate machine. About 500 items per second.
I made several debugs in the Zabbix code for detect the reason for Zabbix Server history cache decrease very fast and increase slowly, several times, making the zabbix queue increase too and number of VPS reduce...
My diagnostic showed that when the same itemid arrive in less of 5s, the "while" loop is skipped into funcion DCsync_history() before finishing "syncs". In other words, the syncs variable, that determines how many times the "DBSyncer Process" will sincronize the items, not reached 0 (according the "while" condition) and, the end of funcion, debug is showing that history syncer process recorded 0 items, according the zabbix_server.log attached.
When the dbsyncer enters within the "do while" loop, each iteration is compared "skipped_clock < max_delay". The skipped_clock starts 0 in the begin and change based on 2 conditions: "SUCCEED == uint64_array_exists(cache->itemids, cache->itemids_num,cache->history[f].itemid)" or "1 < num && 0 == skipped_clock".
I don't understand the reason of this particular "if's", but it seems correct (I think that it is because another dbsyncer is syncing this specific registry). Curiosity: what's the meaning of set the skipped_clock variable?
The great problem is the while condition. When dbsyncer enter on the funcion and the condition "skipped_clock < max_delay" is true. despite having more than 10000 items to sync, the process skip and sync nothing... sleeping for "CONFIG_HISTSYNCER_FREQUENCY". On a medium environment, this five seconds are essential to keep the history synchronized.
The changes made in the source code for debugging is in debug_zabbix.patch.
On my real environment, the problem was caused by several log monitorings createad without filter. I was only able to find the problem debugging the source code and seeing several returns with the specific itemid...
I tried simulate the real enviroment on HML using zabbix_sender every one second... attached the log file of Zabbix on HML enviroment...Pay attention to the history_num and number of syncronized items on the same minute.