-
Incident report
-
Resolution: Commercial support required
-
Minor
-
None
-
None
-
None
-
Zabbix 3.2.6
Hello,
We have issues with the zabbix housekeeper process.
We are receiving alarms like:
Trigger: Zabbix housekeeper processes more than 75% busy Trigger status: OK Trigger severity: Average Trigger URL: Item values: 1. Zabbix busy housekeeper processes, in % (Zabbix server:zabbix[process,housekeeper,avg,busy]): 0 % 2. *UNKNOWN* (*UNKNOWN*:*UNKNOWN*): *UNKNOWN* 3. *UNKNOWN* (*UNKNOWN*:*UNKNOWN*): *UNKNOWN* Original event ID: 613938
Also, I have noticed slow queries in the zabbix_server.log.
Example:
21244:20171214:065752.959 executing housekeeper 21244:20171214:065759.352 slow query: 6.323679 sec, "delete from history where itemid=23252 and clock<1505455072" 21244:20171214:071222.820 slow query: 7.942164 sec, "delete from history_uint where itemid=23287 and clock<1505455072" 21244:20171214:071230.196 slow query: 6.683755 sec, "delete from history_uint where itemid=23291 and clock<1505455072" 21244:20171214:071235.907 slow query: 5.711109 sec, "delete from history_uint where itemid=23292 and clock<1505455072" 21244:20171214:071240.972 slow query: 4.469797 sec, "delete from history_uint where itemid=23294 and clock<1505455072" 21244:20171214:071244.160 slow query: 3.187227 sec, "delete from history_uint where itemid=23298 and clock<1505455072" 21244:20171214:071253.229 slow query: 3.036727 sec, "delete from history_uint where itemid=23316 and clock<1505455072" 21244:20171214:072023.377 slow query: 30.825332 sec, "delete from history_uint where ctid = any(array(select ctid from history_uint where itemid=36109 limit 5000))" 21244:20171214:072103.028 slow query: 39.650996 sec, "delete from history_uint where ctid = any(array(select ctid from history_uint where itemid=36841 limit 5000))" 21244:20171214:072133.741 slow query: 30.712282 sec, "delete from history_uint where ctid = any(array(select ctid from history_uint where itemid=36842 limit 5000))" 21244:20171214:072208.737 slow query: 34.996242 sec, "delete from history_uint where ctid = any(array(select ctid from history_uint where itemid=36843 limit 5000))" 21244:20171214:072248.499 slow query: 39.761517 sec, "delete from history_uint where ctid = any(array(select ctid from history_uint where itemid=36254 limit 5000))" 21244:20171214:072320.169 slow query: 31.669664 sec, "delete from history_uint where ctid = any(array(select ctid from history_uint where itemid=36255 limit 5000))" 21244:20171214:072348.205 slow query: 28.036313 sec, "delete from history_uint where ctid = any(array(select ctid from history_uint where itemid=36256 limit 5000))" 21244:20171214:072419.781 slow query: 31.576354 sec, "delete from history_uint where ctid = any(array(select ctid from history_uint where itemid=36347 limit 5000))" 21244:20171214:072446.360 slow query: 26.578785 sec, "delete from history_uint where ctid = any(array(select ctid from history_uint where itemid=36348 limit 5000))" 21244:20171214:072501.634 slow query: 15.273431 sec, "delete from history_uint where ctid = any(array(select ctid from history_uint where itemid=36349 limit 5000))" 21244:20171214:072517.577 slow query: 15.943202 sec, "delete from history_uint where ctid = any(array(select ctid from history_uint where itemid=36417 limit 5000))" 21244:20171214:072531.230 slow query: 13.652308 sec, "delete from history_uint where ctid = any(array(select ctid from history_uint where itemid=36418 limit 5000))" 21244:20171214:072542.757 slow query: 11.526944 sec, "delete from history_uint where ctid = any(array(select ctid from history_uint where itemid=36419 limit 5000))" 21244:20171214:072610.986 slow query: 28.229420 sec, "delete from history_uint where ctid = any(array(select ctid from history_uint where itemid=36485 limit 5000))" 21244:20171214:072633.733 slow query: 22.747093 sec, "delete from history_uint where ctid = any(array(select ctid from history_uint where itemid=36486 limit 5000))" 21244:20171214:072658.452 slow query: 24.718797 sec, "delete from history_uint where ctid = any(array(select ctid from history_uint where itemid=36487 limit 5000))" 21244:20171214:072721.191 slow query: 22.738987 sec, "delete from history_uint where ctid = any(array(select ctid from history_uint where itemid=36107 limit 5000))" 21244:20171214:072740.080 slow query: 18.888824 sec, "delete from history_uint where ctid = any(array(select ctid from history_uint where itemid=36108 limit 5000))" 21244:20171214:072740.315 housekeeper [deleted 636679 hist/trends, 90000 items, 0 events, 0 problems, 0 sessions, 0 alarms, 0 audit items in 1787.348534 sec, idle for 1 hour(s)]
Values for housekeeper from zabbix_server.conf file are:
### Option: HousekeepingFrequency # How often Zabbix will perform housekeeping procedure (in hours). # Housekeeping is removing outdated information from the database. # To prevent Housekeeper from being overloaded, no more than 4 times HousekeepingFrequency # hours of outdated information are deleted in one housekeeping cycle, for each item. # To lower load on server startup housekeeping is postponed for 30 minutes after server start. # With HousekeepingFrequency=0 the housekeeper can be only executed using the runtime control option. # In this case the period of outdated information deleted in one housekeeping cycle is 4 times the # period since the last housekeeping cycle, but not less than 4 hours and not greater than 4 days. # # Mandatory: no # Range: 0-24 # Default: # HousekeepingFrequency=1 ### Option: MaxHousekeeperDelete # The table "housekeeper" contains "tasks" for housekeeping procedure in the format: # [housekeeperid], [tablename], [field], [value]. # No more than 'MaxHousekeeperDelete' rows (corresponding to [tablename], [field], [value]) # will be deleted per one task in one housekeeping cycle. # SQLite3 does not use this parameter, deletes all corresponding rows without a limit. # If set to 0 then no limit is used at all. In this case you must know what you are doing! # # Mandatory: no # Range: 0-1000000 # Default: # MaxHousekeeperDelete=5000
In the attachment you can find housekeeping configuration from web gui.
What can I do to resolve this issue?
Best Regards,
Ivan