[ZBX-13286] Zabbix housekeeper problem Created: 2018 Jan 03  Updated: 2018 Jan 03  Resolved: 2018 Jan 03

Status: Closed
Project: ZABBIX BUGS AND ISSUES
Component/s: Server (S)
Affects Version/s: None
Fix Version/s: None

Type: Incident report Priority: Minor
Reporter: Ivan Gašparec Assignee: Unassigned
Resolution: Commercial support required Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

Zabbix 3.2.6


Attachments: JPEG File housekeeping.JPG    

 Description   

Hello,

We have issues with the zabbix housekeeper process.
We are receiving alarms like:

Trigger: Zabbix housekeeper processes more than 75% busy Trigger status: OK Trigger severity: Average Trigger URL: 

Item values:

1. Zabbix busy housekeeper processes, in % (Zabbix server:zabbix[process,housekeeper,avg,busy]): 0 % 2. *UNKNOWN* (*UNKNOWN*:*UNKNOWN*): *UNKNOWN* 3. *UNKNOWN* (*UNKNOWN*:*UNKNOWN*): *UNKNOWN*

Original event ID: 613938

Also, I have noticed slow queries in the zabbix_server.log.
Example:

21244:20171214:065752.959 executing housekeeper
 21244:20171214:065759.352 slow query: 6.323679 sec, "delete from history where itemid=23252 and clock<1505455072"
 21244:20171214:071222.820 slow query: 7.942164 sec, "delete from history_uint where itemid=23287 and clock<1505455072"
 21244:20171214:071230.196 slow query: 6.683755 sec, "delete from history_uint where itemid=23291 and clock<1505455072"
 21244:20171214:071235.907 slow query: 5.711109 sec, "delete from history_uint where itemid=23292 and clock<1505455072"
 21244:20171214:071240.972 slow query: 4.469797 sec, "delete from history_uint where itemid=23294 and clock<1505455072"
 21244:20171214:071244.160 slow query: 3.187227 sec, "delete from history_uint where itemid=23298 and clock<1505455072"
 21244:20171214:071253.229 slow query: 3.036727 sec, "delete from history_uint where itemid=23316 and clock<1505455072"
 21244:20171214:072023.377 slow query: 30.825332 sec, "delete from history_uint where ctid = any(array(select ctid from history_uint where itemid=36109 limit 5000))"
 21244:20171214:072103.028 slow query: 39.650996 sec, "delete from history_uint where ctid = any(array(select ctid from history_uint where itemid=36841 limit 5000))"
 21244:20171214:072133.741 slow query: 30.712282 sec, "delete from history_uint where ctid = any(array(select ctid from history_uint where itemid=36842 limit 5000))"
 21244:20171214:072208.737 slow query: 34.996242 sec, "delete from history_uint where ctid = any(array(select ctid from history_uint where itemid=36843 limit 5000))"
 21244:20171214:072248.499 slow query: 39.761517 sec, "delete from history_uint where ctid = any(array(select ctid from history_uint where itemid=36254 limit 5000))"
 21244:20171214:072320.169 slow query: 31.669664 sec, "delete from history_uint where ctid = any(array(select ctid from history_uint where itemid=36255 limit 5000))"
 21244:20171214:072348.205 slow query: 28.036313 sec, "delete from history_uint where ctid = any(array(select ctid from history_uint where itemid=36256 limit 5000))"
 21244:20171214:072419.781 slow query: 31.576354 sec, "delete from history_uint where ctid = any(array(select ctid from history_uint where itemid=36347 limit 5000))"
 21244:20171214:072446.360 slow query: 26.578785 sec, "delete from history_uint where ctid = any(array(select ctid from history_uint where itemid=36348 limit 5000))"
 21244:20171214:072501.634 slow query: 15.273431 sec, "delete from history_uint where ctid = any(array(select ctid from history_uint where itemid=36349 limit 5000))"
 21244:20171214:072517.577 slow query: 15.943202 sec, "delete from history_uint where ctid = any(array(select ctid from history_uint where itemid=36417 limit 5000))"
 21244:20171214:072531.230 slow query: 13.652308 sec, "delete from history_uint where ctid = any(array(select ctid from history_uint where itemid=36418 limit 5000))"
 21244:20171214:072542.757 slow query: 11.526944 sec, "delete from history_uint where ctid = any(array(select ctid from history_uint where itemid=36419 limit 5000))"
 21244:20171214:072610.986 slow query: 28.229420 sec, "delete from history_uint where ctid = any(array(select ctid from history_uint where itemid=36485 limit 5000))"
 21244:20171214:072633.733 slow query: 22.747093 sec, "delete from history_uint where ctid = any(array(select ctid from history_uint where itemid=36486 limit 5000))"
 21244:20171214:072658.452 slow query: 24.718797 sec, "delete from history_uint where ctid = any(array(select ctid from history_uint where itemid=36487 limit 5000))"
 21244:20171214:072721.191 slow query: 22.738987 sec, "delete from history_uint where ctid = any(array(select ctid from history_uint where itemid=36107 limit 5000))"
 21244:20171214:072740.080 slow query: 18.888824 sec, "delete from history_uint where ctid = any(array(select ctid from history_uint where itemid=36108 limit 5000))"
 21244:20171214:072740.315 housekeeper [deleted 636679 hist/trends, 90000 items, 0 events, 0 problems, 0 sessions, 0 alarms, 0 audit items in 1787.348534 sec, idle for 1 hour(s)]

Values for housekeeper from zabbix_server.conf file are:

### Option: HousekeepingFrequency
#       How often Zabbix will perform housekeeping procedure (in hours).
#       Housekeeping is removing outdated information from the database.
#       To prevent Housekeeper from being overloaded, no more than 4 times HousekeepingFrequency
#       hours of outdated information are deleted in one housekeeping cycle, for each item.
#       To lower load on server startup housekeeping is postponed for 30 minutes after server start.
#       With HousekeepingFrequency=0 the housekeeper can be only executed using the runtime control option.
#       In this case the period of outdated information deleted in one housekeeping cycle is 4 times the
#       period since the last housekeeping cycle, but not less than 4 hours and not greater than 4 days.
#
# Mandatory: no
# Range: 0-24
# Default:
# HousekeepingFrequency=1

### Option: MaxHousekeeperDelete
#       The table "housekeeper" contains "tasks" for housekeeping procedure in the format:
#       [housekeeperid], [tablename], [field], [value].
#       No more than 'MaxHousekeeperDelete' rows (corresponding to [tablename], [field], [value])
#       will be deleted per one task in one housekeeping cycle.
#       SQLite3 does not use this parameter, deletes all corresponding rows without a limit.
#       If set to 0 then no limit is used at all. In this case you must know what you are doing!
#
# Mandatory: no
# Range: 0-1000000
# Default:
# MaxHousekeeperDelete=5000

In the attachment you can find housekeeping configuration from web gui.

What can I do to resolve this issue?

Best Regards,
Ivan



 Comments   
Comment by Olegs Vasiljevs (Inactive) [ 2018 Jan 03 ]

Hello Ivan!

Seems like you have deleted an array of items with existing historical data. Housekeeper will normalize once it deletes old historical values. You can expand database resources for delete queries to operate faster, or expand the MaxHousekeeperDelete setting to force housekeeper to delete more than 5000 to fasten the process, or force housekeeper execution from command line interface to be more regular than once per an hour (currently it runs for 30 minutes, therefore with good timing you can execute it twice per an hour), or delete such historical values manually from the database if you know what itemid's have been removed.

Nevertheless, this is a bug tracker, not a support channel. Support options are mentioned on the following page: http://zabbix.org/wiki/Getting_help

Closing the ticket.

Regards,
Oleg

Generated at Thu May 01 07:37:00 EEST 2025 using Jira 9.12.4#9120004-sha1:625303b708afdb767e17cb2838290c41888e9ff0.