Uploaded image for project: 'ZABBIX BUGS AND ISSUES'
  1. ZABBIX BUGS AND ISSUES
  2. ZBX-13286

Zabbix housekeeper problem

XMLWordPrintable

    • Icon: Incident report Incident report
    • Resolution: Commercial support required
    • Icon: Minor Minor
    • None
    • None
    • Server (S)
    • None
    • Zabbix 3.2.6

      Hello,

      We have issues with the zabbix housekeeper process.
      We are receiving alarms like:

      Trigger: Zabbix housekeeper processes more than 75% busy Trigger status: OK Trigger severity: Average Trigger URL: 
      
      Item values:
      
      1. Zabbix busy housekeeper processes, in % (Zabbix server:zabbix[process,housekeeper,avg,busy]): 0 % 2. *UNKNOWN* (*UNKNOWN*:*UNKNOWN*): *UNKNOWN* 3. *UNKNOWN* (*UNKNOWN*:*UNKNOWN*): *UNKNOWN*
      
      Original event ID: 613938
      

      Also, I have noticed slow queries in the zabbix_server.log.
      Example:

      21244:20171214:065752.959 executing housekeeper
       21244:20171214:065759.352 slow query: 6.323679 sec, "delete from history where itemid=23252 and clock<1505455072"
       21244:20171214:071222.820 slow query: 7.942164 sec, "delete from history_uint where itemid=23287 and clock<1505455072"
       21244:20171214:071230.196 slow query: 6.683755 sec, "delete from history_uint where itemid=23291 and clock<1505455072"
       21244:20171214:071235.907 slow query: 5.711109 sec, "delete from history_uint where itemid=23292 and clock<1505455072"
       21244:20171214:071240.972 slow query: 4.469797 sec, "delete from history_uint where itemid=23294 and clock<1505455072"
       21244:20171214:071244.160 slow query: 3.187227 sec, "delete from history_uint where itemid=23298 and clock<1505455072"
       21244:20171214:071253.229 slow query: 3.036727 sec, "delete from history_uint where itemid=23316 and clock<1505455072"
       21244:20171214:072023.377 slow query: 30.825332 sec, "delete from history_uint where ctid = any(array(select ctid from history_uint where itemid=36109 limit 5000))"
       21244:20171214:072103.028 slow query: 39.650996 sec, "delete from history_uint where ctid = any(array(select ctid from history_uint where itemid=36841 limit 5000))"
       21244:20171214:072133.741 slow query: 30.712282 sec, "delete from history_uint where ctid = any(array(select ctid from history_uint where itemid=36842 limit 5000))"
       21244:20171214:072208.737 slow query: 34.996242 sec, "delete from history_uint where ctid = any(array(select ctid from history_uint where itemid=36843 limit 5000))"
       21244:20171214:072248.499 slow query: 39.761517 sec, "delete from history_uint where ctid = any(array(select ctid from history_uint where itemid=36254 limit 5000))"
       21244:20171214:072320.169 slow query: 31.669664 sec, "delete from history_uint where ctid = any(array(select ctid from history_uint where itemid=36255 limit 5000))"
       21244:20171214:072348.205 slow query: 28.036313 sec, "delete from history_uint where ctid = any(array(select ctid from history_uint where itemid=36256 limit 5000))"
       21244:20171214:072419.781 slow query: 31.576354 sec, "delete from history_uint where ctid = any(array(select ctid from history_uint where itemid=36347 limit 5000))"
       21244:20171214:072446.360 slow query: 26.578785 sec, "delete from history_uint where ctid = any(array(select ctid from history_uint where itemid=36348 limit 5000))"
       21244:20171214:072501.634 slow query: 15.273431 sec, "delete from history_uint where ctid = any(array(select ctid from history_uint where itemid=36349 limit 5000))"
       21244:20171214:072517.577 slow query: 15.943202 sec, "delete from history_uint where ctid = any(array(select ctid from history_uint where itemid=36417 limit 5000))"
       21244:20171214:072531.230 slow query: 13.652308 sec, "delete from history_uint where ctid = any(array(select ctid from history_uint where itemid=36418 limit 5000))"
       21244:20171214:072542.757 slow query: 11.526944 sec, "delete from history_uint where ctid = any(array(select ctid from history_uint where itemid=36419 limit 5000))"
       21244:20171214:072610.986 slow query: 28.229420 sec, "delete from history_uint where ctid = any(array(select ctid from history_uint where itemid=36485 limit 5000))"
       21244:20171214:072633.733 slow query: 22.747093 sec, "delete from history_uint where ctid = any(array(select ctid from history_uint where itemid=36486 limit 5000))"
       21244:20171214:072658.452 slow query: 24.718797 sec, "delete from history_uint where ctid = any(array(select ctid from history_uint where itemid=36487 limit 5000))"
       21244:20171214:072721.191 slow query: 22.738987 sec, "delete from history_uint where ctid = any(array(select ctid from history_uint where itemid=36107 limit 5000))"
       21244:20171214:072740.080 slow query: 18.888824 sec, "delete from history_uint where ctid = any(array(select ctid from history_uint where itemid=36108 limit 5000))"
       21244:20171214:072740.315 housekeeper [deleted 636679 hist/trends, 90000 items, 0 events, 0 problems, 0 sessions, 0 alarms, 0 audit items in 1787.348534 sec, idle for 1 hour(s)]
      

      Values for housekeeper from zabbix_server.conf file are:

      ### Option: HousekeepingFrequency
      #       How often Zabbix will perform housekeeping procedure (in hours).
      #       Housekeeping is removing outdated information from the database.
      #       To prevent Housekeeper from being overloaded, no more than 4 times HousekeepingFrequency
      #       hours of outdated information are deleted in one housekeeping cycle, for each item.
      #       To lower load on server startup housekeeping is postponed for 30 minutes after server start.
      #       With HousekeepingFrequency=0 the housekeeper can be only executed using the runtime control option.
      #       In this case the period of outdated information deleted in one housekeeping cycle is 4 times the
      #       period since the last housekeeping cycle, but not less than 4 hours and not greater than 4 days.
      #
      # Mandatory: no
      # Range: 0-24
      # Default:
      # HousekeepingFrequency=1
      
      ### Option: MaxHousekeeperDelete
      #       The table "housekeeper" contains "tasks" for housekeeping procedure in the format:
      #       [housekeeperid], [tablename], [field], [value].
      #       No more than 'MaxHousekeeperDelete' rows (corresponding to [tablename], [field], [value])
      #       will be deleted per one task in one housekeeping cycle.
      #       SQLite3 does not use this parameter, deletes all corresponding rows without a limit.
      #       If set to 0 then no limit is used at all. In this case you must know what you are doing!
      #
      # Mandatory: no
      # Range: 0-1000000
      # Default:
      # MaxHousekeeperDelete=5000
      

      In the attachment you can find housekeeping configuration from web gui.

      What can I do to resolve this issue?

      Best Regards,
      Ivan

            Unassigned Unassigned
            igasparec Ivan Gašparec
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated:
              Resolved: