Uploaded image for project: 'ZABBIX BUGS AND ISSUES'
  1. ZABBIX BUGS AND ISSUES
  2. ZBX-23690

Utilization of history syncer processes 100% when creating and updating many maintenances

XMLWordPrintable

    • S24-W10/11, S24-W12/13, S24-W14/15, S24-W16/17, S24-W18/19
    • 2

      Steps to reproduce:
      Create/update maintenances 700+ times during 30 minutes.
      An example of such a maintenance:

      The hosts group is the parent group for all other groups. Contains 28000+ hosts.
      The tag stands for the hostname. We cannot use maintenance for regular Zabbix hosts, as we have events not directly related to Zabbix hosts. They alert from "virtual" hosts. Events from such hosts can be associated with different virtual machines, e.g.
      How it works:

      An external system handles product deployment. When it's time to deploy a new version of a product to a virtual machine, it creates maintenance via the Zabbix API. When the product is updated and running, this system sends maintenance updates by changing the "Active till" and "Maintenance period length" to "now+5 min" for example.

      If need to update several hundreds of VMs one by one in a short period of time (30-60 minutes), this leads to a history syncer problem. Zabbix then cannot even process new data from the proxies and is effectively down during and for some time after the service ends.
      At this time, there is no heavy load on the operating system, database or database disks.
      Only slow queries in Zabbix server logs:
      2771378:20231109:063027.178 slow query: 3.015999 sec, "lock table maintenances in share mode"
      2771381:20231109:063709.894 slow query: 32.688270 sec, "lock table maintenances in share mode"

      Of course, we try to do multi-host maintenance when possible, so as not to spam Zabbix with a huge number of separate maintenances, but unfortunately, this is not always possible.

      StartDBSyncers=64
      Result:
      !
      !
      Expected:
      Creating and updating maintenance should not greatly affect the history syncer and other processes.
      ~23 maintenance changes per minute doesn't seem like that many.

            wiper Andris Zeila
            yuriip Yurii Polenok
            Team A
            Votes:
            2 Vote for this issue
            Watchers:
            12 Start watching this issue

              Created:
              Updated: