Uploaded image for project: 'ZABBIX BUGS AND ISSUES'
  1. ZABBIX BUGS AND ISSUES
  2. ZBX-22363

Slow configuration sync after upgrade from 5 to 6

XMLWordPrintable

    • Sprint 98 (Mar 2023), Sprint 99 (Apr 2023)
    • 2

      Hi, we have managed to reproduce the problem in a new Zabbix and a new Oracle installation. Here are step-by-step instructions so that you can reproduce this problem in your environment.

      Setup test environment

      Zabbix-Oracle-DB:

      2 CPU, 4GB RAM,50G disk, Oracle Version 19.3.0.0.0, OS: RHEL 8.2

      Zabbix-Server:

      2 CPU, 4GB RAM,50G disk, Zabbix server 5.0.14, OS: RHEL 8.2

      NOTE: You must have two servers on the same subnet because the problem is more noticeable when DB is on a remote server.

      Populate Zabbix with dummy data

      Duplicate out-of-box host "Zabbix server" 17 000 times using API to create lots of configuration data (triggers and items).

      NOTE: Make sure to stop the Zabbix agent on the server so that Zabbix doesn't crash when trying to poll that many hosts.

      Run tests on Zabbix 5.0.14

      Run the SQL query that the "Configuration Syncer" process is using and measure run time.

      NOTE: We have these run times: 1. test: 00:00:23.49, 2. test: 00:00:26.35

      Upgrade Zabbix

      Upgrade Zabbix to the 6.0.9 version

      Run tests on Zabbix 6.0.9

      Run the SQL query that the "Configuration Syncer" process is using and measure run time.

      NOTE: We have these run times: 1. test: 00:05:16.57, 2. test: 00:05:24.73

      Conclusion

      • After the upgrade, the runtime of the Conf Syncer process increased from 24 seconds to 320 seconds, which is a 13.13-fold increase!
      • At our other site, the number of hosts exceeds 20,000, and there are over 3 million metrics, and the results are the same! The runtime of the Conf Syncer process went from 3.5 minutes to 45 minutes which is a 12.8-fold increase.
      • The new upcoming feature "Incremental Configuration Syncer Update" from Zabbix 6.4 and 7.0 will not completely fix this problem. This is because every time the Zabbix server is restarted, it must wait for the Configuration Syncer update to finish, which in our customer's case, takes 45 minutes of Zabbix downtime.
      • Here are our observations:
        a) New NCLOB fields may cause this problem, is it possible to revert these field back to NVARCHAR2?
        b) There has been some changes in SQL queries used by Config. Syncer, could those be the reason?

        1. screenshot-1.png
          screenshot-1.png
          2 kB
        2. 0001-.-ZBX-22363-fixed-dbschema-and-dbpatch-for-testing.patch
          6 kB
        3. items_preproc_text_to_short-1.sql
          1.0 kB
        4. zabbix_proxy.log
          57 kB
        5. Archive.7z
          1.65 MB
        6. select_optimized1-1.sql.7z
          26 kB
        7. ZBX-22363-item_rtdata-5.diff
          10 kB

            vso Vladislavs Sokurenko
            agavrilovs Aleksandrs Petrovs-Gavrilovs
            Team A
            Votes:
            12 Vote for this issue
            Watchers:
            29 Start watching this issue

              Created:
              Updated:
              Resolved: