[ZBX-22363] Slow configuration sync after upgrade from 5 to 6 Created: 2023 Feb 15  Updated: 2024 Apr 10  Resolved: 2023 May 16

Status: Closed
Project: ZABBIX BUGS AND ISSUES
Component/s: API (A), Frontend (F), Proxy (P), Server (S)
Affects Version/s: 6.0.13
Fix Version/s: 6.0.17rc1, 6.4.2rc1, 7.0.0alpha1, 7.0 (plan)

Type: Problem report Priority: Blocker
Reporter: Aleksandrs Petrovs-Gavrilovs Assignee: Vladislavs Sokurenko
Resolution: Fixed Votes: 12
Labels: Oracle
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: Text File 0001-.-ZBX-22363-fixed-dbschema-and-dbpatch-for-testing.patch     File Archive.7z     File ZBX-22363-item_rtdata-5.diff     File items_preproc_text_to_short-1.sql     PNG File screenshot-1.png     File select_optimized1-1.sql.7z     Text File zabbix_proxy.log    
Issue Links:
Causes
Duplicate
Sub-task
depends on ZBX-22799 Slow retrieval of configuration for Z... Closed
part of ZBX-21808 Slow configuration sync after upgrade... Closed
Team: Team A
Sprint: Sprint 98 (Mar 2023), Sprint 99 (Apr 2023)
Story Points: 2

 Description   

Hi, we have managed to reproduce the problem in a new Zabbix and a new Oracle installation. Here are step-by-step instructions so that you can reproduce this problem in your environment.

Setup test environment

Zabbix-Oracle-DB:

2 CPU, 4GB RAM,50G disk, Oracle Version 19.3.0.0.0, OS: RHEL 8.2

Zabbix-Server:

2 CPU, 4GB RAM,50G disk, Zabbix server 5.0.14, OS: RHEL 8.2

NOTE: You must have two servers on the same subnet because the problem is more noticeable when DB is on a remote server.

Populate Zabbix with dummy data

Duplicate out-of-box host "Zabbix server" 17 000 times using API to create lots of configuration data (triggers and items).

NOTE: Make sure to stop the Zabbix agent on the server so that Zabbix doesn't crash when trying to poll that many hosts.

Run tests on Zabbix 5.0.14

Run the SQL query that the "Configuration Syncer" process is using and measure run time.

NOTE: We have these run times: 1. test: 00:00:23.49, 2. test: 00:00:26.35

Upgrade Zabbix

Upgrade Zabbix to the 6.0.9 version

Run tests on Zabbix 6.0.9

Run the SQL query that the "Configuration Syncer" process is using and measure run time.

NOTE: We have these run times: 1. test: 00:05:16.57, 2. test: 00:05:24.73

Conclusion

  • After the upgrade, the runtime of the Conf Syncer process increased from 24 seconds to 320 seconds, which is a 13.13-fold increase!
  • At our other site, the number of hosts exceeds 20,000, and there are over 3 million metrics, and the results are the same! The runtime of the Conf Syncer process went from 3.5 minutes to 45 minutes which is a 12.8-fold increase.
  • The new upcoming feature "Incremental Configuration Syncer Update" from Zabbix 6.4 and 7.0 will not completely fix this problem. This is because every time the Zabbix server is restarted, it must wait for the Configuration Syncer update to finish, which in our customer's case, takes 45 minutes of Zabbix downtime.
  • Here are our observations:
    a) New NCLOB fields may cause this problem, is it possible to revert these field back to NVARCHAR2?
    b) There has been some changes in SQL queries used by Config. Syncer, could those be the reason?


 Comments   
Comment by Vladislavs Sokurenko [ 2023 Mar 01 ]

a) New NCLOB fields may cause this problem, is it possible to revert these field back to NVARCHAR2?

Yes it is possible if data that is used fits in all fields. Was it confirmed that changing to NCLOB alone in 5.0 introduces this problem or do you think this is because of other query changes ?

nclob was added in ZBXNEXT-6196, don't know if it causes problems though we might want to run some internal tests.

Comment by Vladislavs Sokurenko [ 2023 Mar 02 ]

If it's not too much to ask, if test instance exists and it is possible to test performance then please try building 6.0 with a patch 0001-.-ZBX-22363-fixed-dbschema-and-dbpatch-for-testing.patchand then upgrading to 6.0 using patched binary. It will skip database upgrade patches that change varchar to NCLOB.

Comment by Vladislavs Sokurenko [ 2023 Mar 13 ]

Root cause for a problem is that in ZBXNEXT-6196 item.params, item.description, item.posts, item.headers and item_preproc.params fields were changed from nvarchar2(2048) to nclob, here is example how to change them back [^items_preproc_text_to_short.sql] and observe that it eliminates the issue. Realistically it is not always required to have those fields as nclob as it can be uncommon to have such large data in those fields. There is no official solution yet and only Oracle database is affected, there are no such performance problems with MySQL, MariaDB or PostgreSQL.

Comment by Vladislavs Sokurenko [ 2023 Mar 15 ]

Wonder if something like this can do the trick:

select i.itemid, i.hostid, i.status, i.type, i.value_type, i.key_, i.snmp_oid, i.ipmi_sensor, i.delay, i.trapper_hosts, i.logtimefmt,  CASE WHEN i.type in (13,14,11,15) THEN i.params ELSE NULL end, ir.state, i.authtype, i.username, i.password, i.publickey, i.privatekey, i.flags, i.interfaceid, ir.lastlogsize, ir.mtime, i.history, i.trends, i.inventory_link, i.valuemapid, i.units, ir.error, i.jmx_endpoint, i.master_itemid, i.timeout, i.url, i.query_fields, CASE WHEN i.type in (19) THEN i.posts ELSE NULL end, i.status_codes, i.follow_redirects, i.post_type, i.http_proxy, CASE WHEN i.type in (19) THEN i.headers ELSE NULL end, i.retrieve_mode, i.request_method, i.output_format, i.ssl_cert_file, i.ssl_key_file, i.ssl_key_password, i.verify_peer, i.verify_host, i.allow_traps, i.templateid, null from items i inner join hosts h on i.hostid=h.hostid join item_rtdata ir on i.itemid=ir.itemid where h.status in (0,1) and i.flags<>2; 
Comment by Vladislavs Sokurenko [ 2023 Mar 22 ]

Fixed in pull request feature/ZBX-22363-6.0-3

Comment by Karol Woronowicz [ 2023 Mar 29 ]

The same situation exists on a large environment on postgres14.

Comment by Alex Kalimulin [ 2023 Mar 29 ]

karol97, this problem has been identified as Oracle-specific. If you experience slow configuration sync on Postgres, perhaps, it is another problem that deserves a separate ticket.

Comment by Vladislavs Sokurenko [ 2023 Apr 04 ]

Fixed in:

Comment by LivreAcesso.Pro [ 2023 Apr 05 ]

Same issue on PostgreSQL:
ZBXNEXT-8358 @karol97

Comment by Marina Generalova [ 2023 Apr 06 ]

Documentation updated:

Generated at Tue Jun 17 07:47:01 EEST 2025 using Jira 9.12.4#9120004-sha1:625303b708afdb767e17cb2838290c41888e9ff0.