-
Type:
Incident report
-
Resolution: Unresolved
-
Priority:
Trivial
-
None
-
Affects Version/s: None
-
Component/s: None
-
None
-
Environment:Zabbix 7.0.11, Ubuntu 24.04.2.
Number of hosts (enabled/disabled) 3607 3559 / 48
Number of templates 422
Number of items (enabled/disabled/not supported) 1159798 1046652 / 21116 / 92030
Number of triggers (enabled/disabled [problem/ok]) 470093 453380 / 16713 [768 / 452612]
Number of users (online) 217 18
Required server performance, new values per second 7570.36
10 proxies
DB size: 600-650GB (TimescaleDB)
Zabbix 7.0.11, Ubuntu 24.04.2. Number of hosts (enabled/disabled) 3607 3559 / 48 Number of templates 422 Number of items (enabled/disabled/not supported) 1159798 1046652 / 21116 / 92030 Number of triggers (enabled/disabled [problem/ok]) 470093 453380 / 16713 [768 / 452612] Number of users (online) 217 18 Required server performance, new values per second 7570.36 10 proxies DB size: 600-650GB (TimescaleDB)
Steps to reproduce:
- Set HANodeName to <hostname> and NodeAddress to <host_fqdn> on a single zabbix server host.
- Restart zabbix
- Increase HistoryCacheSize and HistoryIndexCacheSize
- restart zabbix
Result:
History cache fills up and when it's full, queues grow rapidly until zabbix is shut down again. A few values are recieved, but the queues show no signs of being processed as one would expect.
When HistoryCacheSize is increased, zabbix functions normally, (ie queues are low, values are processed) until history cache fills up again and queues subsequently start to grow.
After disabling HA, the history cache is swiftly emptied and queues qucikly processed and zabbix returns to a stable state.
Attached 4 sets of graphs showing caches and queue for the 4 times zabbix was restarted with different settings.
First restart
This enabled HA by setting:
HANodeName=myzabbixhost
NodeAddress=myzabbixhost.mydomain.com
Cache settings:
HistoryCacheSize=128M
HistoryIndexCacheSize=64M
Timestamps:
20250610:121911.986 Got signal [signal:15(SIGTERM),sender_pid:4035515,sender_uid:0,reason:0]. Exiting ...
20250610:121926.055 Zabbix Server stopped.
20250610:121926.113 Starting Zabbix Server.
**
Second restart (HA enabled)
Cache settings:
HistoryCacheSize=512M
HistoryIndexCacheSize=128M
Timestamps:
20250610:134911.167 Got signal [signal:15(SIGTERM),sender_pid:298268,sender_uid:0,reason:0]. Exiting ...
20250610:140538.395 Starting Zabbix Server.
Third restart (HA enabled)
Cache settings:
HistoryCacheSize=1G
HistoryIndexCacheSize=512M
20250610:143652.439 Got signal [signal:15(SIGTERM),sender_pid:490534,sender_uid:0,reason:0]. Exiting ...
20250610:144432.535 Zabbix Server stopped.
20250610:144432.574 Starting Zabbix Server.
**
Fourth restart - HA disabled
Cache settings:
HistoryCacheSize=2G
HistoryIndexCacheSize=512M
Timestamps:
20250610:153343.619 Got signal [signal:15(SIGTERM),sender_pid:745894,sender_uid:0,reason:0]. Exiting ...
20250610:160815.365 Zabbix Server stopped.
20250610:160815.404 Starting Zabbix Server.
Expected:
That enabling HA on a single node would not cause these issues and History cache to be emptied quickly after restart and value processing to continue as when started without HA enabled.
