-
Type:
Problem report
-
Resolution: Unresolved
-
Priority:
Trivial
-
None
-
Affects Version/s: 7.0.28rc1, 7.4.12rc1
-
Component/s: Server (S)
-
None
-
Environment:7.0.27 + pgsql 18
-
Support backlog
hello,
i've noticed two issues with current HA implementation in Zabbix,
case 1 - DB node is overloaded and doesn't really work:
When this problem happens, DB host is not responding due to being overloaded , but DB engine is still running.
When this happens, both nodes from cluster will try to restart itself, due to issues with not responding HA manager, example:
# node 1 8231:20260707:101533.892 "oom1" node switched to "standby" mode 8812:20260707:101533.893 executing housekeeper 8811:20260707:101533.902 cannot read LLD manager service request 8810:20260707:101533.903 cannot read LLD manager service request 8970:20260707:101534.068 starting HA manager 8970:20260707:101534.068 HA manager started in standby mode 8231:20260707:101642.143 HA manager is not responding in standby mode, restarting it. 8987:20260707:101642.145 starting HA manager 8987:20260707:101642.146 HA manager started in standby mode 8231:20260707:101750.219 HA manager is not responding in standby mode, restarting it. 8989:20260707:101750.221 starting HA manager 8989:20260707:101750.222 HA manager started in standby mode
# node 2 219787:20260707:101535.595 HA manager is not responding in standby mode, restarting it. 220596:20260707:101535.599 starting HA manager 220596:20260707:101535.599 HA manager started in standby mode 219787:20260707:101643.674 HA manager is not responding in standby mode, restarting it. 220612:20260707:101643.677 starting HA manager 220612:20260707:101643.678 HA manager started in standby mode 219787:20260707:101751.749 HA manager is not responding in standby mode, restarting it. 220658:20260707:101751.771 starting HA manager 220658:20260707:101751.777 HA manager started in standby mode
case 2 - split brain when DB engine crashed
in this case, it's possible to achieve split brain scenario, if for example DB engine did crash due to OOM, and it's restarting, example:
220658:20260707:101751.777 [Z3005] query failed: [0] PGRES_FATAL_ERROR:server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
[select ha_failover_delay,auditlog_enabled,auditlog_mode from config]
220658:20260707:101751.777 [Z3005] query failed: [0] result is NULL [rollback;]
in such a case, since node cannot check which system is running as active, two nodes can start to run in active mode.
both situations can be reproduced - for first one, run this .py script:
import mmap, time size = 7000 * 1024 * 1024 print(f"Allocating {size // 1024 // 1024} MB...") m = mmap.mmap(-1, size) for i in range(0, size, 4096): m[i:i+1] = b'\x00' if i % (100 * 1024 * 1024) == 0: print(f" Written {i // 1024 // 1024} MB", flush=True) print("locking mem") time.sleep(10000)
then, on both of nodes of cluster, tail log file with -f flag (tail -f /var/log/zabbix/zabbix_server.log)
for case 2, it's needed to crash PostgreSQL with OOM - easiest way would be to edit systemd unit file for postgresql,
place this in unit file and reload daemon + restart service:
# systemctl edit postgresql-XX.service [Service] MemoryMax=1024M MemorySwapMax=0
then, run "heavy" sql