Summary

The Zabbix server (HA cluster, active node) crashed with SIGSEGV in a trapper process while it was building configuration for an active proxy (zbx_send_proxyconfig → zbx_proxyconfig_get_data → zbx_dc_get_proxy_config_updates). The main process then exited ("One child process died") and the HA cluster failed over to the standby node.

The fault is a general protection fault (trapno 13) caused by dereferencing a non-canonical pointer (rsi = 0x8000000000000020). It is not a NULL dereference. The faulting loop iterates over a vector embedded in a configuration-cache object. That vector's element count is nonsensical (0x32a54c ≈ 3.3 M) and its data pointer is invalid. This suggests the object was stale or freed (a use-after-free or race), or the configuration cache memory was corrupted.

This happened once in ~3 months of available logs. No core dump is available.

Environment

• Zabbix server: 7.0.20 (revision e9302d4d6fc), official package zabbix-server-pgsql-7.0.20-release1.el9.x86_64
• OS: RHEL 9.5 compatible, kernel 5.15.0-304.171.4.3.el9uek.x86_64; 8 vCPU, 16 GB RAM
• Libraries: openssl-libs 3.5.1, libevent 2.1.12, pcre2 10.40
• DB: PostgreSQL (Patroni cluster behind HAProxy + PgBouncer), schema in a non-public schema (DBSchema set)
• Native HA: 2 nodes (node1 active, node2 standby), failover delay 60 s
• Size (from System information): 3537 hosts (3327 enabled), 494 templates, 910,394 items (611,223 enabled / 293,272 disabled / 5,899 not supported), 363,705 triggers, required performance 7,887 NVPS
• 20 active proxies (no passive proxies, no encryption). 15 of them are in proxy groups (5 groups of 1-4 proxies each) and 5 are not in any group. One more proxy group was created ~2.5 h before the crash, see below.
• Proxy versions are mixed, and some proxies are newer than the server:

Relevant server configuration (non-default values):

CacheSize=8G
HistoryCacheSize=1G
HistoryIndexCacheSize=400M
TrendCacheSize=1G
TrendFunctionCacheSize=1G
ValueCacheSize=4G
StartPollers=10
StartPollersUnreachable=4
StartDBSyncers=20
StartLLDProcessors=4
Timeout=4
LogSlowQueries=3000
EnableGlobalScripts=0
HANodeName=<node1>
NodeAddress=<node1>:10051
# StartTrappers default (5)

Timeline (local time, CET)

The request that triggered the crash was not logged, so we cannot identify which proxy sent it.

Crash log

Got signal [signal:11(SIGSEGV),reason:128,refaddr:(nil)]. Crashing ...
====== Fatal information: ======
Program counter: 0x5559f1d3c2db
=== Registers: ===
r8      =                0
r9      =     7ffff20f8af0
r10     =                4
r11     =     7fce4536ac80
r12     =     7fcbcbae5c20
r13     =     7ffff20f8af0
r14     =                0
r15     =     7fcbab347408
rdi     =     7ffff20f8af0
rsi     = 8000000000000020
rbp     =     7ffff20f8ac0
rbx     =               3a
rdx     =                0
rax     =                1
rcx     =           32a54c
rsp     =     7ffff20f8890
rip     =     5559f1d3c2db
efl     =            10246
csgsfs  =   2b000000000033
err     =                0
trapno  =                d
oldmask =                0
cr2     =                0
=== Backtrace: ===
17: /usr/sbin/zabbix_server: trapper #3 [processing data](zbx_backtrace+0x41) [0x5559f1e9f601]
16: /usr/sbin/zabbix_server: trapper #3 [processing data](zbx_log_fatal_info+0x26d) [0x5559f1e9f99d]
15: /usr/sbin/zabbix_server: trapper #3 [processing data](+0x421ba6) [0x5559f1e9fba6]
14: /lib64/libc.so.6(+0x3e730) [0x7fce451af730]
13: /usr/sbin/zabbix_server: trapper #3 [processing data](zbx_dc_get_proxy_config_updates+0x20b) [0x5559f1d3c2db]
12: /usr/sbin/zabbix_server: trapper #3 [processing data](zbx_proxyconfig_get_data+0x108d) [0x5559f1bf245d]
11: /usr/sbin/zabbix_server: trapper #3 [processing data](zbx_send_proxyconfig+0x32f) [0x5559f1bf5ebf]
10: /usr/sbin/zabbix_server: trapper #3 [processing data](zbx_trapper_process_request_server+0x164) [0x5559f1ba5dc4]
9: /usr/sbin/zabbix_server: trapper #3 [processing data](+0x471f77) [0x5559f1eeff77]
8: /usr/sbin/zabbix_server: trapper #3 [processing data](zbx_trapper_thread+0x41a) [0x5559f1b9630a]
7: /usr/sbin/zabbix_server: trapper #3 [processing data](zbx_thread_start+0x20) [0x5559f1dbf650]
6: /usr/sbin/zabbix_server: trapper #3 [processing data](+0x46aa0a) [0x5559f1ee8a0a]
5: /usr/sbin/zabbix_server: trapper #3 [processing data](MAIN_ZABBIX_ENTRY+0xc12) [0x5559f1b55852]
4: /usr/sbin/zabbix_server: trapper #3 [processing data](zbx_daemon_start+0x145) [0x5559f1ea2235]
3: /usr/sbin/zabbix_server: trapper #3 [processing data](main+0x3f5) [0x5559f1b49c55]
2: /lib64/libc.so.6(+0x295d0) [0x7fce4519a5d0]
1: /lib64/libc.so.6(__libc_start_main+0x80) [0x7fce4519a680]
0: /usr/sbin/zabbix_server: trapper #3 [processing data](_start+0x25) [0x5559f1b50ff5]

r12 (0x7fcbcbae5c20) and r15 point into the SysV shared memory segments (configuration cache). The full memory map is attached (ZBX_incident_2026-11-12_memory_map.txt).

Disassembly at the faulting address

From the shipped binary /usr/sbin/zabbix_server_pgsql (symbol zbx_dc_get_proxy_config_updates at 0x2be0d0; fault at +0x20b = 0x2be2db):

2be2b0:  mov    (%r12),%rsi              ; first field of object (id)
2be2b4:  mov    0x18(%rsp),%rdi
2be2b9:  callq  zbx_vector_uint64_append ; append object id
2be2be:  mov    0xa0(%r12),%ecx          ; vector->values_num  -> ecx = 0x32a54c (!)
2be2c6:  test   %ecx,%ecx
2be2c8:  jle    ...
2be2ca:  xor    %r14d,%r14d              ; i = 0
2be2d0:  mov    0x98(%r12),%rsi          ; vector->values      -> 0x8000000000000020 (!)
2be2d8:  mov    %r13,%rdi
2be2db:  mov    (%rsi,%r14,8),%rsi       ; <== FAULT: values[0], non-canonical address -> #GP
2be2df:  add    $0x1,%r14
2be2e3:  mov    (%rsi),%rsi              ; element->id
2be2e6:  callq  zbx_vector_uint64_append

Interpretation: while collecting IDs for the proxy configuration, the code appended the ID of an object pointed to by r12. It then iterated over the object's embedded pointer vector (offset 0x98/0xa0), appending each element's ID. In that object the vector holds garbage: values = 0x8000000000000020 and values_num = 3,319,116. Most likely the object reached through the proxy's lists was already freed or reused (for example a host removed from or reassigned to a proxy or proxy group during a configuration sync), or the configuration cache was otherwise corrupted.

Context that may be relevant

1. A proxy group was created the same day (~2.5 h before the crash). Its internal zabbix[proxy group,<name>,...] items immediately became not supported with "Preprocessing failed" on the discovery JSON.
2. Very high configuration churn for some active proxies, mostly hosts populated by Kubernetes LLD. Configuration sends that day, up to the crash:

proxy-k8s-A pulled configuration 5 times in the 100 seconds before the crash.

1. Many hosts trigger "cannot send list of active checks ... host is monitored by another proxy". This shows that hosts are assigned to proxy groups or proxies different from the agents' configured ones.
2. No DB errors, no out-of-memory and no cache-full messages around the crash. The OS uptime was 219 days.

Steps to reproduce

Not known. Single occurrence. Probably a race between configuration cache sync (host/item changes from LLD and/or proxy group host reassignment) and a trapper building the incremental proxy configuration.

Questions

1. Is this a known issue, and is it fixed in a later 7.0.x release (latest is 7.0.32)?
2. Can proxies newer than the server (7.0.29-7.0.32 with a 7.0.20 server) contribute to this? The crashing code path is the server building incremental config for a proxy.
3. Which object type is at the faulting location (offset 0x98/0xa0 vector inside the iterated object in zbx_dc_get_proxy_config_updates), so we can correlate it with configuration changes made before the crash?
4. Is there anything we should collect next time (e.g., enabling core dumps, DebugLevel for trappers via -R log_level_increase=trapper)?