-
Incident report
-
Resolution: Unresolved
-
Critical
-
None
-
6.0.40
-
None
-
Zabbix Server 6.0.19
RHEL 9.2 / PostgreSQL 15 (TimescaleDB)
We’re seeing a SIGSEGV crash happening in the escalator processes.
- From the logs, there are repeated "escalation canceled" caused by deleted triggers before the crash.
39307:20250527:085255.715 escalation canceled: trigger id:395707 deleted. 39307:20250527:085255.715 escalation canceled: trigger id:395707 deleted. 39307:20250527:085255.715 escalation canceled: trigger id:395707 deleted. 39302:20250527:085255.715 escalation canceled: trigger id:389724 deleted. 39302:20250527:085255.715 escalation canceled: trigger id:389724 deleted. 39295:20250527:085255.715 escalation canceled: trigger id:388830 deleted. 39295:20250527:085255.715 escalation canceled: trigger id:388830 deleted. 39295:20250527:085255.716 escalation canceled: trigger id:388840 deleted. 39295:20250527:085255.716 escalation canceled: trigger id:388840 deleted. 39295:20250527:085255.716 escalation canceled: trigger id:388840 deleted. 39295:20250527:085255.716 escalation canceled: trigger id:388850 deleted. 39295:20250527:085255.716 escalation canceled: trigger id:388860 deleted. 39307:20250527:085255.716 Got signal [signal:11(SIGSEGV),reason:1,refaddr:(nil)]. Crashing ...
- Looking further into the debug log, multiple escalators crash around the same time during check_escalation():
42379 - [escalator #5] 42379:20250527:103441.387 In check_escalation() escalationid:23098 status:sleep 42379:20250527:103441.388 Got signal [signal:11(SIGSEGV),reason:1,refaddr:(nil)]. Crashing ... ... 42372 - [escalator #1] 42372:20250527:103441.404 In check_escalation() escalationid:22424 status:sleep 42372:20250527:103441.404 Got signal [signal:11(SIGSEGV),reason:1,refaddr:(nil)]. Crashing ... ... 42388 - [escalator #10] 42388:20250527:103441.409 In check_escalation() escalationid:23875 status:sleep 42388:20250527:103441.409 Got signal [signal:11(SIGSEGV),reason:1,refaddr:(nil)]. Crashing ...
- Backtrace from the crash:
42379:20250527:103441.388 === Backtrace: === 42379:20250527:103441.389 21: /usr/sbin/zabbix_server: escalator #5 [processed 0 escalations in 0.000000 sec, processing escalations](zbx_backtrace+0x42) [0x5634a73a1e84] 42379:20250527:103441.389 20: /usr/sbin/zabbix_server: escalator #5 [processed 0 escalations in 0.000000 sec, processing escalations](zbx_log_fatal_info+0x151) [0x5634a73a20fb] 42379:20250527:103441.389 19: /usr/sbin/zabbix_server: escalator #5 [processed 0 escalations in 0.000000 sec, processing escalations](+0x261963) [0x5634a73a2963] 42379:20250527:103441.389 18: /lib64/libc.so.6(+0x54df0) [0x7fac5e454df0] 42379:20250527:103441.389 17: /lib64/libc.so.6(+0xce2fc) [0x7fac5e4ce2fc] 42379:20250527:103441.389 16: /lib64/libc.so.6(__strdup+0x13) [0x7fac5e4b1d43] 42379:20250527:103441.389 15: /usr/sbin/zabbix_server: escalator #5 [processed 0 escalations in 0.000000 sec, processing escalations](zbx_strdup2+0x4f) [0x5634a73c2d1d] 42379:20250527:103441.389 14: /usr/sbin/zabbix_server: escalator #5 [processed 0 escalations in 0.000000 sec, processing escalations](DCget_trigger+0x13e) [0x5634a733909e] 42379:20250527:103441.389 13: /usr/sbin/zabbix_server: escalator #5 [processed 0 escalations in 0.000000 sec, processing escalations](DCconfig_get_triggers_by_triggerids+0xe9) [0x5634a733a280] 42379:20250527:103441.390 12: /usr/sbin/zabbix_server: escalator #5 [processed 0 escalations in 0.000000 sec, processing escalations](+0xc9805) [0x5634a720a805] 42379:20250527:103441.390 11: /usr/sbin/zabbix_server: escalator #5 [processed 0 escalations in 0.000000 sec, processing escalations](+0xc9ec1) [0x5634a720aec1] 42379:20250527:103441.390 10: /usr/sbin/zabbix_server: escalator #5 [processed 0 escalations in 0.000000 sec, processing escalations](+0xcb9d2) [0x5634a720c9d2] 42379:20250527:103441.390 9: /usr/sbin/zabbix_server: escalator #5 [processed 0 escalations in 0.000000 sec, processing escalations](+0xcc9e5) [0x5634a720d9e5] 42379:20250527:103441.390 8: /usr/sbin/zabbix_server: escalator #5 [processed 0 escalations in 0.000000 sec, processing escalations](escalator_thread+0x21c) [0x5634a720dc77] 42379:20250527:103441.390 7: /usr/sbin/zabbix_server: escalator #5 [processed 0 escalations in 0.000000 sec, processing escalations](zbx_thread_start+0x37) [0x5634a73a4ced] 42379:20250527:103441.390 6: /usr/sbin/zabbix_server: escalator #5 [processed 0 escalations in 0.000000 sec, processing escalations](+0x69d75) [0x5634a71aad75] 42379:20250527:103441.390 5: /usr/sbin/zabbix_server: escalator #5 [processed 0 escalations in 0.000000 sec, processing escalations](MAIN_ZABBIX_ENTRY+0x830) [0x5634a71abd7b] 42379:20250527:103441.390 4: /usr/sbin/zabbix_server: escalator #5 [processed 0 escalations in 0.000000 sec, processing escalations](daemon_start+0x3ab) [0x5634a73a1a7e] 42379:20250527:103441.390 3: /usr/sbin/zabbix_server: escalator #5 [processed 0 escalations in 0.000000 sec, processing escalations](main+0x357) [0x5634a71aa22f] 42379:20250527:103441.390 2: /lib64/libc.so.6(+0x3feb0) [0x7fac5e43feb0] 42379:20250527:103441.390 1: /lib64/libc.so.6(__libc_start_main+0x80) [0x7fac5e43ff60] 42379:20250527:103441.390 0: /usr/sbin/zabbix_server: escalator #5 [processed 0 escalations in 0.000000 sec, processing escalations](_start+0x25) [0x5634a71a8ee5]
According to the backtrace, the crash may have occurred during the call from DCconfig_get_triggers_by_triggerids(), DCget_trigger()