Uploaded image for project: 'ZABBIX BUGS AND ISSUES'
  1. ZABBIX BUGS AND ISSUES
  2. ZBX-26522

Crash in escalator during check_escalation()

XMLWordPrintable

    • Icon: Incident report Incident report
    • Resolution: Unresolved
    • Icon: Critical Critical
    • None
    • 6.0.40
    • Server (S)
    • None
    • Zabbix Server 6.0.19
      RHEL 9.2 / PostgreSQL 15 (TimescaleDB)

      We’re seeing a SIGSEGV crash happening in the escalator processes.

      • From the logs, there are repeated "escalation canceled" caused by deleted triggers before the crash.
         39307:20250527:085255.715 escalation canceled: trigger id:395707 deleted.
         39307:20250527:085255.715 escalation canceled: trigger id:395707 deleted.
         39307:20250527:085255.715 escalation canceled: trigger id:395707 deleted.
         39302:20250527:085255.715 escalation canceled: trigger id:389724 deleted.
         39302:20250527:085255.715 escalation canceled: trigger id:389724 deleted.
         39295:20250527:085255.715 escalation canceled: trigger id:388830 deleted.
         39295:20250527:085255.715 escalation canceled: trigger id:388830 deleted.
         39295:20250527:085255.716 escalation canceled: trigger id:388840 deleted.
         39295:20250527:085255.716 escalation canceled: trigger id:388840 deleted.
         39295:20250527:085255.716 escalation canceled: trigger id:388840 deleted.
         39295:20250527:085255.716 escalation canceled: trigger id:388850 deleted.
         39295:20250527:085255.716 escalation canceled: trigger id:388860 deleted.
         39307:20250527:085255.716 Got signal [signal:11(SIGSEGV),reason:1,refaddr:(nil)]. Crashing ...
        
      • Looking further into the debug log, multiple escalators crash around the same time during check_escalation():
         42379 - [escalator #5]
         42379:20250527:103441.387 In check_escalation() escalationid:23098 status:sleep
         42379:20250527:103441.388 Got signal [signal:11(SIGSEGV),reason:1,refaddr:(nil)]. Crashing ...
        ...
         42372 - [escalator #1]
         42372:20250527:103441.404 In check_escalation() escalationid:22424 status:sleep
         42372:20250527:103441.404 Got signal [signal:11(SIGSEGV),reason:1,refaddr:(nil)]. Crashing ...
        ...
         42388 - [escalator #10]
         42388:20250527:103441.409 In check_escalation() escalationid:23875 status:sleep
         42388:20250527:103441.409 Got signal [signal:11(SIGSEGV),reason:1,refaddr:(nil)]. Crashing ...
        
      • Backtrace from the crash:
         42379:20250527:103441.388 === Backtrace: ===
         42379:20250527:103441.389 21: /usr/sbin/zabbix_server: escalator #5 [processed 0 escalations in 0.000000 sec, processing escalations](zbx_backtrace+0x42) [0x5634a73a1e84]
         42379:20250527:103441.389 20: /usr/sbin/zabbix_server: escalator #5 [processed 0 escalations in 0.000000 sec, processing escalations](zbx_log_fatal_info+0x151) [0x5634a73a20fb]
         42379:20250527:103441.389 19: /usr/sbin/zabbix_server: escalator #5 [processed 0 escalations in 0.000000 sec, processing escalations](+0x261963) [0x5634a73a2963]
         42379:20250527:103441.389 18: /lib64/libc.so.6(+0x54df0) [0x7fac5e454df0]
         42379:20250527:103441.389 17: /lib64/libc.so.6(+0xce2fc) [0x7fac5e4ce2fc]
         42379:20250527:103441.389 16: /lib64/libc.so.6(__strdup+0x13) [0x7fac5e4b1d43]
         42379:20250527:103441.389 15: /usr/sbin/zabbix_server: escalator #5 [processed 0 escalations in 0.000000 sec, processing escalations](zbx_strdup2+0x4f) [0x5634a73c2d1d]
         42379:20250527:103441.389 14: /usr/sbin/zabbix_server: escalator #5 [processed 0 escalations in 0.000000 sec, processing escalations](DCget_trigger+0x13e) [0x5634a733909e]
         42379:20250527:103441.389 13: /usr/sbin/zabbix_server: escalator #5 [processed 0 escalations in 0.000000 sec, processing escalations](DCconfig_get_triggers_by_triggerids+0xe9) [0x5634a733a280]
         42379:20250527:103441.390 12: /usr/sbin/zabbix_server: escalator #5 [processed 0 escalations in 0.000000 sec, processing escalations](+0xc9805) [0x5634a720a805]
         42379:20250527:103441.390 11: /usr/sbin/zabbix_server: escalator #5 [processed 0 escalations in 0.000000 sec, processing escalations](+0xc9ec1) [0x5634a720aec1]
         42379:20250527:103441.390 10: /usr/sbin/zabbix_server: escalator #5 [processed 0 escalations in 0.000000 sec, processing escalations](+0xcb9d2) [0x5634a720c9d2]
         42379:20250527:103441.390 9: /usr/sbin/zabbix_server: escalator #5 [processed 0 escalations in 0.000000 sec, processing escalations](+0xcc9e5) [0x5634a720d9e5]
         42379:20250527:103441.390 8: /usr/sbin/zabbix_server: escalator #5 [processed 0 escalations in 0.000000 sec, processing escalations](escalator_thread+0x21c) [0x5634a720dc77]
         42379:20250527:103441.390 7: /usr/sbin/zabbix_server: escalator #5 [processed 0 escalations in 0.000000 sec, processing escalations](zbx_thread_start+0x37) [0x5634a73a4ced]
         42379:20250527:103441.390 6: /usr/sbin/zabbix_server: escalator #5 [processed 0 escalations in 0.000000 sec, processing escalations](+0x69d75) [0x5634a71aad75]
         42379:20250527:103441.390 5: /usr/sbin/zabbix_server: escalator #5 [processed 0 escalations in 0.000000 sec, processing escalations](MAIN_ZABBIX_ENTRY+0x830) [0x5634a71abd7b]
         42379:20250527:103441.390 4: /usr/sbin/zabbix_server: escalator #5 [processed 0 escalations in 0.000000 sec, processing escalations](daemon_start+0x3ab) [0x5634a73a1a7e]
         42379:20250527:103441.390 3: /usr/sbin/zabbix_server: escalator #5 [processed 0 escalations in 0.000000 sec, processing escalations](main+0x357) [0x5634a71aa22f]
         42379:20250527:103441.390 2: /lib64/libc.so.6(+0x3feb0) [0x7fac5e43feb0]
         42379:20250527:103441.390 1: /lib64/libc.so.6(__libc_start_main+0x80) [0x7fac5e43ff60]
         42379:20250527:103441.390 0: /usr/sbin/zabbix_server: escalator #5 [processed 0 escalations in 0.000000 sec, processing escalations](_start+0x25) [0x5634a71a8ee5]
        

        According to the backtrace, the crash may have occurred during the call from DCconfig_get_triggers_by_triggerids(), DCget_trigger()

            zabbix.support Zabbix Support Team
            JKKim Kim Jongkwon
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated: