Server crash in preprocessing caused by JSON error check step on integer item value

XMLWordPrintable

    • Sprint candidates
    • 0.25

      Fix is required at least for 7.0 and 7.4 versions

         181:20260129:082354.681 In zbx_snmp_get_values() num:1 level:0
          33:20260129:082358.613 [43] Got signal [signal:11(SIGSEGV),reason:128,refaddr:(nil)]. Crashing ...
          33:20260129:082358.613 [43] ====== Fatal information: ======
          33:20260129:082358.613 [43] Program counter: 0x5555827f1229
          33:20260129:082358.613 [43] === Registers: ===
          33:20260129:082358.613 [43] r8      =                1 =                    1 =                    1
          33:20260129:082358.613 [43] r9      =                1 =                    1 =                    1
          33:20260129:082358.613 [43] r10     =     7fb0c9c05aba =      140397275798202 =      140397275798202
          33:20260129:082358.613 [43] r11     =                1 =                    1 =                    1
          33:20260129:082358.613 [43] r12     =     7faf2d292830 =      140390353676336 =      140390353676336
          33:20260129:082358.613 [43] r13     =     7faf2d292820 =      140390353676320 =      140390353676320
          33:20260129:082358.613 [43] r14     =     5555873d0250 =       93825829503568 =       93825829503568
          33:20260129:082358.613 [43] r15     =     7faf2d292830 =      140390353676336 =      140390353676336
          33:20260129:082358.614 [43] rdi     = 402d99999999999a =  4624521277365393818 =  4624521277365393818
          33:20260129:082358.614 [43] rsi     =     7faf2d292830 =      140390353676336 =      140390353676336
          33:20260129:082358.614 [43] rbp     =     555582970dfd =       93825751518717 =       93825751518717
          33:20260129:082358.614 [43] rbx     = 402d99999999999a =  4624521277365393818 =  4624521277365393818
          33:20260129:082358.614 [43] rdx     =                0 =                    0 =                    0
          33:20260129:082358.614 [43] rax     =                0 =                    0 =                    0
          33:20260129:082358.614 [43] rcx     =                4 =                    4 =                    4
          33:20260129:082358.614 [43] rsp     =     7faf2d2927f0 =      140390353676272 =      140390353676272
          33:20260129:082358.614 [43] rip     =     5555827f1229 =       93825749946921 =       93825749946921
          33:20260129:082358.614 [43] efl     =            10206 =                66054 =                66054
          33:20260129:082358.614 [43] csgsfs  =   2b000000000033 =    12103423998558259 =    12103423998558259
          33:20260129:082358.614 [43] err     =                0 =                    0 =                    0
          33:20260129:082358.614 [43] trapno  =                d =                   13 =                   13
          33:20260129:082358.614 [43] oldmask =             4a07 =                18951 =                18951
          33:20260129:082358.614 [43] cr2     =                0 =                    0 =                    0
          33:20260129:082358.614 [43] === Backtrace: ===
          33:20260129:082358.620 [43] 10: /usr/sbin/zabbix_server: preprocessing manager #1 [queued 3979, processed 4032 values, idle 4.946807 sec during 5.006355 sec](zbx_backtrace+0x52) [0x5555828bf9f2]
          33:20260129:082358.620 [43] 9: /usr/sbin/zabbix_server: preprocessing manager #1 [queued 3979, processed 4032 values, idle 4.946807 sec during 5.006355 sec](zbx_log_fatal_info+0x13d) [0x5555828bfc2d]
          33:20260129:082358.620 [43] 8: /usr/sbin/zabbix_server: preprocessing manager #1 [queued 3979, processed 4032 values, idle 4.946807 sec during 5.006355 sec](+0x40c6ce) [0x5555828c06ce]
          33:20260129:082358.620 [43] 7: /lib/x86_64-linux-gnu/libc.so.6(+0x45330) [0x7fb0c9bfc330]
          33:20260129:082358.620 [43] 6: /usr/sbin/zabbix_server: preprocessing manager #1 [queued 3979, processed 4032 values, idle 4.946807 sec during 5.006355 sec](zbx_json_open+0x19) [0x5555827f1229]
          33:20260129:082358.620 [43] 5: /usr/sbin/zabbix_server: preprocessing manager #1 [queued 3979, processed 4032 values, idle 4.946807 sec during 5.006355 sec](item_preproc_get_error_from_json+0x66) [0x5555827b29b6]
          33:20260129:082358.620 [43] 4: /usr/sbin/zabbix_server: preprocessing manager #1 [queued 3979, processed 4032 values, idle 4.946807 sec during 5.006355 sec](pp_execute_step+0x87b) [0x5555827b061b]
          33:20260129:082358.620 [43] 3: /usr/sbin/zabbix_server: preprocessing manager #1 [queued 3979, processed 4032 values, idle 4.946807 sec during 5.006355 sec](pp_execute+0x319) [0x5555827b1129]
          33:20260129:082358.620 [43] 2: /usr/sbin/zabbix_server: preprocessing manager #1 [queued 3979, processed 4032 values, idle 4.946807 sec during 5.006355 sec](+0x2f5287) [0x5555827a9287]
          33:20260129:082358.620 [43] 1: /lib/x86_64-linux-gnu/libc.so.6(+0x9caa4) [0x7fb0c9c53aa4]
          33:20260129:082358.620 [43] 0: /lib/x86_64-linux-gnu/libc.so.6(+0x129c6c) [0x7fb0c9ce0c6c]
          33:20260129:082358.620 [43] === Memory map: ===
      ...
      33:20260129:082358.630 [29] thread stopped [preprocessing worker #29]
          33:20260129:082358.630 [19] thread stopped [preprocessing worker #19]
          33:20260129:082358.630 [42] thread stopped [preprocessing worker #42]
          33:20260129:082358.630 [13] thread stopped [preprocessing worker #13]
          33:20260129:082358.630 [58] thread stopped [preprocessing worker #58]
          33:20260129:082358.630 [60] thread stopped [preprocessing worker #60]
          33:20260129:082358.630 [27] thread stopped [preprocessing worker #27]
          33:20260129:082358.630 [30] thread stopped [preprocessing worker #30]
          33:20260129:082358.630 [56] thread stopped [preprocessing worker #56]
          33:20260129:082358.630 [63] thread stopped [preprocessing worker #63]
          33:20260129:082358.630 [7] thread stopped [preprocessing worker #7]
          33:20260129:082358.630 [14] thread stopped [preprocessing worker #14]
          33:20260129:082358.630 [39] thread stopped [preprocessing worker #39]
          33:20260129:082358.630 [5] thread stopped [preprocessing worker #5]
      ...
      108:20260129:082358.955 cannot write to IPC socket: Broken pipe
         108:20260129:082358.955 cannot send data to preprocessing service
           1:20260129:082358.967 One child process died (PID:33,exitcode/signal:0). Exiting ...
          23:20260129:082358.967 cannot read alert manager service request
          55:20260129:082358.968 syncing history data in progress...
          25:20260129:082358.968 cannot read alert manager service request
         778:20260129:082358.968 cannot read alert syncer request
          26:20260129:082358.968 cannot read alert manager service request
          29:20260129:082358.968 cannot read alert manager service request
          27:20260129:082358.968 cannot read alert manager service request
          35:20260129:082358.968 cannot read LLD manager service request
          30:20260129:082358.968 cannot read alert manager service request
          31:20260129:082358.968 cannot read alert manager service request
          32:20260129:082358.968 cannot read alert manager service request
          36:20260129:082358.969 cannot read LLD manager service request
          24:20260129:082358.969 cannot read alert manager service request
           1:20260129:082358.969 cannot write to IPC socket: Bad file descriptor
           1:20260129:082358.969 cannot send RTC notification to client "poller" #38
          54:20260129:082358.972 thread stopped [discovery worker #1]
          54:20260129:082358.972 thread stopped [discovery worker #3]
          54:20260129:082358.972 thread stopped [discovery worker #4]
          54:20260129:082358.972 thread stopped [discovery worker #5]
          54:20260129:082358.972 thread stopped [discovery worker #2]
         804:20260129:082358.976 thread stopped
         803:20260129:082358.977 thread stopped
          19:20260129:082358.977 HA manager has been paused
          56:20260129:082359.011 syncing history data... 100.000000%
          56:20260129:082359.011 syncing history data done
          56:20260129:082359.016 syncing trend data in progress...
          68:20260129:082359.096 cannot process proxy data from passive proxy at "XXXXX: Zabbix server shutdown in progress
         360:20260129:082359.211 cannot write to IPC socket: Broken pipe
      ...
      zabbix_server [1]: child process exited (PID:744,exitcode:1).
      zabbix_server [1]: child process exited (PID:747,exitcode:1).
      zabbix_server [1]: child process exited (PID:748,exitcode:1).
      zabbix_server [1]: child process exited (PID:749,exitcode:1).
      zabbix_server [1]: child process exited (PID:750,exitcode:1).
      zabbix_server [1]: child process exited (PID:755,exitcode:1).
      zabbix_server [1]: child process exited (PID:757,exitcode:1).
         805:20260129:082420.286 thread stopped
          19:20260129:082420.673 HA manager has been stopped
           1:20260129:082420.713 syncing history data...
           1:20260129:082420.734 syncing history data... 100.000000%
           1:20260129:082420.734 syncing history data done
           1:20260129:082420.745 syncing trend data... 99.997842%
           1:20260129:082420.766 syncing trend data... 100.000000%
           1:20260129:082420.766 syncing trend data done
           1:20260129:082420.828 Zabbix Server stopped. Zabbix 7.4.6 (revision 626d2b8).
      

      Can see zbx_json_open, then item_preproc_get_error_from_json and pp_execute_step in above log fragments.

      Identified a template that had this 2nd step (Check for error in JSON) on numeric SNMP item

      After removing it, server seems stable again.

            Assignee:
            Natalja Romancaka
            Reporter:
            Michal Kudlacz
            Team A
            Votes:
            1 Vote for this issue
            Watchers:
            5 Start watching this issue

              Created:
              Updated:

                Estimated:
                Original Estimate - Not Specified
                Not Specified
                Remaining:
                Remaining Estimate - Not Specified
                Not Specified
                Logged:
                Time Spent - 0.5h
                0.5h