Uploaded image for project: 'ZABBIX BUGS AND ISSUES'
  1. ZABBIX BUGS AND ISSUES
  2. ZBX-25070

Crash in server when processing proxy data

XMLWordPrintable

    • Icon: Incident report Incident report
    • Resolution: Won't fix
    • Icon: Critical Critical
    • None
    • 7.0.2
    • Server (S)
    • None
    • S24-W34/35

      Suddenly our zabbix server crahes after startup.

      It then does endless restarts.

      We tracked this down to the processing of an incomming large proxy message

      System runnig on Debian 12.6 with Zabbix server 7.0.2 (revision d1b0c3308ce)

      Attached are

      • the zabbix server config (with sensitive things masked)
      • the DebugLevel=4 of the crashing process, the PSK are masked and some client names masked too

       

      The relevant thiings seems to be here:

      141115:20240819:174955.196 zbx_setproctitle() title:'trapper #5 [processed data in 0.026269 sec, waiting for connection]'
      141115:20240819:174955.196 In zbx_tls_accept()
      141115:20240819:174955.197 zbx_psk_server_cb() requested PSK identity "XXXXXXXXXX"
      141115:20240819:174955.215 End of zbx_tls_accept():SUCCEED (established TLSv1.3 TLS_CHACHA20_POLY1305_SHA256)
      141115:20240819:174955.215 zbx_setproctitle() title:'trapper #5 [processing data]'
      141115:20240819:174955.215 In zbx_ipc_async_socket_recv() timeout:0
      141115:20240819:174955.215 End of zbx_ipc_async_socket_recv():0
      141115:20240819:174955.231 trapper got '\{"request":"proxy data","host":"Zabbix proxy Digmesa Ipsach","session":"f00c704ec5b34d5ab03ff49faa2f0c32","interface availability":[{"interfaceid":28,"available":1,"error":""},\{"interfaceid":377,"available":1,"error":""},\{"interfaceid":515,"available":1,"error":""},{"interface
      141115:20240819:174955.232 In recv_proxy_data()
      141115:20240819:174955.235 Proxy "XXXXXXXXXXXX" version 6.0.30 is outdated, only data collection and remote execution is available with server version 7.0.2.
      141115:20240819:174955.235 In zbx_hc_check_proxy() proxyid:10130
      141115:20240819:174955.235 End of zbx_hc_check_proxy():SUCCEED
      141115:20240819:174955.235 In zbx_process_proxy_data()
      141115:20240819:174955.236 zbx_process_proxy_data() flag_win:7/0 flag:48 proxy_status:0 period_end:1724082595 delay:1320 timestamp:1724082595 lastaccess:1724081275 proxy_delay:5159 more:1
      141115:20240819:174955.239 Got signal [signal:11(SIGSEGV),reason:1,refaddr:(nil)]. Crashing ...
      141115:20240819:174955.241 ====== Fatal information: ======
      141115:20240819:174955.241 Program counter: 0x7f00cd63fea6
      141115:20240819:174955.241 === Registers: ===
      141115:20240819:174955.241 r8      =     7f004a626fa4 =      139639224692644 =      139639224692644
      141115:20240819:174955.242 r9      =                0 =                    0 =                    0
      141115:20240819:174955.242 r10     =                0 =                    0 =                    0
      141115:20240819:174955.242 r11     =               20 =                   32 =                   32
      141115:20240819:174955.242 r12     =     7f004a626fa4 =      139639224692644 =      139639224692644
      141115:20240819:174955.242 r13     =     7f004a626fa8 =      139639224692648 =      139639224692648
      141115:20240819:174955.242 r14     =     7f004a626f98 =      139639224692632 =      139639224692632
      141115:20240819:174955.242 r15     =     55d9d4f57870 =       94394069121136 =       94394069121136
      141115:20240819:174955.242 rdi     =                0 =                    0 =                    0
      141115:20240819:174955.243 rsi     =     55d9d4f57870 =       94394069121136 =       94394069121136
      141115:20240819:174955.243 rbp     =     55d9d4f57848 =       94394069121096 =       94394069121096
      141115:20240819:174955.243 rbx     =                3 =                    3 =                    3
      141115:20240819:174955.243 rdx     =     7f004a626f98 =      139639224692632 =      139639224692632
      141115:20240819:174955.243 rax     =                0 =                    0 =                    0
      141115:20240819:174955.243 rcx     =               30 =                   48 =                   48
      141115:20240819:174955.243 rsp     =     7fff51f605c8 =      140734568465864 =      140734568465864
      141115:20240819:174955.244 rip     =     7f00cd63fea6 =      139641422610086 =      139641422610086
      141115:20240819:174955.244 efl     =            10283 =                66179 =                66179
      141115:20240819:174955.245 csgsfs  =   2b000000000033 =    12103423998558259 =    12103423998558259
      141115:20240819:174955.245 err     =                4 =                    4 =                    4
      141115:20240819:174955.245 trapno  =                e =                   14 =                   14
      141115:20240819:174955.245 oldmask =                0 =                    0 =                    0
      141115:20240819:174955.245 cr2     =                0 =                    0 =                    0
      141115:20240819:174955.245 === Backtrace: ===
      141115:20240819:174955.247 18: /usr/sbin/zabbix_server: trapper #5 [processing data](zbx_backtrace+0x3b) [0x55d9d38e7ccb]
      141115:20240819:174955.247 17: /usr/sbin/zabbix_server: trapper #5 [processing data](zbx_log_fatal_info+0x17d) [0x55d9d38e7f3d]
      141115:20240819:174955.248 16: /usr/sbin/zabbix_server: trapper #5 [processing data](+0x3c35f6) [0x55d9d38e85f6]
      141115:20240819:174955.248 15: /lib/x86_64-linux-gnu/libc.so.6(+0x3c050) [0x7f00cd5d4050]
      141115:20240819:174955.248 14: /lib/x86_64-linux-gnu/libc.so.6(+0xa7ea6) [0x7f00cd63fea6]
      141115:20240819:174955.248 13: /usr/sbin/zabbix_server: trapper #5 [processing data](+0x265853) [0x55d9d378a853]
      141115:20240819:174955.248 12: /usr/sbin/zabbix_server: trapper #5 [processing data](zbx_dc_set_interfaces_availability+0x61) [0x55d9d3796901]
      141115:20240819:174955.248 11: /usr/sbin/zabbix_server: trapper #5 [processing data](zbx_process_proxy_data+0x23db) [0x55d9d38abe5b]
      141115:20240819:174955.248 10: /usr/sbin/zabbix_server: trapper #5 [processing data](recv_proxy_data+0x315) [0x55d9d363acf5]
      141115:20240819:174955.248 9: /usr/sbin/zabbix_server: trapper #5 [processing data](zbx_trapper_process_request_server+0xe4) [0x55d9d363a7b4]
      141115:20240819:174955.248 8: /usr/sbin/zabbix_server: trapper #5 [processing data](+0x1093f8) [0x55d9d362e3f8]
      141115:20240819:174955.248 7: /usr/sbin/zabbix_server: trapper #5 [processing data](zbx_trapper_thread+0x422) [0x55d9d362f2c2]
      141115:20240819:174955.248 6: /usr/sbin/zabbix_server: trapper #5 [processing data](zbx_thread_start+0x20) [0x55d9d381c600]
      141115:20240819:174955.248 5: /usr/sbin/zabbix_server: trapper #5 [processing data](+0xcf44a) [0x55d9d35f444a]
      141115:20240819:174955.249 4: /usr/sbin/zabbix_server: trapper #5 [processing data](MAIN_ZABBIX_ENTRY+0xe55) [0x55d9d35f5885]
      141115:20240819:174955.249 3: /usr/sbin/zabbix_server: trapper #5 [processing data](main+0x37e) [0x55d9d35e9b5e]
      141115:20240819:174955.249 2: /lib/x86_64-linux-gnu/libc.so.6(+0x2724a) [0x7f00cd5bf24a]
      141115:20240819:174955.249 1: /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x85) [0x7f00cd5bf305]
      141115:20240819:174955.249 0: /usr/sbin/zabbix_server: trapper #5 [processing data](_start+0x21) [0x55d9d35f0bb1]
      141115:20240819:174955.249 === Memory map: ===
      141115:20240819:174955.249 55d9d3525000-55d9d35e2000 r--p 00000000 fe:01 152010                     /usr/sbin/zabbix_server
      141115:20240819:174955.250 55d9d35e2000-55d9d391b000 r-xp 000bd000 fe:01 152010                     /usr/sbin/zabbix_server
      141115:20240819:174955.250 55d9d391b000-55d9d3a4e000 r--p 003f6000 fe:01 152010                     /usr/sbin/zabbix_server
      141115:20240819:174955.250 55d9d3a4e000-55d9d3a52000 r--p 00528000 fe:01 152010                     /usr/sbin/zabbix_server
      141115:20240819:174955.250 55d9d3a52000-55d9d3b51000 rw-p 0052c000 fe:01 152010                     /usr/sbin/zabbix_server
      141115:20240819:174955.250 55d9d3b51000-55d9d3b63000 rw-p 00000000 00:00 0
      141115:20240819:174955.254 55d9d4ea2000-55d9d4ee4000 rw-p 00000000 00:00 0                          [heap]
      141115:20240819:174955.254 55d9d4ee4000-55d9d4f06000 rw-p 00000000 00:00 0                          [heap]
      141115:20240819:174955.254 55d9d4f06000-55d9d5494000 rw-p 00000000 00:00 0                          [heap]
      141115:20240819:174955.254 7f00498b0000-7f00498f0000 rw-s 00000000 00:01 35258375                   /SYSV00000000 (deleted)
      
       
      
      

       

      After chaning the PSK of the specific proxy to some invalid values (so no more data is processed) the server remains up and running.

      But of course all data from that proxy is no longer ingested/processed.

      The JSON data structure of the failing proxy data is about 136300 bytes in size

        1. 141115.log
          419 kB
          André Schild
        2. zabbix_server.conf
          0.6 kB
          André Schild

            wiper Andris Zeila
            aschild André Schild
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

              Created:
              Updated:
              Resolved: