Uploaded image for project: 'ZABBIX BUGS AND ISSUES'
  1. ZABBIX BUGS AND ISSUES
  2. ZBX-19961

server crash: something impossible just happened in misc.c,deserialize_variant

    XMLWordPrintable

Details

    • Team A
    • Sprint 84 (Jan 2022)
    • 1

    Description

      Our zabbix server crashed, and got stuck in an endless loop:

      zabbix_server [1950]: ERROR [file and function: <misc.c,deserialize_variant>, revision:68dc2b05cb, line:138] Something impossible has just happened.
        1950:20210914:184118.946 === Backtrace: ===
        1950:20210914:184118.947 14: /usr/sbin/zabbix_server: history syncer #8 [processed 0 values, 0 triggers in 0.000042 sec, syncing history](zbx_backtrace+0x52) [0x55f65d5097d2]
        1950:20210914:184118.947 13: /usr/sbin/zabbix_server: history syncer #8 [processed 0 values, 0 triggers in 0.000042 sec, syncing history](+0x22a832) [0x55f65d4f8832]
        1950:20210914:184118.947 12: /usr/sbin/zabbix_server: history syncer #8 [processed 0 values, 0 triggers in 0.000042 sec, syncing history](zbx_eval_deserialize+0x17f) [0x55f65d4f8d00]
        1950:20210914:184118.947 11: /usr/sbin/zabbix_server: history syncer #8 [processed 0 values, 0 triggers in 0.000042 sec, syncing history](zbx_eval_deserialize_dyn+0x67) [0x55f65d4f9c33]
        1950:20210914:184118.947 10: /usr/sbin/zabbix_server: history syncer #8 [processed 0 values, 0 triggers in 0.000042 sec, syncing history](prepare_triggers+0x5a) [0x55f65d4dc1d5]
        1950:20210914:184118.947 9: /usr/sbin/zabbix_server: history syncer #8 [processed 0 values, 0 triggers in 0.000042 sec, syncing history](+0x1b0825) [0x55f65d47e825]
        1950:20210914:184118.947 8: /usr/sbin/zabbix_server: history syncer #8 [processed 0 values, 0 triggers in 0.000042 sec, syncing history](+0x1b44c5) [0x55f65d4824c5]
        1950:20210914:184118.947 7: /usr/sbin/zabbix_server: history syncer #8 [processed 0 values, 0 triggers in 0.000042 sec, syncing history](zbx_sync_history_cache+0x8b) [0x55f65d483021]
        1950:20210914:184118.947 6: /usr/sbin/zabbix_server: history syncer #8 [processed 0 values, 0 triggers in 0.000042 sec, syncing history](dbsyncer_thread+0x236) [0x55f65d338b2f]
        1950:20210914:184118.947 5: /usr/sbin/zabbix_server: history syncer #8 [processed 0 values, 0 triggers in 0.000042 sec, syncing history](zbx_thread_start+0x3b) [0x55f65d51ad17]
        1950:20210914:184118.947 4: /usr/sbin/zabbix_server: history syncer #8 [processed 0 values, 0 triggers in 0.000042 sec, syncing history](MAIN_ZABBIX_ENTRY+0xe4b) [0x55f65d325a9d]
        1950:20210914:184118.947 3: /usr/sbin/zabbix_server: history syncer #8 [processed 0 values, 0 triggers in 0.000042 sec, syncing history](daemon_start+0x303) [0x55f65d509314]
        1950:20210914:184118.947 2: /usr/sbin/zabbix_server: history syncer #8 [processed 0 values, 0 triggers in 0.000042 sec, syncing history](main+0x309) [0x55f65d324b34]
        1950:20210914:184118.947 1: /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf3) [0x7fc7dcac40b3]
        1950:20210914:184118.947 0: /usr/sbin/zabbix_server: history syncer #8 [processed 0 values, 0 triggers in 0.000042 sec, syncing history](_start+0x2e) [0x55f65d32394e]
      [..]
         622:20210914:184119.008 One child process died (PID:1950,exitcode/signal:1). Exiting ...
         622:20210914:184119.726 syncing history data...
         622:20210914:184119.785 syncing history data... 88.565022%
         622:20210914:184119.785 syncing history data... 88.565022%
         622:20210914:184119.785 syncing history data... 88.565022%
         622:20210914:184119.785 syncing history data... 88.565022%
        *endlessly repeating*
      

      We had to manually stop the server (on stuck process) and restart it.

      The relevant part of the logfile and the objdump have been attached.

      Attachments

        1. all-cases
          64 kB
          Oleksii Zagorskyi
        2. debug_5.4.5.patch
          3 kB
          Vladislavs Sokurenko
        3. item_trigger_link2.diff
          26 kB
          Andris Zeila
        4. zabbix_crash_2021-09-14.log
          136 kB
          Kevin Daudt
        5. zabbix_server_5.4.3.objdump.gz
          14.15 MB
          Kevin Daudt
        6. zabbix_server.objdump-1.xz
          7.85 MB
          Andrey Tocko
        7. ZBX-19961-LLD-patch.diff
          14 kB
          Vladislavs Sokurenko

        Issue Links

          Activity

            People

              wiper Andris Zeila
              kdaudt Kevin Daudt
              Votes:
              0 Vote for this issue
              Watchers:
              9 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: