Uploaded image for project: 'ZABBIX BUGS AND ISSUES'
  1. ZABBIX BUGS AND ISSUES
  2. ZBX-19420

Server restarting/crashing after update to v5.4

XMLWordPrintable

    • Icon: Problem report Problem report
    • Resolution: Fixed
    • Icon: Major Major
    • 5.4.1rc1, 6.0.0alpha1, 6.0 (plan)
    • None
    • Server (S)
    • None
    • CentOS Linux 8, kernel 5.10.32, MySQL 8.0.25, PHP 7.4.19. Server with 192 cores and 512GB of RAM.
    • Sprint 76 (May 2021)
    • 1

      Steps to reproduce:

      1. Updating server from 5.2 to 5.4

      Result:

      zabbix_server.log

      zabbix_server [1717091]: ERROR [file and function: <variant.c,variant_compare_dbl>, revision:5059e20a75, line:572] Something impossible has just happened.
      1717091:20210519:170705.943 === Backtrace: ===
      1717091:20210519:170705.944 18: /usr/sbin/zabbix_server: history syncer #2 [processed 0 values, 0 triggers in 0.000217 sec, syncing history](zbx_backtrace+0x3f) [0x55555561130b]
      1717091:20210519:170705.944 17: /usr/sbin/zabbix_server: history syncer #2 [processed 0 values, 0 triggers in 0.000217 sec, syncing history](+0x238183) [0x555555638183]
      1717091:20210519:170705.944 16: /usr/sbin/zabbix_server: history syncer #2 [processed 0 values, 0 triggers in 0.000217 sec, syncing history](zbx_variant_compare+0x14e) [0x555555638328]
      1717091:20210519:170705.944 15: /usr/sbin/zabbix_server: history syncer #2 [processed 0 values, 0 triggers in 0.000217 sec, syncing history](+0x1fb0ed) [0x5555555fb0ed]
      1717091:20210519:170705.944 14: /usr/sbin/zabbix_server: history syncer #2 [processed 0 values, 0 triggers in 0.000217 sec, syncing history](+0x1fb34c) [0x5555555fb34c]
      1717091:20210519:170705.944 13: /usr/sbin/zabbix_server: history syncer #2 [processed 0 values, 0 triggers in 0.000217 sec, syncing history](+0x2011a3) [0x5555556011a3]
      1717091:20210519:170705.944 12: /usr/sbin/zabbix_server: history syncer #2 [processed 0 values, 0 triggers in 0.000217 sec, syncing history](zbx_eval_execute+0x4f) [0x55555560158c]
      1717091:20210519:170705.944 11: /usr/sbin/zabbix_server: history syncer #2 [processed 0 values, 0 triggers in 0.000217 sec, syncing history](+0x1e777a) [0x5555555e777a]
      1717091:20210519:170705.944 10: /usr/sbin/zabbix_server: history syncer #2 [processed 0 values, 0 triggers in 0.000217 sec, syncing history](evaluate_expressions+0x1dc) [0x5555555e7a58]
      1717091:20210519:170705.944 9: /usr/sbin/zabbix_server: history syncer #2 [processed 0 values, 0 triggers in 0.000217 sec, syncing history](+0x190f33) [0x555555590f33]
      1717091:20210519:170705.944 8: /usr/sbin/zabbix_server: history syncer #2 [processed 0 values, 0 triggers in 0.000217 sec, syncing history](+0x19454a) [0x55555559454a]
      1717091:20210519:170705.944 7: /usr/sbin/zabbix_server: history syncer #2 [processed 0 values, 0 triggers in 0.000217 sec, syncing history](zbx_sync_history_cache+0x87) [0x555555594f81]
      1717091:20210519:170705.944 6: /usr/sbin/zabbix_server: history syncer #2 [processed 0 values, 0 triggers in 0.000217 sec, syncing history](dbsyncer_thread+0x21f) [0x555555465ff2]
      1717091:20210519:170705.944 5: /usr/sbin/zabbix_server: history syncer #2 [processed 0 values, 0 triggers in 0.000217 sec, syncing history](zbx_thread_start+0x37) [0x55555562125c]
      1717091:20210519:170705.944 4: /usr/sbin/zabbix_server: history syncer #2 [processed 0 values, 0 triggers in 0.000217 sec, syncing history](MAIN_ZABBIX_ENTRY+0xdff) [0x555555454da3]
      1717091:20210519:170705.944 3: /usr/sbin/zabbix_server: history syncer #2 [processed 0 values, 0 triggers in 0.000217 sec, syncing history](daemon_start+0x2ff) [0x555555610f35]
      1717091:20210519:170705.944 2: /usr/sbin/zabbix_server: history syncer #2 [processed 0 values, 0 triggers in 0.000217 sec, syncing history](main+0x2f6) [0x555555453ed0]
      1717091:20210519:170705.944 1: /lib64/libc.so.6(__libc_start_main+0xf3) [0x7ffff47cc7b3]
      1717091:20210519:170705.944 0: /usr/sbin/zabbix_server: history syncer #2 [processed 0 values, 0 triggers in 0.000217 sec, syncing history](_start+0x2e) [0x555555452dce]
      1717030:20210519:170705.950 One child process died (PID:1717091,exitcode/signal:1). Exiting ...
      zabbix_server [1717030]: Error waiting for process with PID 1717091: [10] No child processes
      1717030:20210519:170705.983 syncing history data...
      zabbix_server [1717030]: ERROR [file and function: <variant.c,variant_compare_dbl>, revision:5059e20a75, line:572] Something impossible has just happened.
      1717030:20210519:170705.999 === Backtrace: ===
      1717030:20210519:170706.000 22: /usr/sbin/zabbix_server(zbx_backtrace+0x3f) [0x55555561130b]
      1717030:20210519:170706.000 21: /usr/sbin/zabbix_server(+0x238183) [0x555555638183]
      1717030:20210519:170706.000 20: /usr/sbin/zabbix_server(zbx_variant_compare+0x14e) [0x555555638328]
      1717030:20210519:170706.000 19: /usr/sbin/zabbix_server(+0x1fb0ed) [0x5555555fb0ed]
      1717030:20210519:170706.000 18: /usr/sbin/zabbix_server(+0x1fb34c) [0x5555555fb34c]
      1717030:20210519:170706.000 17: /usr/sbin/zabbix_server(+0x2011a3) [0x5555556011a3]
      1717030:20210519:170706.000 16: /usr/sbin/zabbix_server(zbx_eval_execute+0x4f) [0x55555560158c]
      1717030:20210519:170706.000 15: /usr/sbin/zabbix_server(+0x1e777a) [0x5555555e777a]
      1717030:20210519:170706.000 14: /usr/sbin/zabbix_server(evaluate_expressions+0x1dc) [0x5555555e7a58]
      1717030:20210519:170706.000 13: /usr/sbin/zabbix_server(+0x190f33) [0x555555590f33]
      1717030:20210519:170706.000 12: /usr/sbin/zabbix_server(+0x19454a) [0x55555559454a]
      1717030:20210519:170706.000 11: /usr/sbin/zabbix_server(+0x194b9a) [0x555555594b9a]
      1717030:20210519:170706.000 10: /usr/sbin/zabbix_server(+0x196ff6) [0x555555596ff6]
      1717030:20210519:170706.000 9: /usr/sbin/zabbix_server(free_database_cache+0x31) [0x555555597060]
      1717030:20210519:170706.000 8: /usr/sbin/zabbix_server(zbx_on_exit+0xcf) [0x5555554553af]
      1717030:20210519:170706.000 7: /usr/sbin/zabbix_server(+0x212102) [0x555555612102]
      1717030:20210519:170706.000 6: /lib64/libpthread.so.0(+0x12b20) [0x7ffff69b3b20]
      1717030:20210519:170706.000 5: /lib64/libpthread.so.0(__wait+0x22) [0x7ffff69b33d2]
      1717030:20210519:170706.000 4: /usr/sbin/zabbix_server(MAIN_ZABBIX_ENTRY+0x12f5) [0x555555455299]
      1717030:20210519:170706.000 3: /usr/sbin/zabbix_server(daemon_start+0x2ff) [0x555555610f35]
      1717030:20210519:170706.000 2: /usr/sbin/zabbix_server(main+0x2f6) [0x555555453ed0]
      1717030:20210519:170706.000 1: /lib64/libc.so.6(__libc_start_main+0xf3) [0x7ffff47cc7b3]
      1717030:20210519:170706.000 0: /usr/sbin/zabbix_server(_start+0x2e) [0x555555452dce]
      1727931:20210519:170716.028 Starting Zabbix Server. Zabbix 5.4.0 (revision 5059e20a75).
      

      journalctl log

      May 19 17:02:06 server-hostname systemd[1]: zabbix-server.service: Main process exited, code=exited, status=1/FAILURE
      May 19 17:02:06 server-hostname systemd[1]: zabbix-server.service: Failed with result 'exit-code'.
      May 19 17:02:16 server-hostname systemd[1]: zabbix-server.service: Service RestartSec=10s expired, scheduling restart.
      May 19 17:02:16 server-hostname systemd[1]: zabbix-server.service: Scheduled restart job, restart counter is at 6.
      May 19 17:02:16 server-hostname systemd[1]: Stopped Zabbix Server.
      May 19 17:02:16 server-hostname systemd[1]: Starting Zabbix Server...
      May 19 17:02:16 server-hostname systemd[1]: zabbix-server.service: Supervising process 1715814 which is not our child. We'll most likely not notice when it exits.
      May 19 17:02:16 server-hostname systemd[1]: Started Zabbix Server.
      May 19 17:02:41 server-hostname systemd[1]: zabbix-server.service: Main process exited, code=exited, status=1/FAILURE
      May 19 17:02:41 server-hostname systemd[1]: zabbix-server.service: Failed with result 'exit-code'.
      May 19 17:02:51 server-hostname systemd[1]: zabbix-server.service: Service RestartSec=10s expired, scheduling restart.
      May 19 17:02:51 server-hostname systemd[1]: zabbix-server.service: Scheduled restart job, restart counter is at 7.
      May 19 17:02:51 server-hostname systemd[1]: Stopped Zabbix Server.
      May 19 17:02:51 server-hostname systemd[1]: Starting Zabbix Server...
      May 19 17:02:51 server-hostname systemd[1]: zabbix-server.service: Supervising process 1717030 which is not our child. We'll most likely not notice when it exits.
      May 19 17:02:51 server-hostname systemd[1]: Started Zabbix Server.
      May 19 17:07:06 server-hostname systemd[1]: zabbix-server.service: Main process exited, code=exited, status=1/FAILURE
      May 19 17:07:06 server-hostname systemd[1]: zabbix-server.service: Failed with result 'exit-code'.
      May 19 17:07:16 server-hostname systemd[1]: zabbix-server.service: Service RestartSec=10s expired, scheduling restart.
      May 19 17:07:16 server-hostname systemd[1]: zabbix-server.service: Scheduled restart job, restart counter is at 8.
      May 19 17:07:16 server-hostname systemd[1]: Stopped Zabbix Server.
      May 19 17:07:16 server-hostname systemd[1]: Starting Zabbix Server...
      May 19 17:07:16 server-hostname systemd[1]: zabbix-server.service: Supervising process 1727931 which is not our child. We'll most likely not notice when it exits.
      May 19 17:07:16 server-hostname systemd[1]: Started Zabbix Server.
      May 19 17:07:42 server-hostname systemd[1]: zabbix-server.service: Main process exited, code=exited, status=1/FAILURE
      May 19 17:07:42 server-hostname systemd[1]: zabbix-server.service: Failed with result 'exit-code'.
      May 19 17:07:52 server-hostname systemd[1]: zabbix-server.service: Service RestartSec=10s expired, scheduling restart.
      May 19 17:07:52 server-hostname systemd[1]: zabbix-server.service: Scheduled restart job, restart counter is at 9.
      May 19 17:07:52 server-hostname systemd[1]: Stopped Zabbix Server.
      May 19 17:07:52 server-hostname systemd[1]: Starting Zabbix Server...
      May 19 17:07:52 server-hostname systemd[1]: zabbix-server.service: Supervising process 1729240 which is not our child. We'll most likely not notice when it exits.
      May 19 17:07:52 server-hostname systemd[1]: Started Zabbix Server.
      May 19 17:12:05 server-hostname systemd[1]: zabbix-server.service: Main process exited, code=exited, status=1/FAILURE
      May 19 17:12:05 server-hostname systemd[1]: zabbix-server.service: Failed with result 'exit-code'.
      May 19 17:12:16 server-hostname systemd[1]: zabbix-server.service: Service RestartSec=10s expired, scheduling restart.
      May 19 17:12:16 server-hostname systemd[1]: zabbix-server.service: Scheduled restart job, restart counter is at 10.
      May 19 17:12:16 server-hostname systemd[1]: Stopped Zabbix Server.
      May 19 17:12:16 server-hostname systemd[1]: Starting Zabbix Server...
      May 19 17:12:16 server-hostname systemd[1]: zabbix-server.service: Supervising process 1740019 which is not our child. We'll most likely not notice when it exits.
      May 19 17:12:16 server-hostname systemd[1]: Started Zabbix Server.
      

      zabbix-server just keeps restarting. If all hosts that are being monitored via zabbix-agent are disabled, server stops restarting (agents are also updated to latest version).

      Expected:
      Fix.

        1. zabbix_item.png
          45 kB
          Dusan Umljenovic

            wiper Andris Zeila
            dusan Dusan Umljenovic
            Team A
            Votes:
            0 Vote for this issue
            Watchers:
            9 Start watching this issue

              Created:
              Updated:
              Resolved: