Uploaded image for project: 'ZABBIX BUGS AND ISSUES'
  1. ZABBIX BUGS AND ISSUES
  2. ZBX-24049

Zabbix server crash: [file and function: <audit.c,zbx_audit_update_json_append_no_value>, revision:623853f8d6f, line:538] Something impossible has just happened.

XMLWordPrintable

    • Team C
    • S24-W6/7, S24-W8/9, S24-W10/11
    • 1

      Zabbix server can sometimes crash with following log:

       27369:20240206:181828.136 Starting Zabbix Server. Zabbix 6.0.26 (revision 623853f8d6f).
       27369:20240206:181828.136 ****** Enabled features ******
       27369:20240206:181828.136 SNMP monitoring:           YES
       27369:20240206:181828.136 IPMI monitoring:           YES
       27369:20240206:181828.136 Web monitoring:            YES
       27369:20240206:181828.136 VMware monitoring:         YES
       27369:20240206:181828.136 SMTP authentication:       YES
       27369:20240206:181828.136 ODBC:                      YES
       27369:20240206:181828.136 SSH support:               YES
       27369:20240206:181828.136 IPv6 support:              YES
       27369:20240206:181828.136 TLS support:               YES
       27369:20240206:181828.136 ******************************
       27369:20240206:181828.136 using configuration file: /etc/zabbix/zabbix_server.conf
       27369:20240206:181828.438 current database version (mandatory/optional): 06000000/06000044
       27369:20240206:181828.438 required mandatory version: 06000000
       27375:20240206:181828.589 starting HA manager
       27375:20240206:181828.628 HA manager started in active mode
       27369:20240206:181828.629 server #0 started [main process]
       27376:20240206:181828.631 server #1 started [service manager #1]
       27377:20240206:181828.632 server #2 started [configuration syncer #1]
       27377:20240206:181828.808 cannot get secrets for path "zabbix/macros/training": "VaultToken" configuration parameter or "VAULT_TOKEN" environment variable should be defined
       27378:20240206:181828.853 server #3 started [alert manager #1]
       27379:20240206:181828.854 server #4 started [alerter #1]
       27380:20240206:181828.854 server #5 started [alerter #2]
       27381:20240206:181828.855 server #6 started [alerter #3]
       27382:20240206:181828.855 server #7 started [preprocessing manager #1]
       27383:20240206:181828.856 server #8 started [preprocessing worker #1]
       27384:20240206:181828.857 server #9 started [preprocessing worker #2]
       27385:20240206:181828.857 server #10 started [preprocessing worker #3]
       27386:20240206:181828.857 server #11 started [lld manager #1]
       27387:20240206:181828.858 server #12 started [lld worker #1]
       27388:20240206:181828.859 server #13 started [lld worker #2]
       27389:20240206:181828.860 server #14 started [housekeeper #1]
       27390:20240206:181828.860 server #15 started [timer #1]
       27391:20240206:181828.861 server #16 started [http poller #1]
       27392:20240206:181828.884 server #17 started [discoverer #1]
       27393:20240206:181828.889 server #18 started [history syncer #1]
       27394:20240206:181828.892 server #19 started [history syncer #2]
       27395:20240206:181828.893 server #20 started [history syncer #3]
       27396:20240206:181828.896 server #21 started [history syncer #4]
       27397:20240206:181828.899 server #22 started [escalator #1]
       27398:20240206:181828.902 server #23 started [proxy poller #1]
       27399:20240206:181828.904 server #24 started [self-monitoring #1]
       27400:20240206:181828.905 server #25 started [task manager #1]
       27401:20240206:181828.905 server #26 started [poller #1]
       27402:20240206:181828.936 server #27 started [poller #2]
       27403:20240206:181828.943 server #28 started [poller #3]
       27404:20240206:181828.946 server #29 started [poller #4]
       27405:20240206:181828.986 server #30 started [poller #5]
       27406:20240206:181828.991 server #31 started [unreachable poller #1]
       27407:20240206:181828.993 server #32 started [trapper #1]
       27408:20240206:181828.995 server #33 started [trapper #2]
       27409:20240206:181828.997 server #34 started [trapper #3]
       27410:20240206:181828.999 server #35 started [trapper #4]
       27411:20240206:181829.001 server #36 started [trapper #5]
       27412:20240206:181829.004 server #37 started [icmp pinger #1]
       27413:20240206:181829.005 server #38 started [alert syncer #1]
       27414:20240206:181829.005 server #39 started [history poller #1]
       27415:20240206:181829.008 server #40 started [history poller #2]
       27416:20240206:181829.010 server #41 started [history poller #3]
       27417:20240206:181829.014 server #42 started [history poller #4]
       27418:20240206:181829.029 server #43 started [history poller #5]
       27419:20240206:181829.031 server #44 started [availability manager #1]
       27420:20240206:181829.032 server #45 started [trigger housekeeper #1]
       27421:20240206:181829.032 server #46 started [odbc poller #1]
       27407:20240206:181837.345 cannot get secrets for path "zabbix/macros/training": "VaultToken" configuration parameter or "VAULT_TOKEN" environment variable should be defined
       27407:20240206:181837.347 sending configuration data to proxy "Training proxy active" at "127.0.0.1", datalen 40252, bytes 6256 with compression ratio 6.4
       27398:20240206:181903.043 sending configuration data to proxy "Training proxy passive" at "143.110.187.140", datalen 90069, bytes 11998 with compression ratio 7.5
       27409:20240206:181918.011 Failed to find audit entry: 19954, id_table: 4, key: triggerprototype.dependencies[55418]
       27409:20240206:181918.011 Audit entries:
       27409:20240206:181918.011 Audit entry existing id: 29171, id_table: 3
       27409:20240206:181918.011 Audit entry existing id: 29241, id_table: 3
       27409:20240206:181918.011 Audit entry existing id: 31299, id_table: 3
       27409:20240206:181918.011 Audit entry existing id: 29172, id_table: 3
       27409:20240206:181918.012 Audit entry existing id: 29189, id_table: 3
       27409:20240206:181918.012 Audit entry existing id: 29161, id_table: 3
       27409:20240206:181918.012 Audit entry existing id: 29240, id_table: 3
       27409:20240206:181918.012 Audit entry existing id: 20654, id_table: 4
       27409:20240206:181918.012 Audit entry existing id: 20642, id_table: 4
       27409:20240206:181918.012 Audit entry existing id: 29162, id_table: 3
       27409:20240206:181918.012 Audit entry existing id: 20657, id_table: 4
       27409:20240206:181918.012 Audit entry existing id: 29165, id_table: 3
       27409:20240206:181918.012 Audit entry existing id: 29249, id_table: 3
       27409:20240206:181918.012 Audit entry existing id: 31302, id_table: 3
       27409:20240206:181918.012 Audit entry existing id: 29247, id_table: 3
       27409:20240206:181918.012 Audit entry existing id: 20640, id_table: 4
       27409:20240206:181918.012 Audit entry existing id: 20647, id_table: 4
       27409:20240206:181918.012 Audit entry existing id: 20658, id_table: 4
       27409:20240206:181918.012 Audit entry existing id: 29191, id_table: 3
       27409:20240206:181918.012 Audit entry existing id: 29188, id_table: 3
       27409:20240206:181918.012 Audit entry existing id: 31283, id_table: 3
       27409:20240206:181918.012 Audit entry existing id: 23327, id_table: 3
       27409:20240206:181918.012 Audit entry existing id: 29180, id_table: 3
       27409:20240206:181918.012 Audit entry existing id: 29164, id_table: 3
       27409:20240206:181918.012 Audit entry existing id: 29176, id_table: 3
       27409:20240206:181918.012 Audit entry existing id: 20659, id_table: 4
       27409:20240206:181918.012 Audit entry existing id: 29238, id_table: 3
       27409:20240206:181918.012 Audit entry existing id: 29193, id_table: 3
       27409:20240206:181918.012 Audit entry existing id: 29170, id_table: 3
       27409:20240206:181918.012 Audit entry existing id: 29244, id_table: 3
       27409:20240206:181918.012 Audit entry existing id: 23288, id_table: 3
       27409:20240206:181918.012 Audit entry existing id: 29248, id_table: 3
       27409:20240206:181918.012 Audit entry existing id: 20648, id_table: 4
       27409:20240206:181918.012 Audit entry existing id: 29186, id_table: 3
       27409:20240206:181918.013 Audit entry existing id: 37105, id_table: 3
       27409:20240206:181918.013 Audit entry existing id: 29178, id_table: 3
       27409:20240206:181918.013 Audit entry existing id: 31301, id_table: 3
       27409:20240206:181918.013 Audit entry existing id: 20639, id_table: 4
       27409:20240206:181918.013 Audit entry existing id: 38706, id_table: 3
       27409:20240206:181918.013 Audit entry existing id: 20645, id_table: 4
       27409:20240206:181918.013 Audit entry existing id: 29175, id_table: 3
       27409:20240206:181918.013 Audit entry existing id: 10084, id_table: 1
       27409:20240206:181918.013 Audit entry existing id: 29163, id_table: 3
       27409:20240206:181918.013 Audit entry existing id: 29173, id_table: 3
       27409:20240206:181918.013 Audit entry existing id: 31284, id_table: 3
       27409:20240206:181918.013 Audit entry existing id: 20650, id_table: 4
       27409:20240206:181918.013 Audit entry existing id: 29190, id_table: 3
       27409:20240206:181918.013 Audit entry existing id: 20641, id_table: 4
       27409:20240206:181918.013 Audit entry existing id: 20644, id_table: 4
       27409:20240206:181918.013 Audit entry existing id: 31297, id_table: 3
       27409:20240206:181918.013 Audit entry existing id: 20656, id_table: 4
       27409:20240206:181918.013 Audit entry existing id: 31275, id_table: 3
       27409:20240206:181918.013 Audit entry existing id: 29557, id_table: 3
       27409:20240206:181918.013 Audit entry existing id: 20655, id_table: 4
       27409:20240206:181918.013 Audit entry existing id: 29250, id_table: 3
       27409:20240206:181918.013 Audit entry existing id: 29209, id_table: 3
       27409:20240206:181918.013 Audit entry existing id: 29167, id_table: 3
       27409:20240206:181918.013 Audit entry existing id: 31300, id_table: 3
       27409:20240206:181918.013 Audit entry existing id: 29174, id_table: 3
       27409:20240206:181918.013 Audit entry existing id: 29187, id_table: 3
       27409:20240206:181918.013 Audit entry existing id: 20652, id_table: 4
       27409:20240206:181918.013 Audit entry existing id: 29200, id_table: 3
       27409:20240206:181918.013 Audit entry existing id: 29168, id_table: 3
       27409:20240206:181918.013 Audit entry existing id: 29182, id_table: 3
       27409:20240206:181918.013 Audit entry existing id: 29239, id_table: 3
       27409:20240206:181918.013 Audit entry existing id: 29195, id_table: 3
       27409:20240206:181918.013 Audit entry existing id: 29207, id_table: 3
       27409:20240206:181918.013 Audit entry existing id: 29245, id_table: 3
       27409:20240206:181918.013 Audit entry existing id: 29185, id_table: 3
       27409:20240206:181918.013 Audit entry existing id: 20643, id_table: 4
       27409:20240206:181918.013 Audit entry existing id: 29197, id_table: 3
       27409:20240206:181918.013 Audit entry existing id: 29169, id_table: 3
       27409:20240206:181918.013 Audit entry existing id: 31298, id_table: 3
       27409:20240206:181918.013 Audit entry existing id: 29192, id_table: 3
       27409:20240206:181918.013 Audit entry existing id: 31282, id_table: 3
       27409:20240206:181918.013 Audit entry existing id: 29181, id_table: 3
       27409:20240206:181918.014 Audit entry existing id: 29246, id_table: 3
       27409:20240206:181918.014 Audit entry existing id: 20661, id_table: 4
       27409:20240206:181918.014 Audit entry existing id: 20651, id_table: 4
       27409:20240206:181918.014 Audit entry existing id: 31272, id_table: 3
       27409:20240206:181918.014 Audit entry existing id: 29251, id_table: 3
       27409:20240206:181918.014 Audit entry existing id: 29166, id_table: 3
       27409:20240206:181918.014 Audit entry existing id: 23287, id_table: 3
       27409:20240206:181918.014 Audit entry existing id: 29177, id_table: 3
       27409:20240206:181918.014 Audit entry existing id: 20646, id_table: 4
       27409:20240206:181918.014 Audit entry existing id: 29184, id_table: 3
       27409:20240206:181918.014 Audit entry existing id: 20660, id_table: 4
       27409:20240206:181918.014 Audit entry existing id: 20653, id_table: 4
       27409:20240206:181918.014 Audit entry existing id: 20649, id_table: 4
       27409:20240206:181918.014 Audit entry existing id: 29196, id_table: 3
       27409:20240206:181918.014 Audit entry existing id: 29179, id_table: 3
      zabbix_server [27409]: ERROR [file and function: <audit.c,zbx_audit_update_json_append_no_value>, revision:623853f8d6f, line:538] Something impossible has just happened.
       27409:20240206:181918.014 === Backtrace: ===
       27409:20240206:181918.015 23: /usr/sbin/zabbix_server: trapper #3 [processing data](zbx_backtrace+0x3c) [0x5555fef7ea7c]
       27409:20240206:181918.016 22: /usr/sbin/zabbix_server: trapper #3 [processing data](zbx_audit_update_json_append_no_value+0x125) [0x5555fef6a275]
       27409:20240206:181918.016 21: /usr/sbin/zabbix_server: trapper #3 [processing data](zbx_audit_trigger_update_json_remove_dependency+0x59) [0x5555fef64d69]
       27409:20240206:181918.016 20: /usr/sbin/zabbix_server: trapper #3 [processing data](DBsync_template_dependencies_for_triggers+0x9be) [0x5555feec328e]
       27409:20240206:181918.016 19: /usr/sbin/zabbix_server: trapper #3 [processing data](DBcopy_template_triggers+0x176a) [0x5555feeccf5a]
       27409:20240206:181918.016 18: /usr/sbin/zabbix_server: trapper #3 [processing data](DBcopy_template_elements+0xf5f) [0x5555feef3ebf]
       27409:20240206:181918.016 17: /usr/sbin/zabbix_server: trapper #3 [processing data](op_template_add+0x80) [0x5555ff006bc0]
       27409:20240206:181918.016 16: /usr/sbin/zabbix_server: trapper #3 [processing data](process_actions+0x815) [0x5555ff00f355]
       27409:20240206:181918.016 15: /usr/sbin/zabbix_server: trapper #3 [processing data](+0x1e1cfa) [0x5555ff00fcfa]
       27409:20240206:181918.016 14: /usr/sbin/zabbix_server: trapper #3 [processing data](zbx_process_events+0x223) [0x5555ff0107c3]
       27409:20240206:181918.016 13: /usr/sbin/zabbix_server: trapper #3 [processing data](DBregister_host_flush+0x502) [0x5555feefc172]
       27409:20240206:181918.016 12: /usr/sbin/zabbix_server: trapper #3 [processing data](DBregister_host+0x78) [0x5555feefc6a8]
       27409:20240206:181918.016 11: /usr/sbin/zabbix_server: trapper #3 [processing data](+0x288ccb) [0x5555ff0b6ccb]
       27409:20240206:181918.016 10: /usr/sbin/zabbix_server: trapper #3 [processing data](+0x288e76) [0x5555ff0b6e76]
       27409:20240206:181918.016 9: /usr/sbin/zabbix_server: trapper #3 [processing data](send_list_of_active_checks_json+0x1ed) [0x5555ff0b75ad]
       27409:20240206:181918.016 8: /usr/sbin/zabbix_server: trapper #3 [processing data](+0x28ba3b) [0x5555ff0b9a3b]
       27409:20240206:181918.016 7: /usr/sbin/zabbix_server: trapper #3 [processing data](trapper_thread+0x2cf) [0x5555ff0bae9f]
       27409:20240206:181918.016 6: /usr/sbin/zabbix_server: trapper #3 [processing data](zbx_thread_start+0x27) [0x5555fef76797]
       27409:20240206:181918.016 5: /usr/sbin/zabbix_server: trapper #3 [processing data](+0x797f4) [0x5555feea77f4]
       27409:20240206:181918.016 4: /usr/sbin/zabbix_server: trapper #3 [processing data](MAIN_ZABBIX_ENTRY+0xa43) [0x5555ff0e3193]
       27409:20240206:181918.016 3: /usr/sbin/zabbix_server: trapper #3 [processing data](daemon_start+0x228) [0x5555fef80ce8]
       27409:20240206:181918.016 2: /usr/sbin/zabbix_server: trapper #3 [processing data](main+0x4a9) [0x5555fee99209]
       27409:20240206:181918.016 1: /lib64/libc.so.6(__libc_start_main+0xe5) [0x7fe8b91557e5]
       27409:20240206:181918.016 0: /usr/sbin/zabbix_server: trapper #3 [processing data](_start+0x2e) [0x5555fee9ffee]
       27369:20240206:181918.022 One child process died (PID:27409,exitcode/signal:1). Exiting ...
       27369:20240206:181918.023 PROCESS EXIT: 27409
       27375:20240206:181918.023 HA manager has been paused
      zabbix_server [27369]: Error waiting for process with PID 27409: [10] No child processes
       27375:20240206:181918.110 HA manager has been stopped
       27369:20240206:181918.155 syncing history data...
       27369:20240206:181918.164 syncing history data... 100.000000%
       27369:20240206:181918.164 syncing history data done
       27369:20240206:181918.164 syncing trend data...
       27369:20240206:181918.247 syncing trend data done
       27369:20240206:181918.249 Zabbix Server stopped. Zabbix 6.0.26 (revision 623853f8d6f).
      

      This is hard to reproduce, but once it starts happening it crashes al the time.
      Workaround is to disable audit log. After some time audit log can be enabled again and then it keeps working.

      Will attach more details in comments.

            arimdjonoks Artjoms Rimdjonoks
            zux Edgars Melveris
            Team C
            Votes:
            2 Vote for this issue
            Watchers:
            9 Start watching this issue

              Created:
              Updated:
              Resolved: