-
Problem report
-
Resolution: Fixed
-
Major
-
6.0.26, 6.4.13rc1, 7.0.0beta2
-
S24-W6/7, S24-W8/9, S24-W10/11
-
1
Zabbix server can sometimes crash with following log:
27369:20240206:181828.136 Starting Zabbix Server. Zabbix 6.0.26 (revision 623853f8d6f). 27369:20240206:181828.136 ****** Enabled features ****** 27369:20240206:181828.136 SNMP monitoring: YES 27369:20240206:181828.136 IPMI monitoring: YES 27369:20240206:181828.136 Web monitoring: YES 27369:20240206:181828.136 VMware monitoring: YES 27369:20240206:181828.136 SMTP authentication: YES 27369:20240206:181828.136 ODBC: YES 27369:20240206:181828.136 SSH support: YES 27369:20240206:181828.136 IPv6 support: YES 27369:20240206:181828.136 TLS support: YES 27369:20240206:181828.136 ****************************** 27369:20240206:181828.136 using configuration file: /etc/zabbix/zabbix_server.conf 27369:20240206:181828.438 current database version (mandatory/optional): 06000000/06000044 27369:20240206:181828.438 required mandatory version: 06000000 27375:20240206:181828.589 starting HA manager 27375:20240206:181828.628 HA manager started in active mode 27369:20240206:181828.629 server #0 started [main process] 27376:20240206:181828.631 server #1 started [service manager #1] 27377:20240206:181828.632 server #2 started [configuration syncer #1] 27377:20240206:181828.808 cannot get secrets for path "zabbix/macros/training": "VaultToken" configuration parameter or "VAULT_TOKEN" environment variable should be defined 27378:20240206:181828.853 server #3 started [alert manager #1] 27379:20240206:181828.854 server #4 started [alerter #1] 27380:20240206:181828.854 server #5 started [alerter #2] 27381:20240206:181828.855 server #6 started [alerter #3] 27382:20240206:181828.855 server #7 started [preprocessing manager #1] 27383:20240206:181828.856 server #8 started [preprocessing worker #1] 27384:20240206:181828.857 server #9 started [preprocessing worker #2] 27385:20240206:181828.857 server #10 started [preprocessing worker #3] 27386:20240206:181828.857 server #11 started [lld manager #1] 27387:20240206:181828.858 server #12 started [lld worker #1] 27388:20240206:181828.859 server #13 started [lld worker #2] 27389:20240206:181828.860 server #14 started [housekeeper #1] 27390:20240206:181828.860 server #15 started [timer #1] 27391:20240206:181828.861 server #16 started [http poller #1] 27392:20240206:181828.884 server #17 started [discoverer #1] 27393:20240206:181828.889 server #18 started [history syncer #1] 27394:20240206:181828.892 server #19 started [history syncer #2] 27395:20240206:181828.893 server #20 started [history syncer #3] 27396:20240206:181828.896 server #21 started [history syncer #4] 27397:20240206:181828.899 server #22 started [escalator #1] 27398:20240206:181828.902 server #23 started [proxy poller #1] 27399:20240206:181828.904 server #24 started [self-monitoring #1] 27400:20240206:181828.905 server #25 started [task manager #1] 27401:20240206:181828.905 server #26 started [poller #1] 27402:20240206:181828.936 server #27 started [poller #2] 27403:20240206:181828.943 server #28 started [poller #3] 27404:20240206:181828.946 server #29 started [poller #4] 27405:20240206:181828.986 server #30 started [poller #5] 27406:20240206:181828.991 server #31 started [unreachable poller #1] 27407:20240206:181828.993 server #32 started [trapper #1] 27408:20240206:181828.995 server #33 started [trapper #2] 27409:20240206:181828.997 server #34 started [trapper #3] 27410:20240206:181828.999 server #35 started [trapper #4] 27411:20240206:181829.001 server #36 started [trapper #5] 27412:20240206:181829.004 server #37 started [icmp pinger #1] 27413:20240206:181829.005 server #38 started [alert syncer #1] 27414:20240206:181829.005 server #39 started [history poller #1] 27415:20240206:181829.008 server #40 started [history poller #2] 27416:20240206:181829.010 server #41 started [history poller #3] 27417:20240206:181829.014 server #42 started [history poller #4] 27418:20240206:181829.029 server #43 started [history poller #5] 27419:20240206:181829.031 server #44 started [availability manager #1] 27420:20240206:181829.032 server #45 started [trigger housekeeper #1] 27421:20240206:181829.032 server #46 started [odbc poller #1] 27407:20240206:181837.345 cannot get secrets for path "zabbix/macros/training": "VaultToken" configuration parameter or "VAULT_TOKEN" environment variable should be defined 27407:20240206:181837.347 sending configuration data to proxy "Training proxy active" at "127.0.0.1", datalen 40252, bytes 6256 with compression ratio 6.4 27398:20240206:181903.043 sending configuration data to proxy "Training proxy passive" at "143.110.187.140", datalen 90069, bytes 11998 with compression ratio 7.5 27409:20240206:181918.011 Failed to find audit entry: 19954, id_table: 4, key: triggerprototype.dependencies[55418] 27409:20240206:181918.011 Audit entries: 27409:20240206:181918.011 Audit entry existing id: 29171, id_table: 3 27409:20240206:181918.011 Audit entry existing id: 29241, id_table: 3 27409:20240206:181918.011 Audit entry existing id: 31299, id_table: 3 27409:20240206:181918.011 Audit entry existing id: 29172, id_table: 3 27409:20240206:181918.012 Audit entry existing id: 29189, id_table: 3 27409:20240206:181918.012 Audit entry existing id: 29161, id_table: 3 27409:20240206:181918.012 Audit entry existing id: 29240, id_table: 3 27409:20240206:181918.012 Audit entry existing id: 20654, id_table: 4 27409:20240206:181918.012 Audit entry existing id: 20642, id_table: 4 27409:20240206:181918.012 Audit entry existing id: 29162, id_table: 3 27409:20240206:181918.012 Audit entry existing id: 20657, id_table: 4 27409:20240206:181918.012 Audit entry existing id: 29165, id_table: 3 27409:20240206:181918.012 Audit entry existing id: 29249, id_table: 3 27409:20240206:181918.012 Audit entry existing id: 31302, id_table: 3 27409:20240206:181918.012 Audit entry existing id: 29247, id_table: 3 27409:20240206:181918.012 Audit entry existing id: 20640, id_table: 4 27409:20240206:181918.012 Audit entry existing id: 20647, id_table: 4 27409:20240206:181918.012 Audit entry existing id: 20658, id_table: 4 27409:20240206:181918.012 Audit entry existing id: 29191, id_table: 3 27409:20240206:181918.012 Audit entry existing id: 29188, id_table: 3 27409:20240206:181918.012 Audit entry existing id: 31283, id_table: 3 27409:20240206:181918.012 Audit entry existing id: 23327, id_table: 3 27409:20240206:181918.012 Audit entry existing id: 29180, id_table: 3 27409:20240206:181918.012 Audit entry existing id: 29164, id_table: 3 27409:20240206:181918.012 Audit entry existing id: 29176, id_table: 3 27409:20240206:181918.012 Audit entry existing id: 20659, id_table: 4 27409:20240206:181918.012 Audit entry existing id: 29238, id_table: 3 27409:20240206:181918.012 Audit entry existing id: 29193, id_table: 3 27409:20240206:181918.012 Audit entry existing id: 29170, id_table: 3 27409:20240206:181918.012 Audit entry existing id: 29244, id_table: 3 27409:20240206:181918.012 Audit entry existing id: 23288, id_table: 3 27409:20240206:181918.012 Audit entry existing id: 29248, id_table: 3 27409:20240206:181918.012 Audit entry existing id: 20648, id_table: 4 27409:20240206:181918.012 Audit entry existing id: 29186, id_table: 3 27409:20240206:181918.013 Audit entry existing id: 37105, id_table: 3 27409:20240206:181918.013 Audit entry existing id: 29178, id_table: 3 27409:20240206:181918.013 Audit entry existing id: 31301, id_table: 3 27409:20240206:181918.013 Audit entry existing id: 20639, id_table: 4 27409:20240206:181918.013 Audit entry existing id: 38706, id_table: 3 27409:20240206:181918.013 Audit entry existing id: 20645, id_table: 4 27409:20240206:181918.013 Audit entry existing id: 29175, id_table: 3 27409:20240206:181918.013 Audit entry existing id: 10084, id_table: 1 27409:20240206:181918.013 Audit entry existing id: 29163, id_table: 3 27409:20240206:181918.013 Audit entry existing id: 29173, id_table: 3 27409:20240206:181918.013 Audit entry existing id: 31284, id_table: 3 27409:20240206:181918.013 Audit entry existing id: 20650, id_table: 4 27409:20240206:181918.013 Audit entry existing id: 29190, id_table: 3 27409:20240206:181918.013 Audit entry existing id: 20641, id_table: 4 27409:20240206:181918.013 Audit entry existing id: 20644, id_table: 4 27409:20240206:181918.013 Audit entry existing id: 31297, id_table: 3 27409:20240206:181918.013 Audit entry existing id: 20656, id_table: 4 27409:20240206:181918.013 Audit entry existing id: 31275, id_table: 3 27409:20240206:181918.013 Audit entry existing id: 29557, id_table: 3 27409:20240206:181918.013 Audit entry existing id: 20655, id_table: 4 27409:20240206:181918.013 Audit entry existing id: 29250, id_table: 3 27409:20240206:181918.013 Audit entry existing id: 29209, id_table: 3 27409:20240206:181918.013 Audit entry existing id: 29167, id_table: 3 27409:20240206:181918.013 Audit entry existing id: 31300, id_table: 3 27409:20240206:181918.013 Audit entry existing id: 29174, id_table: 3 27409:20240206:181918.013 Audit entry existing id: 29187, id_table: 3 27409:20240206:181918.013 Audit entry existing id: 20652, id_table: 4 27409:20240206:181918.013 Audit entry existing id: 29200, id_table: 3 27409:20240206:181918.013 Audit entry existing id: 29168, id_table: 3 27409:20240206:181918.013 Audit entry existing id: 29182, id_table: 3 27409:20240206:181918.013 Audit entry existing id: 29239, id_table: 3 27409:20240206:181918.013 Audit entry existing id: 29195, id_table: 3 27409:20240206:181918.013 Audit entry existing id: 29207, id_table: 3 27409:20240206:181918.013 Audit entry existing id: 29245, id_table: 3 27409:20240206:181918.013 Audit entry existing id: 29185, id_table: 3 27409:20240206:181918.013 Audit entry existing id: 20643, id_table: 4 27409:20240206:181918.013 Audit entry existing id: 29197, id_table: 3 27409:20240206:181918.013 Audit entry existing id: 29169, id_table: 3 27409:20240206:181918.013 Audit entry existing id: 31298, id_table: 3 27409:20240206:181918.013 Audit entry existing id: 29192, id_table: 3 27409:20240206:181918.013 Audit entry existing id: 31282, id_table: 3 27409:20240206:181918.013 Audit entry existing id: 29181, id_table: 3 27409:20240206:181918.014 Audit entry existing id: 29246, id_table: 3 27409:20240206:181918.014 Audit entry existing id: 20661, id_table: 4 27409:20240206:181918.014 Audit entry existing id: 20651, id_table: 4 27409:20240206:181918.014 Audit entry existing id: 31272, id_table: 3 27409:20240206:181918.014 Audit entry existing id: 29251, id_table: 3 27409:20240206:181918.014 Audit entry existing id: 29166, id_table: 3 27409:20240206:181918.014 Audit entry existing id: 23287, id_table: 3 27409:20240206:181918.014 Audit entry existing id: 29177, id_table: 3 27409:20240206:181918.014 Audit entry existing id: 20646, id_table: 4 27409:20240206:181918.014 Audit entry existing id: 29184, id_table: 3 27409:20240206:181918.014 Audit entry existing id: 20660, id_table: 4 27409:20240206:181918.014 Audit entry existing id: 20653, id_table: 4 27409:20240206:181918.014 Audit entry existing id: 20649, id_table: 4 27409:20240206:181918.014 Audit entry existing id: 29196, id_table: 3 27409:20240206:181918.014 Audit entry existing id: 29179, id_table: 3 zabbix_server [27409]: ERROR [file and function: <audit.c,zbx_audit_update_json_append_no_value>, revision:623853f8d6f, line:538] Something impossible has just happened. 27409:20240206:181918.014 === Backtrace: === 27409:20240206:181918.015 23: /usr/sbin/zabbix_server: trapper #3 [processing data](zbx_backtrace+0x3c) [0x5555fef7ea7c] 27409:20240206:181918.016 22: /usr/sbin/zabbix_server: trapper #3 [processing data](zbx_audit_update_json_append_no_value+0x125) [0x5555fef6a275] 27409:20240206:181918.016 21: /usr/sbin/zabbix_server: trapper #3 [processing data](zbx_audit_trigger_update_json_remove_dependency+0x59) [0x5555fef64d69] 27409:20240206:181918.016 20: /usr/sbin/zabbix_server: trapper #3 [processing data](DBsync_template_dependencies_for_triggers+0x9be) [0x5555feec328e] 27409:20240206:181918.016 19: /usr/sbin/zabbix_server: trapper #3 [processing data](DBcopy_template_triggers+0x176a) [0x5555feeccf5a] 27409:20240206:181918.016 18: /usr/sbin/zabbix_server: trapper #3 [processing data](DBcopy_template_elements+0xf5f) [0x5555feef3ebf] 27409:20240206:181918.016 17: /usr/sbin/zabbix_server: trapper #3 [processing data](op_template_add+0x80) [0x5555ff006bc0] 27409:20240206:181918.016 16: /usr/sbin/zabbix_server: trapper #3 [processing data](process_actions+0x815) [0x5555ff00f355] 27409:20240206:181918.016 15: /usr/sbin/zabbix_server: trapper #3 [processing data](+0x1e1cfa) [0x5555ff00fcfa] 27409:20240206:181918.016 14: /usr/sbin/zabbix_server: trapper #3 [processing data](zbx_process_events+0x223) [0x5555ff0107c3] 27409:20240206:181918.016 13: /usr/sbin/zabbix_server: trapper #3 [processing data](DBregister_host_flush+0x502) [0x5555feefc172] 27409:20240206:181918.016 12: /usr/sbin/zabbix_server: trapper #3 [processing data](DBregister_host+0x78) [0x5555feefc6a8] 27409:20240206:181918.016 11: /usr/sbin/zabbix_server: trapper #3 [processing data](+0x288ccb) [0x5555ff0b6ccb] 27409:20240206:181918.016 10: /usr/sbin/zabbix_server: trapper #3 [processing data](+0x288e76) [0x5555ff0b6e76] 27409:20240206:181918.016 9: /usr/sbin/zabbix_server: trapper #3 [processing data](send_list_of_active_checks_json+0x1ed) [0x5555ff0b75ad] 27409:20240206:181918.016 8: /usr/sbin/zabbix_server: trapper #3 [processing data](+0x28ba3b) [0x5555ff0b9a3b] 27409:20240206:181918.016 7: /usr/sbin/zabbix_server: trapper #3 [processing data](trapper_thread+0x2cf) [0x5555ff0bae9f] 27409:20240206:181918.016 6: /usr/sbin/zabbix_server: trapper #3 [processing data](zbx_thread_start+0x27) [0x5555fef76797] 27409:20240206:181918.016 5: /usr/sbin/zabbix_server: trapper #3 [processing data](+0x797f4) [0x5555feea77f4] 27409:20240206:181918.016 4: /usr/sbin/zabbix_server: trapper #3 [processing data](MAIN_ZABBIX_ENTRY+0xa43) [0x5555ff0e3193] 27409:20240206:181918.016 3: /usr/sbin/zabbix_server: trapper #3 [processing data](daemon_start+0x228) [0x5555fef80ce8] 27409:20240206:181918.016 2: /usr/sbin/zabbix_server: trapper #3 [processing data](main+0x4a9) [0x5555fee99209] 27409:20240206:181918.016 1: /lib64/libc.so.6(__libc_start_main+0xe5) [0x7fe8b91557e5] 27409:20240206:181918.016 0: /usr/sbin/zabbix_server: trapper #3 [processing data](_start+0x2e) [0x5555fee9ffee] 27369:20240206:181918.022 One child process died (PID:27409,exitcode/signal:1). Exiting ... 27369:20240206:181918.023 PROCESS EXIT: 27409 27375:20240206:181918.023 HA manager has been paused zabbix_server [27369]: Error waiting for process with PID 27409: [10] No child processes 27375:20240206:181918.110 HA manager has been stopped 27369:20240206:181918.155 syncing history data... 27369:20240206:181918.164 syncing history data... 100.000000% 27369:20240206:181918.164 syncing history data done 27369:20240206:181918.164 syncing trend data... 27369:20240206:181918.247 syncing trend data done 27369:20240206:181918.249 Zabbix Server stopped. Zabbix 6.0.26 (revision 623853f8d6f).
This is hard to reproduce, but once it starts happening it crashes al the time.
Workaround is to disable audit log. After some time audit log can be enabled again and then it keeps working.
Will attach more details in comments.