[ZBX-19445] Zabbix proxy and something go away Created: 2021 May 24 Updated: 2021 Jun 11 Resolved: 2021 Jun 11 |
|
Status: | Closed |
Project: | ZABBIX BUGS AND ISSUES |
Component/s: | Proxy (P), Server (S) |
Affects Version/s: | 5.4.0 |
Fix Version/s: | None |
Type: | Problem report | Priority: | Trivial |
Reporter: | CHAVIGNY | Assignee: | Alexey Pustovalov |
Resolution: | Won't Do | Votes: | 1 |
Labels: | crash | ||
Remaining Estimate: | Not Specified | ||
Time Spent: | Not Specified | ||
Original Estimate: | Not Specified | ||
Environment: |
CentOS Linux release 8.3.2011 |
Attachments: |
![]() ![]() ![]() ![]() ![]() ![]() |
Description |
Steps to reproduce:
Result: Zabbix proxy randomly crash and not responding. I have approximately ten proxy. I notice this problem since update 5.4 only on the most loaded proxies. zabbix_proxy.log is in attachment. ps aux | grep defunct zabbix 1134938 0.0 0.0 0 0 ? Z 10:15 0:00 [zabbix_proxy] <defunct> For the first time, I have constated similar issue on Zabbix server, without problem logged in log, except journalctl : -- Unit zabbix-server.service has finished shutting down. May 22 00:19:25 <HOSTNAME> systemd[1]: Starting Zabbix Server... -- Subject: Unit zabbix-server.service has begun start-up -- Defined-By: systemd -- Support: https://access.redhat.com/support -- -- Unit zabbix-server.service has begun starting up. May 22 00:19:25 <HOSTNAME> zabbix_server[1711003]: /usr/sbin/zabbix_server: /usr/pgsql-11/lib/libpq.so.5: no version information available (required by /usr/sbin/zabb> May 22 00:19:25 <HOSTNAME> systemd[1]: zabbix-server.service: Can't open PID file /run/zabbix/zabbix_server.pid (yet?) after start: No such file or directory May 22 00:19:25 <HOSTNAME> systemd[1]: Started Zabbix Server. -- Subject: Unit zabbix-server.service has finished start-up -- Defined-By: systemd -- Support: https://access.redhat.com/support -- -- Unit zabbix-server.service has finished starting up. -- Expected: |
Comments |
Comment by CHAVIGNY [ 2021 May 24 ] |
Example for Zabbix Server :
2845786:20210524:143316.358 One child process died (PID:2845841,exitcode/signal:6). Exiting ...
zabbix_server [2845786]: Error waiting for process with PID 2845841: [10] No child processes
2845786:20210524:143316.504 syncing history data...
2845786:20210524:143317.063 syncing history data... 100.000000%
2845786:20210524:143317.063 syncing history data done
2845786:20210524:143317.063 syncing trend data...
2845786:20210524:143327.833 syncing trend data done
2845786:20210524:143327.848 Zabbix Server stopped. Zabbix 5.4.0 (revision 5059e20a75).
2850378:20210524:143338.084 Starting Zabbix Server. Zabbix 5.4.0 (revision 5059e20a75).
2850378:20210524:143338.084 ****** Enabled features ******
2850378:20210524:143338.084 SNMP monitoring: YES
2850378:20210524:143338.084 IPMI monitoring: YES
2850378:20210524:143338.084 Web monitoring: YES
2850378:20210524:143338.084 VMware monitoring: YES
2850378:20210524:143338.084 SMTP authentication: YES
2850378:20210524:143338.084 ODBC: YES
2850378:20210524:143338.084 SSH support: YES
2850378:20210524:143338.084 IPv6 support: YES
2850378:20210524:143338.084 TLS support: YES
2850378:20210524:143338.084 ******************************
2850378:20210524:143338.084 using configuration file: /etc/zabbix/zabbix_server.conf
NB : In this example, Zabbix has successfully restarted by him self, journalctl say : -- -- Unit zabbix-server.service has finished starting up. -- -- The start-up result is done. May 24 14:33:27 <HOSTNAME> kill[2850346]: kill: not enough arguments May 24 14:33:27 <HOSTNAME> systemd[1]: zabbix-server.service: Control process exited, code=exited status=1 May 24 14:33:27 <HOSTNAME> systemd[1]: zabbix-server.service: Failed with result 'exit-code'. -- Subject: Unit failed -- Defined-By: systemd -- Support: https://access.redhat.com/support -- -- The unit zabbix-server.service has entered the 'failed' state with result 'exit-code'. May 24 14:33:38 <HOSTNAME> systemd[1]: zabbix-server.service: Service RestartSec=10s expired, scheduling restart. May 24 14:33:38 <HOSTNAME> systemd[1]: zabbix-server.service: Scheduled restart job, restart counter is at 57. -- Subject: Automatic restarting of a unit has been scheduled -- Defined-By: systemd -- Support: https://access.redhat.com/support -- -- Automatic restarting of the unit zabbix-server.service has been scheduled, as the result for -- the configured Restart= setting for the unit. May 24 14:33:38 <HOSTNAME> systemd[1]: Stopped Zabbix Server. -- Subject: Unit zabbix-server.service has finished shutting down -- Defined-By: systemd -- Support: https://access.redhat.com/support -- -- Unit zabbix-server.service has finished shutting down. May 24 14:33:38 <HOSTNAME> systemd[1]: Starting Zabbix Server... -- Subject: Unit zabbix-server.service has begun start-up -- Defined-By: systemd -- Support: https://access.redhat.com/support -- -- Unit zabbix-server.service has begun starting up. May 24 14:33:38 <HOSTNAME> zabbix_server[2850376]: /usr/sbin/zabbix_server: /usr/pgsql-11/lib/libpq.so.5: no version information available (required by /usr/sbin/zabbix_server) May 24 14:33:38 <HOSTNAME> systemd[1]: zabbix-server.service: Can't open PID file /run/zabbix/zabbix_server.pid (yet?) after start: No such file or directory May 24 14:33:38 <HOSTNAME> systemd[1]: Started Zabbix Server. -- Subject: Unit zabbix-server.service has finished start-up -- Defined-By: systemd -- Support: https://access.redhat.com/support -- -- Unit zabbix-server.service has finished starting up. -- -- The start-up result is done. |
Comment by Vladislavs Sokurenko [ 2021 May 25 ] |
Thank you for your report, could you please be so kind and increase log level and then attach log again ? |
Comment by Vladislavs Sokurenko [ 2021 May 25 ] |
How often does issue happen ? Crash happens in data sender process, could increase log level only for it:
zabbix_proxy -R log_level_increase="data sender"
|
Comment by Vladislavs Sokurenko [ 2021 May 25 ] |
Backtrace for easier searching: 1132913:20210524:101541.740 Got signal [signal:11(SIGSEGV),reason:1,refaddr:(nil)]. Crashing ... 1132913:20210524:101541.740 ====== Fatal information: ====== 1132913:20210524:101541.740 Program counter: 0x55c5ff626965 1132913:20210524:101541.740 === Registers: === 1132913:20210524:101541.740 r8 = 0 = 0 = 0 1132913:20210524:101541.740 r9 = 6 = 6 = 6 1132913:20210524:101541.740 r10 = 0 = 0 = 0 1132913:20210524:101541.740 r11 = 7f941a357dc0 = 140274071600576 = 140274071600576 1132913:20210524:101541.740 r12 = 1 = 1 = 1 1132913:20210524:101541.740 r13 = 55c5ff7cf863 = 94308883298403 = 94308883298403 1132913:20210524:101541.740 r14 = 0 = 0 = 0 1132913:20210524:101541.740 r15 = 0 = 0 = 0 1132913:20210524:101541.740 rdi = 0 = 0 = 0 1132913:20210524:101541.740 rsi = 7ffde8e066ec = 140728510473964 = 140728510473964 1132913:20210524:101541.740 rbp = 7ffde8e047e0 = 140728510466016 = 140728510466016 1132913:20210524:101541.740 rbx = 1 = 1 = 1 1132913:20210524:101541.740 rdx = 7ffde8e066ec = 140728510473964 = 140728510473964 1132913:20210524:101541.740 rax = 0 = 0 = 0 1132913:20210524:101541.740 rcx = 6 = 6 = 6 1132913:20210524:101541.740 rsp = 7ffde8e03fc0 = 140728510463936 = 140728510463936 1132913:20210524:101541.740 rip = 55c5ff626965 = 94308881557861 = 94308881557861 1132913:20210524:101541.740 efl = 10206 = 66054 = 66054 1132913:20210524:101541.740 csgsfs = 2b000000000033 = 12103423998558259 = 12103423998558259 1132913:20210524:101541.740 err = 4 = 4 = 4 1132913:20210524:101541.740 trapno = e = 14 = 14 1132913:20210524:101541.740 oldmask = 0 = 0 = 0 1132913:20210524:101541.740 cr2 = 0 = 0 = 0 1132913:20210524:101541.740 === Backtrace: === 1132913:20210524:101541.742 12: /usr/sbin/zabbix_proxy: data sender [sent 0 values in 36.164480 sec, sending data](zbx_backtrace+0x3f) [0x55c5ff743bbf] 1132913:20210524:101541.742 11: /usr/sbin/zabbix_proxy: data sender [sent 0 values in 36.164480 sec, sending data](zbx_log_fatal_info+0x141) [0x55c5ff743e1c] 1132913:20210524:101541.742 10: /usr/sbin/zabbix_proxy: data sender [sent 0 values in 36.164480 sec, sending data](+0x19d608) [0x55c5ff744608] 1132913:20210524:101541.742 9: /lib64/libpthread.so.0(+0x12b20) [0x7f941c3e4b20] 1132913:20210524:101541.742 8: /usr/sbin/zabbix_proxy: data sender [sent 0 values in 36.164480 sec, sending data](+0x7f965) [0x55c5ff626965] 1132913:20210524:101541.742 7: /usr/sbin/zabbix_proxy: data sender [sent 0 values in 36.164480 sec, sending data](+0x7fe4e) [0x55c5ff626e4e] 1132913:20210524:101541.742 6: /usr/sbin/zabbix_proxy: data sender [sent 0 values in 36.164480 sec, sending data](datasender_thread+0x15b) [0x55c5ff627216] 1132913:20210524:101541.742 5: /usr/sbin/zabbix_proxy: data sender [sent 0 values in 36.164480 sec, sending data](zbx_thread_start+0x37) [0x55c5ff748d49] 1132913:20210524:101541.742 4: /usr/sbin/zabbix_proxy: data sender [sent 0 values in 36.164480 sec, sending data](MAIN_ZABBIX_ENTRY+0xa60) [0x55c5ff5ece7e] 1132913:20210524:101541.742 3: /usr/sbin/zabbix_proxy: data sender [sent 0 values in 36.164480 sec, sending data](daemon_start+0x2ff) [0x55c5ff7437e9] 1132913:20210524:101541.742 2: /usr/sbin/zabbix_proxy: data sender [sent 0 values in 36.164480 sec, sending data](main+0x2f1) [0x55c5ff5ec3d4] 1132913:20210524:101541.742 1: /lib64/libc.so.6(__libc_start_main+0xf3) [0x7f941a1fd7b3] 1132913:20210524:101541.742 0: /usr/sbin/zabbix_proxy: data sender [sent 0 values in 36.164480 sec, sending data](_start+0x2e) [0x55c5ff5eb27e] |
Comment by CHAVIGNY [ 2021 May 25 ] |
You can find new crash report from zabbix_server in "202105251814_zabbix_server.log" after increasing "data sender". Currently, I have no crash on Zabbix Proxy since last time. |
Comment by Vladislavs Sokurenko [ 2021 May 26 ] |
I am sorry, I meant to increase log level on proxy that crash |
Comment by CHAVIGNY [ 2021 May 27 ] |
Ok, I have two crash of zabbix proxy this night in DebugLevel 3. I set this morning to DebugLevel 4 and I have new crash with many information. I will try to explain as best as possible the unfolding of the situation. At around 10:30 a.m. The Zabbix trapper port don't accept any connection, I try in localhost : # telnet 172.31.20.173 10051 Trying 172.31.20.173... But, Zabbix Proxy running normally : # ps aux | grep zabbix_proxy zabbix 1669758 0.0 0.3 345228 6176 ? S 09:29 0:00 /usr/sbin/zabbix_proxy -c /etc/zabbix/zabbix_proxy.conf zabbix 1669759 0.0 1.9 367936 36656 ? S 09:29 0:02 /usr/sbin/zabbix_proxy: configuration syncer [loading configuration] zabbix 1669769 0.0 1.9 581948 35604 ? S 09:29 0:00 /usr/sbin/zabbix_proxy: trapper #1 [processed data in 0.001354 sec, waiting for connection] zabbix 1669770 0.0 1.8 577928 33524 ? S 09:29 0:00 /usr/sbin/zabbix_proxy: trapper #2 [processed data in 0.001677 sec, waiting for connection] zabbix 1669771 0.0 1.6 575180 31148 ? S 09:29 0:00 /usr/sbin/zabbix_proxy: trapper #3 [processed data in 0.000760 sec, waiting for connection] zabbix 1669772 0.0 1.8 577928 33680 ? S 09:29 0:00 /usr/sbin/zabbix_proxy: trapper #4 [processed data in 0.001294 sec, waiting for connection] zabbix 1669773 0.0 2.0 582124 37864 ? S 09:29 0:00 /usr/sbin/zabbix_proxy: trapper #5 [processed data in 0.000610 sec, waiting for connection] zabbix 1669774 0.0 1.6 575044 31080 ? S 09:29 0:00 /usr/sbin/zabbix_proxy: trapper #6 [processed data in 0.000673 sec, waiting for connection] zabbix 1669775 0.0 0.3 345340 7408 ? S 09:29 0:00 /usr/sbin/zabbix_proxy: heartbeat sender [sending heartbeat message success in 0.002648 sec, idle 60 sec] zabbix 1669776 0.0 2.2 374480 41956 ? S 09:29 0:05 /usr/sbin/zabbix_proxy: data sender [sent 0 values in 0.004296 sec, sending data] zabbix 1669777 0.0 0.3 348728 5876 ? S 09:29 0:00 /usr/sbin/zabbix_proxy: ipmi manager #1 [scheduled 0, polled 0 values, idle 5.005919 sec during 5.005974 sec] zabbix 1669778 0.0 0.4 347232 7448 ? S 09:29 0:00 /usr/sbin/zabbix_proxy: housekeeper [connecting to the database] zabbix 1669779 0.0 0.2 345348 5460 ? S 09:29 0:00 /usr/sbin/zabbix_proxy: http poller #1 [got 0 values in 0.000505 sec, getting values] zabbix 1669780 0.0 0.5 558860 10480 ? S 09:29 0:00 /usr/sbin/zabbix_proxy: discoverer #1 [processed 0 rules in 0.000495 sec, performing discovery] zabbix 1669781 0.0 0.8 350408 15080 ? S 09:29 0:03 /usr/sbin/zabbix_proxy: history syncer #1 [processed 0 values in 0.000062 sec, idle 1 sec] zabbix 1669782 0.0 0.7 349816 14656 ? S 09:29 0:02 /usr/sbin/zabbix_proxy: history syncer #2 [processed 0 values in 0.000179 sec, idle 1 sec] zabbix 1669783 0.0 0.7 350204 14808 ? S 09:29 0:01 /usr/sbin/zabbix_proxy: history syncer #3 [processed 0 values in 0.000099 sec, syncing history] zabbix 1669784 0.0 0.8 350648 15652 ? S 09:29 0:02 /usr/sbin/zabbix_proxy: history syncer #4 [processed 0 values in 0.000064 sec, idle 1 sec] zabbix 1669785 0.0 0.2 345228 4404 ? S 09:29 0:00 /usr/sbin/zabbix_proxy: ipmi poller #1 started zabbix 1669786 0.0 0.2 345404 5300 ? S 09:29 0:01 /usr/sbin/zabbix_proxy: snmp trapper [processed data in 0.000141 sec, idle 1 sec] zabbix 1669787 0.0 0.2 345228 4412 ? S 09:29 0:01 /usr/sbin/zabbix_proxy: self-monitoring [processed data in 0.000167 sec, idle 1 sec] zabbix 1669788 1.1 0.9 352020 17724 ? S 09:29 1:30 /usr/sbin/zabbix_proxy: vmware collector #1 [updated 0, removed 0 VMware services in 0.000065 sec, idle 5 sec] zabbix 1669789 1.1 0.9 351960 17648 ? S 09:29 1:28 /usr/sbin/zabbix_proxy: vmware collector #2 [updated 0, removed 0 VMware services in 0.000087 sec, idle 5 sec] zabbix 1669790 1.0 0.9 351980 17832 ? S 09:29 1:18 /usr/sbin/zabbix_proxy: vmware collector #3 [updated 0, removed 0 VMware services in 0.000180 sec, idle 5 sec] zabbix 1669791 0.6 0.9 352060 17828 ? S 09:29 0:46 /usr/sbin/zabbix_proxy: vmware collector #4 [updated 0, removed 0 VMware services in 0.000148 sec, idle 5 sec] zabbix 1669792 0.0 0.5 558860 10540 ? S 09:29 0:00 /usr/sbin/zabbix_proxy: task manager [processing tasks] zabbix 1669793 0.8 1.2 579352 22992 ? S 09:29 1:07 /usr/sbin/zabbix_proxy: poller #1 [got 0 values in 0.000184 sec, getting values] zabbix 1669794 0.8 1.2 579380 23920 ? S 09:29 1:01 /usr/sbin/zabbix_proxy: poller #2 [got 1 values in 0.104406 sec, getting values] zabbix 1669795 0.7 1.2 579568 22908 ? S 09:29 0:59 /usr/sbin/zabbix_proxy: poller #3 [got 0 values in 0.000179 sec, getting values] zabbix 1669796 0.9 1.3 579356 24672 ? S 09:29 1:13 /usr/sbin/zabbix_proxy: poller #4 [got 1 values in 0.083061 sec, getting values] zabbix 1669797 0.9 1.2 579364 22396 ? S 09:29 1:08 /usr/sbin/zabbix_proxy: poller #5 [got 0 values in 0.000179 sec, getting values] zabbix 1669798 0.8 1.2 579448 24152 ? S 09:29 1:04 /usr/sbin/zabbix_proxy: poller #6 [got 1 values in 0.073215 sec, getting values] zabbix 1669799 0.8 1.2 579420 23244 ? S 09:29 1:05 /usr/sbin/zabbix_proxy: poller #7 [got 0 values in 0.000183 sec, getting values] zabbix 1669800 0.7 1.2 580400 24016 ? S 09:29 0:57 /usr/sbin/zabbix_proxy: poller #8 [got 2 values in 0.073889 sec, getting values] zabbix 1669801 0.8 1.2 579356 23032 ? S 09:29 1:02 /usr/sbin/zabbix_proxy: poller #9 [got 1 values in 0.119538 sec, getting values] zabbix 1669802 0.8 1.2 579568 24060 ? S 09:29 1:06 /usr/sbin/zabbix_proxy: poller #10 [got 1 values in 0.089675 sec, getting values] zabbix 1669803 0.0 0.5 558580 10836 ? S 09:29 0:01 /usr/sbin/zabbix_proxy: unreachable poller #1 [got 0 values in 0.000268 sec, getting values] zabbix 1669804 0.0 0.5 558580 10648 ? S 09:29 0:01 /usr/sbin/zabbix_proxy: unreachable poller #2 [got 0 values in 0.000321 sec, getting values] zabbix 1669805 0.0 0.5 558580 11096 ? S 09:29 0:01 /usr/sbin/zabbix_proxy: unreachable poller #3 [got 0 values in 0.000283 sec, getting values] zabbix 1669806 0.0 0.5 558580 10880 ? S 09:29 0:01 /usr/sbin/zabbix_proxy: unreachable poller #4 [got 0 values in 0.000284 sec, getting values] zabbix 1669807 0.0 0.3 348892 6492 ? S 09:29 0:01 /usr/sbin/zabbix_proxy: icmp pinger #1 [pinging hosts] zabbix 1669808 0.0 0.3 348892 6484 ? S 09:29 0:01 /usr/sbin/zabbix_proxy: icmp pinger #2 [getting values] zabbix 1669809 0.0 0.3 348892 6504 ? S 09:29 0:01 /usr/sbin/zabbix_proxy: icmp pinger #3 [getting values] zabbix 1669810 0.0 0.3 348892 6664 ? S 09:29 0:01 /usr/sbin/zabbix_proxy: icmp pinger #4 [getting values] zabbix 1669811 0.0 0.3 348892 6484 ? S 09:29 0:01 /usr/sbin/zabbix_proxy: icmp pinger #5 [getting values] zabbix 1669812 0.0 0.3 348892 6504 ? S 09:29 0:01 /usr/sbin/zabbix_proxy: icmp pinger #6 [getting values] zabbix 1669813 0.3 0.6 347376 12452 ? S 09:29 0:23 /usr/sbin/zabbix_proxy: preprocessing manager #1 [queued 0, processed 0 values, idle 5.006462 sec during 5.006528 sec] zabbix 1669814 0.0 0.2 345340 5316 ? S 09:29 0:01 /usr/sbin/zabbix_proxy: preprocessing worker #1 started zabbix 1669815 0.0 0.2 345340 5316 ? S 09:29 0:01 /usr/sbin/zabbix_proxy: preprocessing worker #2 started zabbix 1669816 0.0 0.2 345340 5312 ? S 09:29 0:01 /usr/sbin/zabbix_proxy: preprocessing worker #3 started zabbix 1669817 0.0 0.4 345488 9128 ? S 09:29 0:00 /usr/sbin/zabbix_proxy: history poller #1 [got 1 values in 0.000577 sec, getting values] zabbix 1669818 0.0 0.2 345228 5300 ? S 09:29 0:01 /usr/sbin/zabbix_proxy: availability manager #1 [queued 0, processed 0 values, idle 5.006362 sec during 5.006433 sec] However, many ports appear to be in "CLOSE_WAIT" (202105271118_netstat.log) I try to stop Zabbix-proxy, without sucess : # (systemctl stop zabbix-proxy &) ; tail -f /var/log/zabbix/zabbix_proxy.log 1669818:20210527:114104.141 In zbx_ipc_service_recv() timeout:1 1669787:20210527:114104.383 zbx_setproctitle() title:'self-monitoring [processing data]' 1669787:20210527:114104.383 In collect_selfmon_stats() 1669787:20210527:114104.383 End of collect_selfmon_stats() 1669787:20210527:114104.383 zbx_setproctitle() title:'self-monitoring [processed data in 0.000206 sec, idle 1 sec]' 1669785:20210527:114104.443 End of zbx_ipc_async_socket_recv():0 1669786:20210527:114104.446 zbx_setproctitle() title:'snmp trapper [processing data]' 1669786:20210527:114104.446 zbx_setproctitle() title:'snmp trapper [processed data in 0.000073 sec, idle 1 sec]' 1669813:20210527:114104.460 End of zbx_ipc_service_recv():2 1669813:20210527:114104.460 In zbx_ipc_service_recv() timeout:1 1669758:20210527:114104.786 Got signal [signal:15(SIGTERM),sender_pid:1677081,sender_uid:0,reason:0]. Exiting ... 1669758:20210527:114104.786 zbx_on_exit() called 1669782:20210527:114104.786 zbx_setproctitle() title:'history syncer #2 [processed 0 values in 0.000189 sec, syncing history]' 1669782:20210527:114104.786 syncing history data in progress... 1669782:20210527:114104.786 In zbx_sync_history_cache() history_num:11 1669786:20210527:114104.787 zbx_setproctitle() title:'snmp trapper [processing data]' 1669786:20210527:114104.787 zbx_setproctitle() title:'snmp trapper [processed data in 0.009810 sec, idle 1 sec]' 1669787:20210527:114104.787 zbx_setproctitle() title:'self-monitoring [processing data]' 1669787:20210527:114104.787 In collect_selfmon_stats() 1669787:20210527:114104.787 End of collect_selfmon_stats() 1669787:20210527:114104.787 zbx_setproctitle() title:'self-monitoring [processed data in 0.010045 sec, idle 1 sec]' 1669784:20210527:114104.787 zbx_setproctitle() title:'history syncer #4 [processed 0 values in 0.000200 sec, syncing history]' 1669784:20210527:114104.787 In zbx_sync_history_cache() history_num:11 1669781:20210527:114104.787 zbx_setproctitle() title:'history syncer #1 [processed 0 values in 0.000161 sec, syncing history]' 1669781:20210527:114104.787 In zbx_sync_history_cache() history_num:11 1669818:20210527:114105.142 End of zbx_ipc_service_recv():2 1669813:20210527:114105.461 End of zbx_ipc_service_recv():2 1669813:20210527:114105.461 zbx_setproctitle() title:'preprocessing manager #1 [terminated]' 1669785:20210527:114105.787 zbx_setproctitle() title:'ipmi poller #1 [terminated]' 1669786:20210527:114105.787 zbx_setproctitle() title:'snmp trapper #1 [terminated]' 1669787:20210527:114105.787 zbx_setproctitle() title:'self-monitoring #1 [terminated]' 1669791:20210527:114106.787 zbx_setproctitle() title:'vmware collector #4 [terminated]' 1669789:20210527:114107.787 zbx_setproctitle() title:'vmware collector #2 [terminated]' 1669790:20210527:114107.787 zbx_setproctitle() title:'vmware collector #3 [terminated]' 1669788:20210527:114107.787 zbx_setproctitle() title:'vmware collector #1 [terminated]' 1669775:20210527:114145.791 zbx_setproctitle() title:'heartbeat sender #1 [terminated]' Zabbix-proxy appear "defunct" after try to stop process # ps aux | grep zabbix_proxy zabbix 1669758 0.0 0.3 345228 6176 ? S 09:29 0:00 /usr/sbin/zabbix_proxy -c /etc/zabbix/zabbix_proxy.conf zabbix 1669759 0.0 1.9 367936 36656 ? S 09:29 0:02 /usr/sbin/zabbix_proxy: configuration syncer [loading configuration] zabbix 1669769 0.0 1.9 581948 35604 ? S 09:29 0:00 /usr/sbin/zabbix_proxy: trapper #1 [processed data in 0.001354 sec, waiting for connection] zabbix 1669770 0.0 1.8 577928 33524 ? S 09:29 0:00 /usr/sbin/zabbix_proxy: trapper #2 [processed data in 0.001677 sec, waiting for connection] zabbix 1669771 0.0 1.6 575180 31148 ? S 09:29 0:00 /usr/sbin/zabbix_proxy: trapper #3 [processed data in 0.000760 sec, waiting for connection] zabbix 1669772 0.0 1.8 577928 33680 ? S 09:29 0:00 /usr/sbin/zabbix_proxy: trapper #4 [processed data in 0.001294 sec, waiting for connection] zabbix 1669773 0.0 2.0 582124 37864 ? S 09:29 0:00 /usr/sbin/zabbix_proxy: trapper #5 [processed data in 0.000610 sec, waiting for connection] zabbix 1669774 0.0 1.6 575044 31080 ? S 09:29 0:00 /usr/sbin/zabbix_proxy: trapper #6 [processed data in 0.000673 sec, waiting for connection] zabbix 1669775 0.0 0.3 345340 7408 ? S 09:29 0:00 /usr/sbin/zabbix_proxy: heartbeat sender #1 [terminated] zabbix 1669776 0.0 2.2 374480 41956 ? S 09:29 0:05 /usr/sbin/zabbix_proxy: data sender [sent 0 values in 0.004296 sec, sending data] zabbix 1669777 0.0 0.3 348728 6092 ? S 09:29 0:00 /usr/sbin/zabbix_proxy: ipmi manager #1 [scheduled 0, polled 0 values, idle 5.005919 sec during 5.005974 sec] zabbix 1669778 0.0 0.4 347232 7448 ? S 09:29 0:00 /usr/sbin/zabbix_proxy: housekeeper [connecting to the database] zabbix 1669779 0.0 0.2 345348 5460 ? S 09:29 0:00 /usr/sbin/zabbix_proxy: http poller #1 [got 0 values in 0.000505 sec, getting values] zabbix 1669780 0.0 0.5 558860 10480 ? S 09:29 0:00 /usr/sbin/zabbix_proxy: discoverer #1 [processed 0 rules in 0.000495 sec, performing discovery] zabbix 1669781 0.0 0.8 350408 15080 ? S 09:29 0:03 /usr/sbin/zabbix_proxy: history syncer #1 [processed 0 values in 0.000161 sec, syncing history] zabbix 1669782 0.0 0.7 349816 14656 ? S 09:29 0:02 /usr/sbin/zabbix_proxy: history syncer #2 [processed 0 values in 0.000189 sec, syncing history] zabbix 1669783 0.0 0.7 350204 14808 ? S 09:29 0:01 /usr/sbin/zabbix_proxy: history syncer #3 [processed 0 values in 0.000099 sec, syncing history] zabbix 1669784 0.0 0.8 350648 15652 ? S 09:29 0:02 /usr/sbin/zabbix_proxy: history syncer #4 [processed 0 values in 0.000200 sec, syncing history] zabbix 1669785 0.0 0.2 345228 4404 ? S 09:29 0:00 /usr/sbin/zabbix_proxy: ipmi poller #1 [terminated] zabbix 1669786 0.0 0.2 345340 5500 ? S 09:29 0:01 /usr/sbin/zabbix_proxy: snmp trapper #1 [terminated] zabbix 1669787 0.0 0.2 345228 4412 ? S 09:29 0:01 /usr/sbin/zabbix_proxy: self-monitoring #1 [terminated] zabbix 1669788 1.1 0.9 352020 17724 ? S 09:29 1:30 /usr/sbin/zabbix_proxy: vmware collector #1 [terminated] zabbix 1669789 1.1 0.9 351960 17648 ? S 09:29 1:28 /usr/sbin/zabbix_proxy: vmware collector #2 [terminated] zabbix 1669790 0.9 0.9 351980 17832 ? S 09:29 1:18 /usr/sbin/zabbix_proxy: vmware collector #3 [terminated] zabbix 1669791 0.5 0.9 352060 17828 ? S 09:29 0:46 /usr/sbin/zabbix_proxy: vmware collector #4 [terminated] zabbix 1669792 0.0 0.5 558860 10540 ? S 09:29 0:00 /usr/sbin/zabbix_proxy: task manager [processing tasks] zabbix 1669793 0.8 1.2 579352 22992 ? S 09:29 1:07 /usr/sbin/zabbix_proxy: poller #1 [got 0 values in 0.000184 sec, getting values] zabbix 1669794 0.7 1.2 579380 23920 ? S 09:29 1:01 /usr/sbin/zabbix_proxy: poller #2 [got 1 values in 0.104406 sec, getting values] zabbix 1669795 0.7 1.2 579568 22908 ? S 09:29 0:59 /usr/sbin/zabbix_proxy: poller #3 [got 0 values in 0.000179 sec, getting values] zabbix 1669796 0.9 1.3 579356 24672 ? S 09:29 1:13 /usr/sbin/zabbix_proxy: poller #4 [got 1 values in 0.083061 sec, getting values] zabbix 1669797 0.8 1.2 579364 22396 ? S 09:29 1:08 /usr/sbin/zabbix_proxy: poller #5 [got 0 values in 0.000179 sec, getting values] zabbix 1669798 0.8 1.2 579448 24152 ? S 09:29 1:04 /usr/sbin/zabbix_proxy: poller #6 [got 1 values in 0.073215 sec, getting values] zabbix 1669799 0.8 1.2 579420 23244 ? S 09:29 1:05 /usr/sbin/zabbix_proxy: poller #7 [got 0 values in 0.000183 sec, getting values] zabbix 1669800 0.7 1.2 580400 24016 ? S 09:29 0:57 /usr/sbin/zabbix_proxy: poller #8 [got 2 values in 0.073889 sec, getting values] zabbix 1669801 0.7 1.2 579356 23032 ? S 09:29 1:02 /usr/sbin/zabbix_proxy: poller #9 [got 1 values in 0.119538 sec, getting values] zabbix 1669802 0.8 1.2 579568 24060 ? S 09:29 1:06 /usr/sbin/zabbix_proxy: poller #10 [got 1 values in 0.089675 sec, getting values] zabbix 1669803 0.0 0.5 558580 10836 ? S 09:29 0:01 /usr/sbin/zabbix_proxy: unreachable poller #1 [got 0 values in 0.000268 sec, getting values] zabbix 1669804 0.0 0.5 558580 10648 ? S 09:29 0:01 /usr/sbin/zabbix_proxy: unreachable poller #2 [got 0 values in 0.000321 sec, getting values] zabbix 1669805 0.0 0.5 558580 11096 ? S 09:29 0:01 /usr/sbin/zabbix_proxy: unreachable poller #3 [got 0 values in 0.000283 sec, getting values] zabbix 1669806 0.0 0.5 558580 10880 ? S 09:29 0:01 /usr/sbin/zabbix_proxy: unreachable poller #4 [got 0 values in 0.000284 sec, getting values] zabbix 1669807 0.0 0.3 348892 6492 ? S 09:29 0:01 /usr/sbin/zabbix_proxy: icmp pinger #1 [pinging hosts] zabbix 1669808 0.0 0.3 348892 6484 ? S 09:29 0:01 /usr/sbin/zabbix_proxy: icmp pinger #2 [getting values] zabbix 1669809 0.0 0.3 348892 6504 ? S 09:29 0:01 /usr/sbin/zabbix_proxy: icmp pinger #3 [getting values] zabbix 1669810 0.0 0.3 348892 6664 ? S 09:29 0:01 /usr/sbin/zabbix_proxy: icmp pinger #4 [getting values] zabbix 1669811 0.0 0.3 348892 6484 ? S 09:29 0:01 /usr/sbin/zabbix_proxy: icmp pinger #5 [getting values] zabbix 1669812 0.0 0.3 348892 6504 ? S 09:29 0:01 /usr/sbin/zabbix_proxy: icmp pinger #6 [getting values] zabbix 1669813 0.2 0.6 347376 12452 ? S 09:29 0:23 /usr/sbin/zabbix_proxy: preprocessing manager #1 [terminated] zabbix 1669814 0.0 0.2 345340 5316 ? S 09:29 0:01 /usr/sbin/zabbix_proxy: preprocessing worker #1 started zabbix 1669815 0.0 0.2 345340 5316 ? S 09:29 0:01 /usr/sbin/zabbix_proxy: preprocessing worker #2 started zabbix 1669816 0.0 0.2 345340 5312 ? S 09:29 0:01 /usr/sbin/zabbix_proxy: preprocessing worker #3 started zabbix 1669817 0.0 0.4 345488 9128 ? S 09:29 0:00 /usr/sbin/zabbix_proxy: history poller #1 [got 1 values in 0.000577 sec, getting values] zabbix 1669818 0.0 0.0 0 0 ? Z 09:29 0:01 [zabbix_proxy] <defunct> My last change is to "killall -s zabbix_proxy". After this command, in journalctl # journalctl -xe -u zabbix-proxy May 27 11:41:04 <HIDDEN> systemd[1]: Stopping Zabbix Proxy... -- Subject: Unit zabbix-proxy.service has begun shutting down -- Defined-By: systemd -- Support: https://access.redhat.com/support -- -- Unit zabbix-proxy.service has begun shutting down. May 27 11:43:05 <HIDDEN> systemd[1]: zabbix-proxy.service: Main process exited, code=killed, status=9/KILL May 27 11:43:05 <HIDDEN> systemd[1]: zabbix-proxy.service: Failed with result 'signal'. -- Subject: Unit failed -- Defined-By: systemd -- Support: https://access.redhat.com/support -- -- The unit zabbix-proxy.service has entered the 'failed' state with result 'signal'. May 27 11:43:05 <HIDDEN> systemd[1]: Stopped Zabbix Proxy. -- Subject: Unit zabbix-proxy.service has finished shutting down -- Defined-By: systemd -- Support: https://access.redhat.com/support -- -- Unit zabbix-proxy.service has finished shutting down. I try to start again (202105271144_zabbix_proxy.log), telnet working now, Zabbix correctly listen again. # telnet 172.31.20.173 10051 Trying 172.31.20.173... Connected to 172.31.20.173. Escape character is '^]'. # netstat -laputen | grep 10051 tcp 0 0 0.0.0.0:10051 0.0.0.0:* LISTEN 1000 73297266 1677241/zabbix_prox You can find Zabbix log before I try to stop process in 202105271118_zabbix_proxy.log, and port status (ZabbixTrapper_10051.PNG). For security reason, some informations are hidden, like name, domain... |
Comment by Vladislavs Sokurenko [ 2021 Jun 04 ] |
Unfortunately there is no information about the crash in log so it must have happened at another time, could you please provide objdump for crashing proxy ? objdump -DSswx
|
Comment by CHAVIGNY [ 2021 Jun 08 ] |
Sorry, I don't understand what you expect from with this command. I have to run this command like this : objdump -DSswx /usr/sbin/zabbix_proxy When zabbix_proxy running ? For the moment, I try to update to 5.4.1. |
Comment by CHAVIGNY [ 2021 Jun 09 ] |
Since 5.4.1 update, I did not have new zabbix proxy crash. |