[ZBX-19445] Zabbix proxy and something go away Created: 2021 May 24  Updated: 2021 Jun 11  Resolved: 2021 Jun 11

Status: Closed
Project: ZABBIX BUGS AND ISSUES
Component/s: Proxy (P), Server (S)
Affects Version/s: 5.4.0
Fix Version/s: None

Type: Problem report Priority: Trivial
Reporter: CHAVIGNY Assignee: Alexey Pustovalov
Resolution: Won't Do Votes: 1
Labels: crash
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

CentOS Linux release 8.3.2011
zabbix_proxy (Zabbix) 5.4.0
Revision 5059e20a75 14 May 2021, compilation time: May 14 2021 14:45:06

Attachments: Text File 202105251814_zabbix_server.log     Text File 202105271118_netstat.log     Text File 202105271118_zabbix_proxy.log     Text File 202105271144_zabbix_proxy.log     PNG File ZabbixTrapper_10051.PNG     Text File zabbix_proxy.log    

 Description   

Steps to reproduce:

  1. Have a proxy with many items

Result:

Zabbix proxy randomly crash and not responding.

I have approximately ten proxy. I notice this problem since update 5.4 only on the most loaded proxies.
When this issue occure, is not possible to restart Zabbix proxy, the only solution is kill process.

zabbix_proxy.log is in attachment.

ps aux | grep defunct

zabbix   1134938  0.0  0.0      0     0 ?        Z    10:15   0:00 [zabbix_proxy] <defunct>

For the first time, I have constated similar issue on Zabbix server, without problem logged in log, except journalctl :

-- Unit zabbix-server.service has finished shutting down.
May 22 00:19:25 <HOSTNAME> systemd[1]: Starting Zabbix Server...
-- Subject: Unit zabbix-server.service has begun start-up
-- Defined-By: systemd
-- Support: https://access.redhat.com/support
--
-- Unit zabbix-server.service has begun starting up.
May 22 00:19:25 <HOSTNAME> zabbix_server[1711003]: /usr/sbin/zabbix_server: /usr/pgsql-11/lib/libpq.so.5: no version information available (required by /usr/sbin/zabb>
May 22 00:19:25 <HOSTNAME> systemd[1]: zabbix-server.service: Can't open PID file /run/zabbix/zabbix_server.pid (yet?) after start: No such file or directory
May 22 00:19:25 <HOSTNAME> systemd[1]: Started Zabbix Server.
-- Subject: Unit zabbix-server.service has finished start-up
-- Defined-By: systemd
-- Support: https://access.redhat.com/support
--
-- Unit zabbix-server.service has finished starting up.
--

Expected:
Zabbix proxy and server running normaly

 Comments   
Comment by CHAVIGNY [ 2021 May 24 ]

Example for Zabbix Server :

2845786:20210524:143316.358 One child process died (PID:2845841,exitcode/signal:6). Exiting ...
zabbix_server [2845786]: Error waiting for process with PID 2845841: [10] No child processes
2845786:20210524:143316.504 syncing history data...
2845786:20210524:143317.063 syncing history data... 100.000000%
2845786:20210524:143317.063 syncing history data done
2845786:20210524:143317.063 syncing trend data...
2845786:20210524:143327.833 syncing trend data done
2845786:20210524:143327.848 Zabbix Server stopped. Zabbix 5.4.0 (revision 5059e20a75).
2850378:20210524:143338.084 Starting Zabbix Server. Zabbix 5.4.0 (revision 5059e20a75).
2850378:20210524:143338.084 ****** Enabled features ******
2850378:20210524:143338.084 SNMP monitoring: YES
2850378:20210524:143338.084 IPMI monitoring: YES
2850378:20210524:143338.084 Web monitoring: YES
2850378:20210524:143338.084 VMware monitoring: YES
2850378:20210524:143338.084 SMTP authentication: YES
2850378:20210524:143338.084 ODBC: YES
2850378:20210524:143338.084 SSH support: YES
2850378:20210524:143338.084 IPv6 support: YES
2850378:20210524:143338.084 TLS support: YES
2850378:20210524:143338.084 ******************************
2850378:20210524:143338.084 using configuration file: /etc/zabbix/zabbix_server.conf

NB : In this example, Zabbix has successfully restarted by him self, journalctl say :

--
-- Unit zabbix-server.service has finished starting up.
--
-- The start-up result is done.
May 24 14:33:27 <HOSTNAME> kill[2850346]: kill: not enough arguments
May 24 14:33:27 <HOSTNAME> systemd[1]: zabbix-server.service: Control process exited, code=exited status=1
May 24 14:33:27 <HOSTNAME> systemd[1]: zabbix-server.service: Failed with result 'exit-code'.
-- Subject: Unit failed
-- Defined-By: systemd
-- Support: https://access.redhat.com/support
--
-- The unit zabbix-server.service has entered the 'failed' state with result 'exit-code'.
May 24 14:33:38 <HOSTNAME> systemd[1]: zabbix-server.service: Service RestartSec=10s expired, scheduling restart.
May 24 14:33:38 <HOSTNAME> systemd[1]: zabbix-server.service: Scheduled restart job, restart counter is at 57.
-- Subject: Automatic restarting of a unit has been scheduled
-- Defined-By: systemd
-- Support: https://access.redhat.com/support
--
-- Automatic restarting of the unit zabbix-server.service has been scheduled, as the result for
-- the configured Restart= setting for the unit.
May 24 14:33:38 <HOSTNAME> systemd[1]: Stopped Zabbix Server.
-- Subject: Unit zabbix-server.service has finished shutting down
-- Defined-By: systemd
-- Support: https://access.redhat.com/support
--
-- Unit zabbix-server.service has finished shutting down.
May 24 14:33:38 <HOSTNAME> systemd[1]: Starting Zabbix Server...
-- Subject: Unit zabbix-server.service has begun start-up
-- Defined-By: systemd
-- Support: https://access.redhat.com/support
--
-- Unit zabbix-server.service has begun starting up.
May 24 14:33:38 <HOSTNAME> zabbix_server[2850376]: /usr/sbin/zabbix_server: /usr/pgsql-11/lib/libpq.so.5: no version information available (required by /usr/sbin/zabbix_server)
May 24 14:33:38 <HOSTNAME> systemd[1]: zabbix-server.service: Can't open PID file /run/zabbix/zabbix_server.pid (yet?) after start: No such file or directory
May 24 14:33:38 <HOSTNAME> systemd[1]: Started Zabbix Server.
-- Subject: Unit zabbix-server.service has finished start-up
-- Defined-By: systemd
-- Support: https://access.redhat.com/support
--
-- Unit zabbix-server.service has finished starting up.
--
-- The start-up result is done.
Comment by Vladislavs Sokurenko [ 2021 May 25 ]

Thank you for your report, could you please be so kind and increase log level and then attach log again ?

Comment by Vladislavs Sokurenko [ 2021 May 25 ]

How often does issue happen ? Crash happens in data sender process, could increase log level only for it:

zabbix_proxy -R log_level_increase="data sender"
Comment by Vladislavs Sokurenko [ 2021 May 25 ]

Backtrace for easier searching:

1132913:20210524:101541.740 Got signal [signal:11(SIGSEGV),reason:1,refaddr:(nil)]. Crashing ...
1132913:20210524:101541.740 ====== Fatal information: ======
1132913:20210524:101541.740 Program counter: 0x55c5ff626965
1132913:20210524:101541.740 === Registers: ===
1132913:20210524:101541.740 r8      =                0 =                    0 =                    0
1132913:20210524:101541.740 r9      =                6 =                    6 =                    6
1132913:20210524:101541.740 r10     =                0 =                    0 =                    0
1132913:20210524:101541.740 r11     =     7f941a357dc0 =      140274071600576 =      140274071600576
1132913:20210524:101541.740 r12     =                1 =                    1 =                    1
1132913:20210524:101541.740 r13     =     55c5ff7cf863 =       94308883298403 =       94308883298403
1132913:20210524:101541.740 r14     =                0 =                    0 =                    0
1132913:20210524:101541.740 r15     =                0 =                    0 =                    0
1132913:20210524:101541.740 rdi     =                0 =                    0 =                    0
1132913:20210524:101541.740 rsi     =     7ffde8e066ec =      140728510473964 =      140728510473964
1132913:20210524:101541.740 rbp     =     7ffde8e047e0 =      140728510466016 =      140728510466016
1132913:20210524:101541.740 rbx     =                1 =                    1 =                    1
1132913:20210524:101541.740 rdx     =     7ffde8e066ec =      140728510473964 =      140728510473964
1132913:20210524:101541.740 rax     =                0 =                    0 =                    0
1132913:20210524:101541.740 rcx     =                6 =                    6 =                    6
1132913:20210524:101541.740 rsp     =     7ffde8e03fc0 =      140728510463936 =      140728510463936
1132913:20210524:101541.740 rip     =     55c5ff626965 =       94308881557861 =       94308881557861
1132913:20210524:101541.740 efl     =            10206 =                66054 =                66054
1132913:20210524:101541.740 csgsfs  =   2b000000000033 =    12103423998558259 =    12103423998558259
1132913:20210524:101541.740 err     =                4 =                    4 =                    4
1132913:20210524:101541.740 trapno  =                e =                   14 =                   14
1132913:20210524:101541.740 oldmask =                0 =                    0 =                    0
1132913:20210524:101541.740 cr2     =                0 =                    0 =                    0
1132913:20210524:101541.740 === Backtrace: ===
1132913:20210524:101541.742 12: /usr/sbin/zabbix_proxy: data sender [sent 0 values in 36.164480 sec, sending data](zbx_backtrace+0x3f) [0x55c5ff743bbf]
1132913:20210524:101541.742 11: /usr/sbin/zabbix_proxy: data sender [sent 0 values in 36.164480 sec, sending data](zbx_log_fatal_info+0x141) [0x55c5ff743e1c]
1132913:20210524:101541.742 10: /usr/sbin/zabbix_proxy: data sender [sent 0 values in 36.164480 sec, sending data](+0x19d608) [0x55c5ff744608]
1132913:20210524:101541.742 9: /lib64/libpthread.so.0(+0x12b20) [0x7f941c3e4b20]
1132913:20210524:101541.742 8: /usr/sbin/zabbix_proxy: data sender [sent 0 values in 36.164480 sec, sending data](+0x7f965) [0x55c5ff626965]
1132913:20210524:101541.742 7: /usr/sbin/zabbix_proxy: data sender [sent 0 values in 36.164480 sec, sending data](+0x7fe4e) [0x55c5ff626e4e]
1132913:20210524:101541.742 6: /usr/sbin/zabbix_proxy: data sender [sent 0 values in 36.164480 sec, sending data](datasender_thread+0x15b) [0x55c5ff627216]
1132913:20210524:101541.742 5: /usr/sbin/zabbix_proxy: data sender [sent 0 values in 36.164480 sec, sending data](zbx_thread_start+0x37) [0x55c5ff748d49]
1132913:20210524:101541.742 4: /usr/sbin/zabbix_proxy: data sender [sent 0 values in 36.164480 sec, sending data](MAIN_ZABBIX_ENTRY+0xa60) [0x55c5ff5ece7e]
1132913:20210524:101541.742 3: /usr/sbin/zabbix_proxy: data sender [sent 0 values in 36.164480 sec, sending data](daemon_start+0x2ff) [0x55c5ff7437e9]
1132913:20210524:101541.742 2: /usr/sbin/zabbix_proxy: data sender [sent 0 values in 36.164480 sec, sending data](main+0x2f1) [0x55c5ff5ec3d4]
1132913:20210524:101541.742 1: /lib64/libc.so.6(__libc_start_main+0xf3) [0x7f941a1fd7b3]
1132913:20210524:101541.742 0: /usr/sbin/zabbix_proxy: data sender [sent 0 values in 36.164480 sec, sending data](_start+0x2e) [0x55c5ff5eb27e]
Comment by CHAVIGNY [ 2021 May 25 ]

You can find new crash report from zabbix_server in "202105251814_zabbix_server.log" after increasing "data sender". Currently, I have no crash on Zabbix Proxy since last time.

Comment by Vladislavs Sokurenko [ 2021 May 26 ]

I am sorry, I meant to increase log level on proxy that crash

Comment by CHAVIGNY [ 2021 May 27 ]

Ok, I have two crash of zabbix proxy this night in DebugLevel 3. I set this morning to DebugLevel 4 and I have new crash with many information. I will try to explain as best as possible the unfolding of the situation.

At around 10:30 a.m. The Zabbix trapper port don't accept any connection, I try in localhost :

# telnet 172.31.20.173 10051
Trying 172.31.20.173...

But, Zabbix Proxy running normally :

# ps aux | grep zabbix_proxy
zabbix   1669758  0.0  0.3 345228  6176 ?        S    09:29   0:00 /usr/sbin/zabbix_proxy -c /etc/zabbix/zabbix_proxy.conf
zabbix   1669759  0.0  1.9 367936 36656 ?        S    09:29   0:02 /usr/sbin/zabbix_proxy: configuration syncer [loading configuration]
zabbix   1669769  0.0  1.9 581948 35604 ?        S    09:29   0:00 /usr/sbin/zabbix_proxy: trapper #1 [processed data in 0.001354 sec, waiting for connection]
zabbix   1669770  0.0  1.8 577928 33524 ?        S    09:29   0:00 /usr/sbin/zabbix_proxy: trapper #2 [processed data in 0.001677 sec, waiting for connection]
zabbix   1669771  0.0  1.6 575180 31148 ?        S    09:29   0:00 /usr/sbin/zabbix_proxy: trapper #3 [processed data in 0.000760 sec, waiting for connection]
zabbix   1669772  0.0  1.8 577928 33680 ?        S    09:29   0:00 /usr/sbin/zabbix_proxy: trapper #4 [processed data in 0.001294 sec, waiting for connection]
zabbix   1669773  0.0  2.0 582124 37864 ?        S    09:29   0:00 /usr/sbin/zabbix_proxy: trapper #5 [processed data in 0.000610 sec, waiting for connection]
zabbix   1669774  0.0  1.6 575044 31080 ?        S    09:29   0:00 /usr/sbin/zabbix_proxy: trapper #6 [processed data in 0.000673 sec, waiting for connection]
zabbix   1669775  0.0  0.3 345340  7408 ?        S    09:29   0:00 /usr/sbin/zabbix_proxy: heartbeat sender [sending heartbeat message success in 0.002648 sec, idle 60 sec]
zabbix   1669776  0.0  2.2 374480 41956 ?        S    09:29   0:05 /usr/sbin/zabbix_proxy: data sender [sent 0 values in 0.004296 sec, sending data]
zabbix   1669777  0.0  0.3 348728  5876 ?        S    09:29   0:00 /usr/sbin/zabbix_proxy: ipmi manager #1 [scheduled 0, polled 0 values, idle 5.005919 sec during 5.005974 sec]
zabbix   1669778  0.0  0.4 347232  7448 ?        S    09:29   0:00 /usr/sbin/zabbix_proxy: housekeeper [connecting to the database]
zabbix   1669779  0.0  0.2 345348  5460 ?        S    09:29   0:00 /usr/sbin/zabbix_proxy: http poller #1 [got 0 values in 0.000505 sec, getting values]
zabbix   1669780  0.0  0.5 558860 10480 ?        S    09:29   0:00 /usr/sbin/zabbix_proxy: discoverer #1 [processed 0 rules in 0.000495 sec, performing discovery]
zabbix   1669781  0.0  0.8 350408 15080 ?        S    09:29   0:03 /usr/sbin/zabbix_proxy: history syncer #1 [processed 0 values in 0.000062 sec, idle 1 sec]
zabbix   1669782  0.0  0.7 349816 14656 ?        S    09:29   0:02 /usr/sbin/zabbix_proxy: history syncer #2 [processed 0 values in 0.000179 sec, idle 1 sec]
zabbix   1669783  0.0  0.7 350204 14808 ?        S    09:29   0:01 /usr/sbin/zabbix_proxy: history syncer #3 [processed 0 values in 0.000099 sec, syncing history]
zabbix   1669784  0.0  0.8 350648 15652 ?        S    09:29   0:02 /usr/sbin/zabbix_proxy: history syncer #4 [processed 0 values in 0.000064 sec, idle 1 sec]
zabbix   1669785  0.0  0.2 345228  4404 ?        S    09:29   0:00 /usr/sbin/zabbix_proxy: ipmi poller #1 started
zabbix   1669786  0.0  0.2 345404  5300 ?        S    09:29   0:01 /usr/sbin/zabbix_proxy: snmp trapper [processed data in 0.000141 sec, idle 1 sec]
zabbix   1669787  0.0  0.2 345228  4412 ?        S    09:29   0:01 /usr/sbin/zabbix_proxy: self-monitoring [processed data in 0.000167 sec, idle 1 sec]
zabbix   1669788  1.1  0.9 352020 17724 ?        S    09:29   1:30 /usr/sbin/zabbix_proxy: vmware collector #1 [updated 0, removed 0 VMware services in 0.000065 sec, idle 5 sec]
zabbix   1669789  1.1  0.9 351960 17648 ?        S    09:29   1:28 /usr/sbin/zabbix_proxy: vmware collector #2 [updated 0, removed 0 VMware services in 0.000087 sec, idle 5 sec]
zabbix   1669790  1.0  0.9 351980 17832 ?        S    09:29   1:18 /usr/sbin/zabbix_proxy: vmware collector #3 [updated 0, removed 0 VMware services in 0.000180 sec, idle 5 sec]
zabbix   1669791  0.6  0.9 352060 17828 ?        S    09:29   0:46 /usr/sbin/zabbix_proxy: vmware collector #4 [updated 0, removed 0 VMware services in 0.000148 sec, idle 5 sec]
zabbix   1669792  0.0  0.5 558860 10540 ?        S    09:29   0:00 /usr/sbin/zabbix_proxy: task manager [processing tasks]
zabbix   1669793  0.8  1.2 579352 22992 ?        S    09:29   1:07 /usr/sbin/zabbix_proxy: poller #1 [got 0 values in 0.000184 sec, getting values]
zabbix   1669794  0.8  1.2 579380 23920 ?        S    09:29   1:01 /usr/sbin/zabbix_proxy: poller #2 [got 1 values in 0.104406 sec, getting values]
zabbix   1669795  0.7  1.2 579568 22908 ?        S    09:29   0:59 /usr/sbin/zabbix_proxy: poller #3 [got 0 values in 0.000179 sec, getting values]
zabbix   1669796  0.9  1.3 579356 24672 ?        S    09:29   1:13 /usr/sbin/zabbix_proxy: poller #4 [got 1 values in 0.083061 sec, getting values]
zabbix   1669797  0.9  1.2 579364 22396 ?        S    09:29   1:08 /usr/sbin/zabbix_proxy: poller #5 [got 0 values in 0.000179 sec, getting values]
zabbix   1669798  0.8  1.2 579448 24152 ?        S    09:29   1:04 /usr/sbin/zabbix_proxy: poller #6 [got 1 values in 0.073215 sec, getting values]
zabbix   1669799  0.8  1.2 579420 23244 ?        S    09:29   1:05 /usr/sbin/zabbix_proxy: poller #7 [got 0 values in 0.000183 sec, getting values]
zabbix   1669800  0.7  1.2 580400 24016 ?        S    09:29   0:57 /usr/sbin/zabbix_proxy: poller #8 [got 2 values in 0.073889 sec, getting values]
zabbix   1669801  0.8  1.2 579356 23032 ?        S    09:29   1:02 /usr/sbin/zabbix_proxy: poller #9 [got 1 values in 0.119538 sec, getting values]
zabbix   1669802  0.8  1.2 579568 24060 ?        S    09:29   1:06 /usr/sbin/zabbix_proxy: poller #10 [got 1 values in 0.089675 sec, getting values]
zabbix   1669803  0.0  0.5 558580 10836 ?        S    09:29   0:01 /usr/sbin/zabbix_proxy: unreachable poller #1 [got 0 values in 0.000268 sec, getting values]
zabbix   1669804  0.0  0.5 558580 10648 ?        S    09:29   0:01 /usr/sbin/zabbix_proxy: unreachable poller #2 [got 0 values in 0.000321 sec, getting values]
zabbix   1669805  0.0  0.5 558580 11096 ?        S    09:29   0:01 /usr/sbin/zabbix_proxy: unreachable poller #3 [got 0 values in 0.000283 sec, getting values]
zabbix   1669806  0.0  0.5 558580 10880 ?        S    09:29   0:01 /usr/sbin/zabbix_proxy: unreachable poller #4 [got 0 values in 0.000284 sec, getting values]
zabbix   1669807  0.0  0.3 348892  6492 ?        S    09:29   0:01 /usr/sbin/zabbix_proxy: icmp pinger #1 [pinging hosts]
zabbix   1669808  0.0  0.3 348892  6484 ?        S    09:29   0:01 /usr/sbin/zabbix_proxy: icmp pinger #2 [getting values]
zabbix   1669809  0.0  0.3 348892  6504 ?        S    09:29   0:01 /usr/sbin/zabbix_proxy: icmp pinger #3 [getting values]
zabbix   1669810  0.0  0.3 348892  6664 ?        S    09:29   0:01 /usr/sbin/zabbix_proxy: icmp pinger #4 [getting values]
zabbix   1669811  0.0  0.3 348892  6484 ?        S    09:29   0:01 /usr/sbin/zabbix_proxy: icmp pinger #5 [getting values]
zabbix   1669812  0.0  0.3 348892  6504 ?        S    09:29   0:01 /usr/sbin/zabbix_proxy: icmp pinger #6 [getting values]
zabbix   1669813  0.3  0.6 347376 12452 ?        S    09:29   0:23 /usr/sbin/zabbix_proxy: preprocessing manager #1 [queued 0, processed 0 values, idle 5.006462 sec during 5.006528 sec]
zabbix   1669814  0.0  0.2 345340  5316 ?        S    09:29   0:01 /usr/sbin/zabbix_proxy: preprocessing worker #1 started
zabbix   1669815  0.0  0.2 345340  5316 ?        S    09:29   0:01 /usr/sbin/zabbix_proxy: preprocessing worker #2 started
zabbix   1669816  0.0  0.2 345340  5312 ?        S    09:29   0:01 /usr/sbin/zabbix_proxy: preprocessing worker #3 started
zabbix   1669817  0.0  0.4 345488  9128 ?        S    09:29   0:00 /usr/sbin/zabbix_proxy: history poller #1 [got 1 values in 0.000577 sec, getting values]
zabbix   1669818  0.0  0.2 345228  5300 ?        S    09:29   0:01 /usr/sbin/zabbix_proxy: availability manager #1 [queued 0, processed 0 values, idle 5.006362 sec during 5.006433 sec]

However, many ports appear to be in "CLOSE_WAIT" (202105271118_netstat.log)

I try to stop Zabbix-proxy, without sucess :

# (systemctl stop zabbix-proxy &) ; tail -f /var/log/zabbix/zabbix_proxy.log
1669818:20210527:114104.141 In zbx_ipc_service_recv() timeout:1
1669787:20210527:114104.383 zbx_setproctitle() title:'self-monitoring [processing data]'
1669787:20210527:114104.383 In collect_selfmon_stats()
1669787:20210527:114104.383 End of collect_selfmon_stats()
1669787:20210527:114104.383 zbx_setproctitle() title:'self-monitoring [processed data in 0.000206 sec, idle 1 sec]'
1669785:20210527:114104.443 End of zbx_ipc_async_socket_recv():0
1669786:20210527:114104.446 zbx_setproctitle() title:'snmp trapper [processing data]'
1669786:20210527:114104.446 zbx_setproctitle() title:'snmp trapper [processed data in 0.000073 sec, idle 1 sec]'
1669813:20210527:114104.460 End of zbx_ipc_service_recv():2
1669813:20210527:114104.460 In zbx_ipc_service_recv() timeout:1
1669758:20210527:114104.786 Got signal [signal:15(SIGTERM),sender_pid:1677081,sender_uid:0,reason:0]. Exiting ...
1669758:20210527:114104.786 zbx_on_exit() called
1669782:20210527:114104.786 zbx_setproctitle() title:'history syncer #2 [processed 0 values in 0.000189 sec, syncing history]'
1669782:20210527:114104.786 syncing history data in progress...
1669782:20210527:114104.786 In zbx_sync_history_cache() history_num:11
1669786:20210527:114104.787 zbx_setproctitle() title:'snmp trapper [processing data]'
1669786:20210527:114104.787 zbx_setproctitle() title:'snmp trapper [processed data in 0.009810 sec, idle 1 sec]'
1669787:20210527:114104.787 zbx_setproctitle() title:'self-monitoring [processing data]'
1669787:20210527:114104.787 In collect_selfmon_stats()
1669787:20210527:114104.787 End of collect_selfmon_stats()
1669787:20210527:114104.787 zbx_setproctitle() title:'self-monitoring [processed data in 0.010045 sec, idle 1 sec]'
1669784:20210527:114104.787 zbx_setproctitle() title:'history syncer #4 [processed 0 values in 0.000200 sec, syncing history]'
1669784:20210527:114104.787 In zbx_sync_history_cache() history_num:11
1669781:20210527:114104.787 zbx_setproctitle() title:'history syncer #1 [processed 0 values in 0.000161 sec, syncing history]'
1669781:20210527:114104.787 In zbx_sync_history_cache() history_num:11
1669818:20210527:114105.142 End of zbx_ipc_service_recv():2
1669813:20210527:114105.461 End of zbx_ipc_service_recv():2
1669813:20210527:114105.461 zbx_setproctitle() title:'preprocessing manager #1 [terminated]'
1669785:20210527:114105.787 zbx_setproctitle() title:'ipmi poller #1 [terminated]'
1669786:20210527:114105.787 zbx_setproctitle() title:'snmp trapper #1 [terminated]'
1669787:20210527:114105.787 zbx_setproctitle() title:'self-monitoring #1 [terminated]'
1669791:20210527:114106.787 zbx_setproctitle() title:'vmware collector #4 [terminated]'
1669789:20210527:114107.787 zbx_setproctitle() title:'vmware collector #2 [terminated]'
1669790:20210527:114107.787 zbx_setproctitle() title:'vmware collector #3 [terminated]'
1669788:20210527:114107.787 zbx_setproctitle() title:'vmware collector #1 [terminated]'
1669775:20210527:114145.791 zbx_setproctitle() title:'heartbeat sender #1 [terminated]'

Zabbix-proxy appear "defunct" after try to stop process

# ps aux | grep zabbix_proxy
zabbix   1669758  0.0  0.3 345228  6176 ?        S    09:29   0:00 /usr/sbin/zabbix_proxy -c /etc/zabbix/zabbix_proxy.conf
zabbix   1669759  0.0  1.9 367936 36656 ?        S    09:29   0:02 /usr/sbin/zabbix_proxy: configuration syncer [loading configuration]
zabbix   1669769  0.0  1.9 581948 35604 ?        S    09:29   0:00 /usr/sbin/zabbix_proxy: trapper #1 [processed data in 0.001354 sec, waiting for connection]
zabbix   1669770  0.0  1.8 577928 33524 ?        S    09:29   0:00 /usr/sbin/zabbix_proxy: trapper #2 [processed data in 0.001677 sec, waiting for connection]
zabbix   1669771  0.0  1.6 575180 31148 ?        S    09:29   0:00 /usr/sbin/zabbix_proxy: trapper #3 [processed data in 0.000760 sec, waiting for connection]
zabbix   1669772  0.0  1.8 577928 33680 ?        S    09:29   0:00 /usr/sbin/zabbix_proxy: trapper #4 [processed data in 0.001294 sec, waiting for connection]
zabbix   1669773  0.0  2.0 582124 37864 ?        S    09:29   0:00 /usr/sbin/zabbix_proxy: trapper #5 [processed data in 0.000610 sec, waiting for connection]
zabbix   1669774  0.0  1.6 575044 31080 ?        S    09:29   0:00 /usr/sbin/zabbix_proxy: trapper #6 [processed data in 0.000673 sec, waiting for connection]
zabbix   1669775  0.0  0.3 345340  7408 ?        S    09:29   0:00 /usr/sbin/zabbix_proxy: heartbeat sender #1 [terminated]
zabbix   1669776  0.0  2.2 374480 41956 ?        S    09:29   0:05 /usr/sbin/zabbix_proxy: data sender [sent 0 values in 0.004296 sec, sending data]
zabbix   1669777  0.0  0.3 348728  6092 ?        S    09:29   0:00 /usr/sbin/zabbix_proxy: ipmi manager #1 [scheduled 0, polled 0 values, idle 5.005919 sec during 5.005974 sec]
zabbix   1669778  0.0  0.4 347232  7448 ?        S    09:29   0:00 /usr/sbin/zabbix_proxy: housekeeper [connecting to the database]
zabbix   1669779  0.0  0.2 345348  5460 ?        S    09:29   0:00 /usr/sbin/zabbix_proxy: http poller #1 [got 0 values in 0.000505 sec, getting values]
zabbix   1669780  0.0  0.5 558860 10480 ?        S    09:29   0:00 /usr/sbin/zabbix_proxy: discoverer #1 [processed 0 rules in 0.000495 sec, performing discovery]
zabbix   1669781  0.0  0.8 350408 15080 ?        S    09:29   0:03 /usr/sbin/zabbix_proxy: history syncer #1 [processed 0 values in 0.000161 sec, syncing history]
zabbix   1669782  0.0  0.7 349816 14656 ?        S    09:29   0:02 /usr/sbin/zabbix_proxy: history syncer #2 [processed 0 values in 0.000189 sec, syncing history]
zabbix   1669783  0.0  0.7 350204 14808 ?        S    09:29   0:01 /usr/sbin/zabbix_proxy: history syncer #3 [processed 0 values in 0.000099 sec, syncing history]
zabbix   1669784  0.0  0.8 350648 15652 ?        S    09:29   0:02 /usr/sbin/zabbix_proxy: history syncer #4 [processed 0 values in 0.000200 sec, syncing history]
zabbix   1669785  0.0  0.2 345228  4404 ?        S    09:29   0:00 /usr/sbin/zabbix_proxy: ipmi poller #1 [terminated]
zabbix   1669786  0.0  0.2 345340  5500 ?        S    09:29   0:01 /usr/sbin/zabbix_proxy: snmp trapper #1 [terminated]
zabbix   1669787  0.0  0.2 345228  4412 ?        S    09:29   0:01 /usr/sbin/zabbix_proxy: self-monitoring #1 [terminated]
zabbix   1669788  1.1  0.9 352020 17724 ?        S    09:29   1:30 /usr/sbin/zabbix_proxy: vmware collector #1 [terminated]
zabbix   1669789  1.1  0.9 351960 17648 ?        S    09:29   1:28 /usr/sbin/zabbix_proxy: vmware collector #2 [terminated]
zabbix   1669790  0.9  0.9 351980 17832 ?        S    09:29   1:18 /usr/sbin/zabbix_proxy: vmware collector #3 [terminated]
zabbix   1669791  0.5  0.9 352060 17828 ?        S    09:29   0:46 /usr/sbin/zabbix_proxy: vmware collector #4 [terminated]
zabbix   1669792  0.0  0.5 558860 10540 ?        S    09:29   0:00 /usr/sbin/zabbix_proxy: task manager [processing tasks]
zabbix   1669793  0.8  1.2 579352 22992 ?        S    09:29   1:07 /usr/sbin/zabbix_proxy: poller #1 [got 0 values in 0.000184 sec, getting values]
zabbix   1669794  0.7  1.2 579380 23920 ?        S    09:29   1:01 /usr/sbin/zabbix_proxy: poller #2 [got 1 values in 0.104406 sec, getting values]
zabbix   1669795  0.7  1.2 579568 22908 ?        S    09:29   0:59 /usr/sbin/zabbix_proxy: poller #3 [got 0 values in 0.000179 sec, getting values]
zabbix   1669796  0.9  1.3 579356 24672 ?        S    09:29   1:13 /usr/sbin/zabbix_proxy: poller #4 [got 1 values in 0.083061 sec, getting values]
zabbix   1669797  0.8  1.2 579364 22396 ?        S    09:29   1:08 /usr/sbin/zabbix_proxy: poller #5 [got 0 values in 0.000179 sec, getting values]
zabbix   1669798  0.8  1.2 579448 24152 ?        S    09:29   1:04 /usr/sbin/zabbix_proxy: poller #6 [got 1 values in 0.073215 sec, getting values]
zabbix   1669799  0.8  1.2 579420 23244 ?        S    09:29   1:05 /usr/sbin/zabbix_proxy: poller #7 [got 0 values in 0.000183 sec, getting values]
zabbix   1669800  0.7  1.2 580400 24016 ?        S    09:29   0:57 /usr/sbin/zabbix_proxy: poller #8 [got 2 values in 0.073889 sec, getting values]
zabbix   1669801  0.7  1.2 579356 23032 ?        S    09:29   1:02 /usr/sbin/zabbix_proxy: poller #9 [got 1 values in 0.119538 sec, getting values]
zabbix   1669802  0.8  1.2 579568 24060 ?        S    09:29   1:06 /usr/sbin/zabbix_proxy: poller #10 [got 1 values in 0.089675 sec, getting values]
zabbix   1669803  0.0  0.5 558580 10836 ?        S    09:29   0:01 /usr/sbin/zabbix_proxy: unreachable poller #1 [got 0 values in 0.000268 sec, getting values]
zabbix   1669804  0.0  0.5 558580 10648 ?        S    09:29   0:01 /usr/sbin/zabbix_proxy: unreachable poller #2 [got 0 values in 0.000321 sec, getting values]
zabbix   1669805  0.0  0.5 558580 11096 ?        S    09:29   0:01 /usr/sbin/zabbix_proxy: unreachable poller #3 [got 0 values in 0.000283 sec, getting values]
zabbix   1669806  0.0  0.5 558580 10880 ?        S    09:29   0:01 /usr/sbin/zabbix_proxy: unreachable poller #4 [got 0 values in 0.000284 sec, getting values]
zabbix   1669807  0.0  0.3 348892  6492 ?        S    09:29   0:01 /usr/sbin/zabbix_proxy: icmp pinger #1 [pinging hosts]
zabbix   1669808  0.0  0.3 348892  6484 ?        S    09:29   0:01 /usr/sbin/zabbix_proxy: icmp pinger #2 [getting values]
zabbix   1669809  0.0  0.3 348892  6504 ?        S    09:29   0:01 /usr/sbin/zabbix_proxy: icmp pinger #3 [getting values]
zabbix   1669810  0.0  0.3 348892  6664 ?        S    09:29   0:01 /usr/sbin/zabbix_proxy: icmp pinger #4 [getting values]
zabbix   1669811  0.0  0.3 348892  6484 ?        S    09:29   0:01 /usr/sbin/zabbix_proxy: icmp pinger #5 [getting values]
zabbix   1669812  0.0  0.3 348892  6504 ?        S    09:29   0:01 /usr/sbin/zabbix_proxy: icmp pinger #6 [getting values]
zabbix   1669813  0.2  0.6 347376 12452 ?        S    09:29   0:23 /usr/sbin/zabbix_proxy: preprocessing manager #1 [terminated]
zabbix   1669814  0.0  0.2 345340  5316 ?        S    09:29   0:01 /usr/sbin/zabbix_proxy: preprocessing worker #1 started
zabbix   1669815  0.0  0.2 345340  5316 ?        S    09:29   0:01 /usr/sbin/zabbix_proxy: preprocessing worker #2 started
zabbix   1669816  0.0  0.2 345340  5312 ?        S    09:29   0:01 /usr/sbin/zabbix_proxy: preprocessing worker #3 started
zabbix   1669817  0.0  0.4 345488  9128 ?        S    09:29   0:00 /usr/sbin/zabbix_proxy: history poller #1 [got 1 values in 0.000577 sec, getting values]
zabbix   1669818  0.0  0.0      0     0 ?        Z    09:29   0:01 [zabbix_proxy] <defunct>

My last change is to "killall -s zabbix_proxy". After this command, in journalctl

# journalctl -xe -u zabbix-proxy
May 27 11:41:04 <HIDDEN> systemd[1]: Stopping Zabbix Proxy...
-- Subject: Unit zabbix-proxy.service has begun shutting down
-- Defined-By: systemd
-- Support: https://access.redhat.com/support
--
-- Unit zabbix-proxy.service has begun shutting down.
May 27 11:43:05 <HIDDEN> systemd[1]: zabbix-proxy.service: Main process exited, code=killed, status=9/KILL
May 27 11:43:05 <HIDDEN> systemd[1]: zabbix-proxy.service: Failed with result 'signal'.
-- Subject: Unit failed
-- Defined-By: systemd
-- Support: https://access.redhat.com/support
--
-- The unit zabbix-proxy.service has entered the 'failed' state with result 'signal'.
May 27 11:43:05 <HIDDEN> systemd[1]: Stopped Zabbix Proxy.
-- Subject: Unit zabbix-proxy.service has finished shutting down
-- Defined-By: systemd
-- Support: https://access.redhat.com/support
--
-- Unit zabbix-proxy.service has finished shutting down.

I try to start again (202105271144_zabbix_proxy.log), telnet working now, Zabbix correctly listen again.

# telnet 172.31.20.173 10051
Trying 172.31.20.173...
Connected to 172.31.20.173.
Escape character is '^]'.
# netstat -laputen | grep 10051
tcp        0      0 0.0.0.0:10051           0.0.0.0:*               LISTEN      1000       73297266   1677241/zabbix_prox

You can find Zabbix log before I try to stop process in 202105271118_zabbix_proxy.log, and port status (ZabbixTrapper_10051.PNG).

For security reason, some informations are hidden, like name, domain...

Comment by Vladislavs Sokurenko [ 2021 Jun 04 ]

Unfortunately there is no information about the crash in log so it must have happened at another time, could you please provide objdump for crashing proxy ?

objdump -DSswx

 

Comment by CHAVIGNY [ 2021 Jun 08 ]

Sorry, I don't understand what you expect from with this command.

I have to run this command like this :

objdump -DSswx /usr/sbin/zabbix_proxy

When zabbix_proxy running ?

For the moment, I try to update to 5.4.1.

Comment by CHAVIGNY [ 2021 Jun 09 ]

Since 5.4.1 update, I did not have new zabbix proxy crash.

Generated at Wed Jun 25 07:17:44 EEST 2025 using Jira 9.12.4#9120004-sha1:625303b708afdb767e17cb2838290c41888e9ff0.