-
Patch request
-
Resolution: Workaround proposed
-
Blocker
-
None
-
4.0.9
-
None
-
Prod
Hi,
Frequently i can see "zabbix server is not running the information displayed may not be current" message in my GUI. while checking on GUI 'Zabbix server is running' showing as 'NO'. While checking on server i can see service is running
[root@zbx_server1v ~]# *systemctl status zabbix-server.service* ● zabbix-server.service - Zabbix Server Loaded: loaded (/usr/lib/systemd/system/zabbix-server.service; enabled; vendor preset: disabled) Active: active (running) since Mon 2019-07-01 04:57:25 CDT; 8min ago Process: 122746 ExecStop=/bin/kill -SIGTERM $MAINPID (code=exited, status=1/FAILURE) Process: 122762 ExecStart=/usr/sbin/zabbix_server -c $CONFFILE (code=exited, status=0/SUCCESS) Main PID: 122764 (zabbix_server) Tasks: 54 CGroup: /system.slice/zabbix-server.service ├─122764 /usr/sbin/zabbix_server -c /etc/zabbix/zabbix_server.conf └─122772 /usr/sbin/zabbix_server: configuration syncer [synced configuration in 1561975476.819242 sec, syncing configuration] Jul 01 04:57:25 zbx_server1v.localhost.com systemd[1]: Starting Zabbix Server... Jul 01 04:57:25 zbx_server1v.localhost.com systemd[1]: zabbix-server.service: Supervising process 122764 which is not our child. We'll most likely not notice when it exits. Jul 01 04:57:25 zbx_server1v.localhost.com systemd[1]: Started Zabbix Server. Warning: zabbix-server.service changed on disk. Run 'systemctl daemon-reload' to reload units. [root@zbx_server1 ~]#
but process are showing in "defunct" state. Every time i've to stop & restart service to get the same fixed. But after some times it's again going to "defunct" state.
[root@zbx_server1 ~]# ps -ef|grep zabbix_server zabbix 122764 1 0 04:57 ? 00:00:01 /usr/sbin/zabbix_server -c /etc/zabbix/zabbix_server.conf zabbix 122772 122764 1 04:57 ? 00:00:06 /usr/sbin/zabbix_server: configuration syncer [synced configuration in 1561975476.819242 sec, syncing configuration] zabbix 122773 122764 0 04:57 ? 00:00:00 [zabbix_server] <defunct> zabbix 122774 122764 0 04:57 ? 00:00:00 [zabbix_server] <defunct> zabbix 122775 122764 0 04:57 ? 00:00:00 [zabbix_server] <defunct> zabbix 122776 122764 0 04:57 ? 00:00:00 [zabbix_server] <defunct> zabbix 122777 122764 0 04:57 ? 00:00:00 [zabbix_server] <defunct> zabbix 122778 122764 0 04:57 ? 00:00:00 [zabbix_server] <defunct> zabbix 122779 122764 0 04:57 ? 00:00:00 [zabbix_server] <defunct> zabbix 122781 122764 0 04:57 ? 00:00:00 [zabbix_server] <defunct> zabbix 122783 122764 0 04:57 ? 00:00:00 [zabbix_server] <defunct> zabbix 122785 122764 0 04:57 ? 00:00:00 [zabbix_server] <defunct> zabbix 122787 122764 0 04:57 ? 00:00:00 [zabbix_server] <defunct> zabbix 122789 122764 0 04:57 ? 00:00:00 [zabbix_server] <defunct> zabbix 122791 122764 0 04:57 ? 00:00:00 [zabbix_server] <defunct> zabbix 122792 122764 0 04:57 ? 00:00:00 [zabbix_server] <defunct> zabbix 122793 122764 0 04:57 ? 00:00:00 [zabbix_server] <defunct> zabbix 122794 122764 0 04:57 ? 00:00:00 [zabbix_server] <defunct> zabbix 122797 122764 0 04:57 ? 00:00:00 [zabbix_server] <defunct> zabbix 122798 122764 0 04:57 ? 00:00:00 [zabbix_server] <defunct> zabbix 122800 122764 0 04:57 ? 00:00:00 [zabbix_server] <defunct> zabbix 122802 122764 0 04:57 ? 00:00:00 [zabbix_server] <defunct> zabbix 122804 122764 0 04:57 ? 00:00:00 [zabbix_server] <defunct> zabbix 122806 122764 0 04:57 ? 00:00:00 [zabbix_server] <defunct> zabbix 122809 122764 0 04:57 ? 00:00:00 [zabbix_server] <defunct> zabbix 122811 122764 0 04:57 ? 00:00:00 [zabbix_server] <defunct> zabbix 122813 122764 0 04:57 ? 00:00:00 [zabbix_server] <defunct> zabbix 122815 122764 0 04:57 ? 00:00:00 [zabbix_server] <defunct> zabbix 122817 122764 0 04:57 ? 00:00:00 [zabbix_server] <defunct> zabbix 122818 122764 0 04:57 ? 00:00:00 [zabbix_server] <defunct> zabbix 122820 122764 0 04:57 ? 00:00:00 [zabbix_server] <defunct> zabbix 122821 122764 0 04:57 ? 00:00:00 [zabbix_server] <defunct> zabbix 122823 122764 0 04:57 ? 00:00:00 [zabbix_server] <defunct> zabbix 122825 122764 0 04:57 ? 00:00:00 [zabbix_server] <defunct> zabbix 122827 122764 0 04:57 ? 00:00:00 [zabbix_server] <defunct> zabbix 122828 122764 0 04:57 ? 00:00:00 [zabbix_server] <defunct> zabbix 122829 122764 0 04:57 ? 00:00:00 [zabbix_server] <defunct> zabbix 122830 122764 0 04:57 ? 00:00:00 [zabbix_server] <defunct> zabbix 122831 122764 0 04:57 ? 00:00:00 [zabbix_server] <defunct> zabbix 122832 122764 0 04:57 ? 00:00:00 [zabbix_server] <defunct> zabbix 122833 122764 0 04:57 ? 00:00:00 [zabbix_server] <defunct> zabbix 122834 122764 0 04:57 ? 00:00:00 [zabbix_server] <defunct> zabbix 122835 122764 0 04:57 ? 00:00:00 [zabbix_server] <defunct> zabbix 122836 122764 0 04:57 ? 00:00:00 [zabbix_server] <defunct> zabbix 122837 122764 0 04:57 ? 00:00:00 [zabbix_server] <defunct> zabbix 122838 122764 0 04:57 ? 00:00:00 [zabbix_server] <defunct> zabbix 122839 122764 0 04:57 ? 00:00:00 [zabbix_server] <defunct> zabbix 122840 122764 0 04:57 ? 00:00:01 [zabbix_server] <defunct> zabbix 122841 122764 0 04:57 ? 00:00:00 [zabbix_server] <defunct> zabbix 122842 122764 0 04:57 ? 00:00:00 [zabbix_server] <defunct> zabbix 122843 122764 0 04:57 ? 00:00:00 [zabbix_server] <defunct> zabbix 122844 122764 0 04:57 ? 00:00:00 [zabbix_server] <defunct> zabbix 122845 122764 0 04:57 ? 00:00:00 [zabbix_server] <defunct> zabbix 122846 122764 0 04:57 ? 00:00:00 [zabbix_server] <defunct> root 123746 123715 0 05:05 pts/6 00:00:00 grep --color=auto zabbix_server [root@zbx_server1 ~]#
Zabbix Server Log shows like below
[root@zbx_server1v ~]# vi /var/log/zabbix/zabbix_server.log 121959:20190701:045715.470 11: /usr/sbin/zabbix_server: history syncer #10 [processed 2 values, 2 triggers in 0.001238 sec, syncing history](+0xb75f0) [0x56060b6565f0] 121959:20190701:045715.470 10: /usr/sbin/zabbix_server: history syncer #10 [processed 2 values, 2 triggers in 0.001238 sec, syncing history](+0xb8663) [0x56060b657663] 121959:20190701:045715.470 9: /usr/sbin/zabbix_server: history syncer #10 [processed 2 values, 2 triggers in 0.001238 sec, syncing history](+0xb8926) [0x56060b657926] 121959:20190701:045715.470 8: /usr/sbin/zabbix_server: history syncer #10 [processed 2 values, 2 triggers in 0.001238 sec, syncing history](+0xbd5ea) [0x56060b65c5ea] 121959:20190701:045715.470 7: /usr/sbin/zabbix_server: history syncer #10 [processed 2 values, 2 triggers in 0.001238 sec, syncing history](zbx_sync_history_cache+0x8a) [0x56060b65ce40] 121959:20190701:045715.470 6: /usr/sbin/zabbix_server: history syncer #10 [processed 2 values, 2 triggers in 0.001238 sec, syncing history](dbsyncer_thread+0x1ca) [0x56060b5dff3a] 121959:20190701:045715.470 5: /usr/sbin/zabbix_server: history syncer #10 [processed 2 values, 2 triggers in 0.001238 sec, syncing history](zbx_thread_start+0x37) [0x56060b6a3c3c] 121959:20190701:045715.470 4: /usr/sbin/zabbix_server: history syncer #10 [processed 2 values, 2 triggers in 0.001238 sec, syncing history](MAIN_ZABBIX_ENTRY+0xbb3) [0x56060b5d8f70] 121959:20190701:045715.470 3: /usr/sbin/zabbix_server: history syncer #10 [processed 2 values, 2 triggers in 0.001238 sec, syncing history](daemon_start+0x31b) [0x56060b695e5a] 121959:20190701:045715.470 2: /usr/sbin/zabbix_server: history syncer #10 [processed 2 values, 2 triggers in 0.001238 sec, syncing history](main+0x312) [0x56060b5d83bb] 121959:20190701:045715.470 1: /lib64/libc.so.6(__libc_start_main+0xf5) [0x7f603a184b55] 121959:20190701:045715.470 0: /usr/sbin/zabbix_server: history syncer #10 [processed 2 values, 2 triggers in 0.001238 sec, syncing history](+0x38529) [0x56060b5d7529] 121928:20190701:045715.474 One child process died (PID:121959,exitcode/signal:1). Exiting ... zabbix_server [121928]: Error waiting for process with PID 121959: [10] No child processes 121928:20190701:045715.591 syncing history data... 121928:20190701:045715.598 __mem_malloc: skipped 0 asked 72 skip_min 18446744073709551615 skip_max 0 121928:20190701:045715.598 [file:dbcache.c,line:4140] zbx_mem_malloc(): out of memory (requested 72 bytes) 121928:20190701:045715.598 [file:dbcache.c,line:4140] zbx_mem_malloc(): please increase TrendCacheSize configuration parameter 121928:20190701:045715.598 === memory statistics for trend cache === 121928:20190701:045715.598 free chunks of size 24 bytes: 2 121928:20190701:045715.598 free chunks of size 32 bytes: 2 121928:20190701:045715.599 free chunks of size 40 bytes: 2 121928:20190701:045715.599 free chunks of size 48 bytes: 4 121928:20190701:045715.599 free chunks of size 56 bytes: 1 121928:20190701:045715.599 free chunks of size 64 bytes: 1 121928:20190701:045715.599 min chunk size: 24 bytes 121928:20190701:045715.599 max chunk size: 64 bytes 121928:20190701:045715.599 memory of total size 4193928 bytes fragmented into 41421 chunks 121928:20190701:045715.599 of those, 504 bytes are in 12 free chunks 121928:20190701:045715.599 of those, 3530704 bytes are in 41409 used chunks 121928:20190701:045715.599 ================================ 121928:20190701:045715.599 === Backtrace: === 121928:20190701:045715.599 20: /usr/sbin/zabbix_server(zbx_backtrace+0x35) [0x56060b696609] 121928:20190701:045715.599 19: /usr/sbin/zabbix_server(__zbx_mem_malloc+0x163) [0x56060b692040] 121928:20190701:045715.599 18: /usr/sbin/zabbix_server(+0xbf879) [0x56060b65e879] 121928:20190701:045715.599 17: /usr/sbin/zabbix_server(zbx_hashset_insert_ext+0x12e) [0x56060b69b843] 121928:20190701:045715.599 16: /usr/sbin/zabbix_server(zbx_hashset_insert+0x2d) [0x56060b69b713] 121928:20190701:045715.599 15: /usr/sbin/zabbix_server(+0xb75f0) [0x56060b6565f0] 121928:20190701:045715.599 14: /usr/sbin/zabbix_server(+0xb8663) [0x56060b657663] 121928:20190701:045715.599 13: /usr/sbin/zabbix_server(+0xb8926) [0x56060b657926] 121928:20190701:045715.599 12: /usr/sbin/zabbix_server(+0xbd5ea) [0x56060b65c5ea] 121928:20190701:045715.599 11: /usr/sbin/zabbix_server(+0xbdca8) [0x56060b65cca8] 121928:20190701:045715.599 10: /usr/sbin/zabbix_server(+0xbfcd7) [0x56060b65ecd7] 121928:20190701:045715.599 9: /usr/sbin/zabbix_server(free_database_cache+0x35) [0x56060b65ed3d] 121928:20190701:045715.599 8: /usr/sbin/zabbix_server(zbx_on_exit+0x8f) [0x56060b5d9354] 121928:20190701:045715.599 7: /usr/sbin/zabbix_server(+0xf7f19) [0x56060b696f19] 121928:20190701:045715.599 6: /lib64/libpthread.so.0(+0xf290) [0x7f603b324290] 121928:20190701:045715.599 5: /lib64/libpthread.so.0(__wait+0x22) [0x7f603b323d62] 121928:20190701:045715.599 4: /usr/sbin/zabbix_server(MAIN_ZABBIX_ENTRY+0xed5) [0x56060b5d9292] 121928:20190701:045715.599 3: /usr/sbin/zabbix_server(daemon_start+0x31b) [0x56060b695e5a] 121928:20190701:045715.599 2: /usr/sbin/zabbix_server(main+0x312) [0x56060b5d83bb] 121928:20190701:045715.599 1: /lib64/libc.so.6(__libc_start_main+0xf5) [0x7f603a184b55] 121928:20190701:045715.600 0: /usr/sbin/zabbix_server(+0x38529) [0x56060b5d7529] 122764:20190701:045725.708 Starting Zabbix Server. Zabbix 4.0.6 (revision 91685). 122764:20190701:045725.708 ****** Enabled features ****** 122764:20190701:045725.708 SNMP monitoring: YES 122764:20190701:045725.708 IPMI monitoring: YES 122764:20190701:045725.708 Web monitoring: YES 122764:20190701:045725.708 VMware monitoring: YES 122764:20190701:045725.708 SMTP authentication: YES 122764:20190701:045725.708 Jabber notifications: YES 122764:20190701:045725.708 Ez Texting notifications: YES 122764:20190701:045725.708 ODBC: YES 122764:20190701:045725.708 SSH2 support: YES 122764:20190701:045725.708 IPv6 support: YES 122764:20190701:045725.708 TLS support: YES 122764:20190701:045725.708 ****************************** 122764:20190701:045725.708 using configuration file: /etc/zabbix/zabbix_server.conf 122764:20190701:045725.719 current database version (mandatory/optional): 04000000/04000003 122764:20190701:045725.719 required mandatory version: 04000000 122764:20190701:045727.799 server #0 started [main process] 122772:20190701:045727.800 server #1 started [configuration syncer #1] 122773:20190701:045727.800 server #2 started [alerter #1] 122774:20190701:045727.800 server #3 started [alerter #2] 122775:20190701:045727.801 server #4 started [alerter #3] 122776:20190701:045727.801 server #5 started [housekeeper #1] 122777:20190701:045727.801 server #6 started [timer #1] 122778:20190701:045727.802 server #7 started [http poller #1] 122779:20190701:045727.802 server #8 started [discoverer #1] 122781:20190701:045727.803 server #9 started [history syncer #1] 122783:20190701:045727.803 server #10 started [history syncer #2] 122785:20190701:045727.804 server #11 started [history syncer #3] 122787:20190701:045727.805 server #12 started [history syncer #4] 122793:20190701:045727.807 server #16 started [history syncer #8] 122792:20190701:045727.808 server #15 started [history syncer #7] 122795:20190701:045727.808 server #18 started [history syncer #10] 122797:20190701:045727.808 server #19 started [history syncer #11] 122798:20190701:045727.809 server #20 started [history syncer #12] 122789:20190701:045727.809 server #13 started [history syncer #5] 122794:20190701:045727.809 server #17 started [history syncer #9] 122800:20190701:045727.810 server #21 started [history syncer #13] 122802:20190701:045727.810 server #22 started [history syncer #14] 122804:20190701:045727.811 server #23 started [history syncer #15] 122791:20190701:045727.811 server #14 started [history syncer #6] 122809:20190701:045727.813 server #25 started [proxy poller #1] 122811:20190701:045727.814 server #26 started [self-monitoring #1] 122806:20190701:045727.815 server #24 started [escalator #1] 122813:20190701:045727.816 server #27 started [task manager #1] 122817:20190701:045727.816 server #29 started [poller #2] 122815:20190701:045727.817 server #28 started [poller #1] 122820:20190701:045727.818 server #31 started [poller #4] 122818:20190701:045727.819 server #30 started [poller #3] 122821:20190701:045727.819 server #32 started [poller #5] 122830:20190701:045727.820 server #38 started [poller #11] 122833:20190701:045727.826 server #41 started [poller #14] 122823:20190701:045727.827 server #33 started [poller #6] 122837:20190701:045727.829 server #45 started [trapper #2] 122832:20190701:045727.831 server #40 started [poller #13] 122825:20190701:045727.836 server #34 started [poller #7] 122842:20190701:045727.836 server #50 started [alert manager #1] 122831:20190701:045727.839 server #39 started [poller #12] 122829:20190701:045727.840 server #37 started [poller #10] 122840:20190701:045727.840 server #48 started [trapper #5] 122835:20190701:045727.841 server #43 started [unreachable poller #1] 122834:20190701:045727.846 server #42 started [poller #15] 122828:20190701:045727.847 server #36 started [poller #9] 122839:20190701:045727.848 server #47 started [trapper #4] 122827:20190701:045727.851 server #35 started [poller #8] 122838:20190701:045727.851 server #46 started [trapper #3] 122841:20190701:045727.854 server #49 started [icmp pinger #1] 122843:20190701:045727.858 server #51 started [preprocessing manager #1] 122836:20190701:045727.861 server #44 started [trapper #1] 122845:20190701:045728.132 server #53 started [preprocessing worker #2] 122844:20190701:045728.140 server #52 started [preprocessing worker #1] 122846:20190701:045728.140 server #54 started [preprocessing worker #3] 122836:20190701:045856.846 cannot send list of active checks to "xxx.xxx.xxx.xxx": host [zbx_server1.localhost.com] not found 122793:20190701:045905.986 item "xxxx:xen.service" became not supported: Timeout while executing a shell script. 122836:20190701:045912.730 sending configuration data to proxy "PROXY SERVER 1" at "xxx.xxx.xxx.xxx", datalen 10522445 122840:20190701:050018.153 sending configuration data to proxy "PROXY SERVER 2" at "xxx.xxx.xxx.xxx", datalen 15998031 122838:20190701:050056.861 cannot send list of active checks to "xxx.xxx.xxx.xxx": host [zbx_server1.localhost.com] not found 122836:20190701:050256.876 cannot send list of active checks to "xxx.xxx.xxx.xxx": host [zbx_server1.localhost.com] not found 122795:20190701:050437.703 __mem_malloc: skipped 0 asked 72 skip_min 18446744073709551615 skip_max 0 122795:20190701:050437.703 [file:dbcache.c,line:4140] zbx_mem_malloc(): out of memory (requested 72 bytes) 122795:20190701:050437.703 [file:dbcache.c,line:4140] zbx_mem_malloc(): please increase TrendCacheSize configuration parameter 122795:20190701:050437.703 === memory statistics for trend cache === 122795:20190701:050437.703 free chunks of size 24 bytes: 2 122795:20190701:050437.703 free chunks of size 32 bytes: 2 122795:20190701:050437.703 free chunks of size 40 bytes: 2 122795:20190701:050437.703 free chunks of size 48 bytes: 4 122795:20190701:050437.703 free chunks of size 56 bytes: 1 122795:20190701:050437.703 free chunks of size 64 bytes: 1 122795:20190701:050437.703 min chunk size: 24 bytes 122795:20190701:050437.703 max chunk size: 64 bytes 122795:20190701:050437.703 memory of total size 4193928 bytes fragmented into 41421 chunks 122795:20190701:050437.703 of those, 504 bytes are in 12 free chunks 122795:20190701:050437.703 of those, 3530704 bytes are in 41409 used chunks 122795:20190701:050437.703 ================================ 122795:20190701:050437.703 === Backtrace: === 122795:20190701:050437.704 16: /usr/sbin/zabbix_server: history syncer #10 [processed 0 values, 0 triggers in 0.000013 sec, syncing history](zbx_backtrace+0x35) [0x561bb678e609] 122795:20190701:050437.704 15: /usr/sbin/zabbix_server: history syncer #10 [processed 0 values, 0 triggers in 0.000013 sec, syncing history](__zbx_mem_malloc+0x163) [0x561bb678a040] 122795:20190701:050437.704 14: /usr/sbin/zabbix_server: history syncer #10 [processed 0 values, 0 triggers in 0.000013 sec, syncing history](+0xbf879) [0x561bb6756879] 122795:20190701:050437.704 13: /usr/sbin/zabbix_server: history syncer #10 [processed 0 values, 0 triggers in 0.000013 sec, syncing history](zbx_hashset_insert_ext+0x12e) [0x561bb6793843] 122795:20190701:050437.704 12: /usr/sbin/zabbix_server: history syncer #10 [processed 0 values, 0 triggers in 0.000013 sec, syncing history](zbx_hashset_insert+0x2d) [0x561bb6793713] 122795:20190701:050437.704 11: /usr/sbin/zabbix_server: history syncer #10 [processed 0 values, 0 triggers in 0.000013 sec, syncing history](+0xb75f0) [0x561bb674e5f0] 122795:20190701:050437.704 10: /usr/sbin/zabbix_server: history syncer #10 [processed 0 values, 0 triggers in 0.000013 sec, syncing history](+0xb8663) [0x561bb674f663] 122795:20190701:050437.704 9: /usr/sbin/zabbix_server: history syncer #10 [processed 0 values, 0 triggers in 0.000013 sec, syncing history](+0xb8926) [0x561bb674f926] 122795:20190701:050437.704 8: /usr/sbin/zabbix_server: history syncer #10 [processed 0 values, 0 triggers in 0.000013 sec, syncing history](+0xbd5ea) [0x561bb67545ea] 122795:20190701:050437.704 7: /usr/sbin/zabbix_server: history syncer #10 [processed 0 values, 0 triggers in 0.000013 sec, syncing history](zbx_sync_history_cache+0x8a) [0x561bb6754e40] 122795:20190701:050437.704 6: /usr/sbin/zabbix_server: history syncer #10 [processed 0 values, 0 triggers in 0.000013 sec, syncing history](dbsyncer_thread+0x1ca) [0x561bb66d7f3a] 122795:20190701:050437.704 5: /usr/sbin/zabbix_server: history syncer #10 [processed 0 values, 0 triggers in 0.000013 sec, syncing history](zbx_thread_start+0x37) [0x561bb679bc3c] 122795:20190701:050437.704 4: /usr/sbin/zabbix_server: history syncer #10 [processed 0 values, 0 triggers in 0.000013 sec, syncing history](MAIN_ZABBIX_ENTRY+0xbb3) [0x561bb66d0f70] 122795:20190701:050437.704 3: /usr/sbin/zabbix_server: history syncer #10 [processed 0 values, 0 triggers in 0.000013 sec, syncing history](daemon_start+0x31b) [0x561bb678de5a] 122795:20190701:050437.704 2: /usr/sbin/zabbix_server: history syncer #10 [processed 0 values, 0 triggers in 0.000013 sec, syncing history](main+0x312) [0x561bb66d03bb] 122795:20190701:050437.704 1: /lib64/libc.so.6(__libc_start_main+0xf5) [0x7fe2345a3b55] 122795:20190701:050437.704 0: /usr/sbin/zabbix_server: history syncer #10 [processed 0 values, 0 triggers in 0.000013 sec, syncing history](+0x38529) [0x561bb66cf529] 122764:20190701:050437.707 One child process died (PID:122795,exitcode/signal:1). Exiting ...
Kindly suggest how can i fixed the same.
Thanks for your kind help.
OS: OEL - 7
Zabbix : zabbix-server-pgsql-4.0.6-1.el7.x86_64
zabbix-web-pgsql-4.0.6-1.el7.noarch
zabbix-web-4.0.6-1.el7.noarch
Proxy : 2
PHP 5.4.16
PostgreSQL : 11