zabbix_server.log
[ZBX-18112] ERROR [file:lld_trigger.c,line:1855] Something impossible has just happened. Created: 2020 Jul 19 Updated: 2025 Apr 22 |
|
| Status: | Need info |
| Project: | ZABBIX BUGS AND ISSUES |
| Component/s: | Server (S) |
| Affects Version/s: | None |
| Fix Version/s: | None |
| Type: | Problem report | Priority: | Trivial |
| Reporter: | Dawood Sajjadi | Assignee: | Andrei Gushchin (Inactive) |
| Resolution: | Unresolved | Votes: | 0 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Environment: |
Ubuntu 16.04.3 |
||
| Attachments: |
zabbix_server.log
|
| Description |
|
Hi, About 7 months ago, I reported an issue in our Zabbix server (https://bit.ly/3ja8sKJ) and one of Zabbix engineers suggested to address this issue by upgrading our Zabbix engine from 4.2 to 4.4. I upgraded the server successfully to 4.4.10, however, the issue still persists and in many occurrences, the housekeeping processes stuck and no real-time graph will be generated in Zabbix web console. Since the error message is not informative at all (i,e,. something impossible just happened), I am not sure what is the best way to pinpoint and resolve the issue. The server information is as follows:
# zabbix_server -V*
{color:#ff0000}zabbix_server (Zabbix) 4.4.10{color}
Revision 4db30afc70 29 June 2020, compilation time: Jun 29 2020 12:01:31
Copyright (C) 2020 Zabbix SIA
License GPLv2+: GNU GPL version 2 or later <[http://gnu.org/licenses/gpl.html|https://gnu.org/licenses/gpl.html]>.
This is free software: you are free to change and redistribute it according to
the license. There is NO WARRANTY, to the extent permitted by law.
This product includes software developed by the OpenSSL Project for use in the OpenSSL Toolkit ([http://www.openssl.org/|https://www.openssl.org/]). Compiled with OpenSSL 1.0.2g-fips 1 Mar 2016 Running with OpenSSL 1.0.2g 1 Mar 2016
# ldd $(which zabbix_server)* linux-vdso.so.1 => (0x00007ffe7dda9000) libmysqlclient.so.20 => /usr/lib/x86_64-linux-gnu/libmysqlclient.so.20 (0x00007fa5b0f88000) libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007fa5b0d6b000) libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007fa5b0b51000) libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007fa5b0848000) librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007fa5b0640000) libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007fa5b043c000) libxml2.so.2 => /usr/lib/x86_64-linux-gnu/libxml2.so.2 (0x00007fa5b0081000) libodbc.so.2 => /usr/lib/x86_64-linux-gnu/libodbc.so.2 (0x00007fa5afe18000) libnetsnmp.so.30 => /usr/lib/x86_64-linux-gnu/libnetsnmp.so.30 (0x00007fa5afb3b000) libssh2.so.1 => /usr/lib/x86_64-linux-gnu/libssh2.so.1 (0x00007fa5af912000) libOpenIPMI.so.0 => /usr/lib/libOpenIPMI.so.0 (0x00007fa5af604000) libOpenIPMIposix.so.0 => /usr/lib/libOpenIPMIposix.so.0 (0x00007fa5af3fe000) libevent-2.0.so.5 => /usr/lib/x86_64-linux-gnu/libevent-2.0.so.5 (0x00007fa5af1b8000) libssl.so.1.0.0 => /lib/x86_64-linux-gnu/libssl.so.1.0.0 (0x00007fa5aef4f000) libcrypto.so.1.0.0 => /lib/x86_64-linux-gnu/libcrypto.so.1.0.0 (0x00007fa5aeb0a000) libldap_r-2.4.so.2 => /usr/lib/x86_64-linux-gnu/libldap_r-2.4.so.2 (0x00007fa5ae8b9000) liblber-2.4.so.2 => /usr/lib/x86_64-linux-gnu/liblber-2.4.so.2 (0x00007fa5ae6aa000) libcurl.so.4 => /usr/lib/x86_64-linux-gnu/libcurl.so.4 (0x00007fa5ae43b000) libresolv.so.2 => /lib/x86_64-linux-gnu/libresolv.so.2 (0x00007fa5ae220000) libpcre.so.3 => /lib/x86_64-linux-gnu/libpcre.so.3 (0x00007fa5adfb0000) libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fa5adbe6000) libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007fa5ad864000) libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007fa5ad64e000) /lib64/ld-linux-x86-64.so.2 (0x00007fa5b1ae1000) libicuuc.so.55 => /usr/lib/x86_64-linux-gnu/libicuuc.so.55 (0x00007fa5ad2ba000) liblzma.so.5 => /lib/x86_64-linux-gnu/liblzma.so.5 (0x00007fa5ad098000) libltdl.so.7 => /usr/lib/x86_64-linux-gnu/libltdl.so.7 (0x00007fa5ace8e000) libgcrypt.so.20 => /lib/x86_64-linux-gnu/libgcrypt.so.20 (0x00007fa5acbad000) libOpenIPMIutils.so.0 => /usr/lib/libOpenIPMIutils.so.0 (0x00007fa5ac9a4000) libsasl2.so.2 => /usr/lib/x86_64-linux-gnu/libsasl2.so.2 (0x00007fa5ac789000) libgssapi.so.3 => /usr/lib/x86_64-linux-gnu/libgssapi.so.3 (0x00007fa5ac548000) libgnutls.so.30 => /usr/lib/x86_64-linux-gnu/libgnutls.so.30 (0x00007fa5ac218000) libidn.so.11 => /usr/lib/x86_64-linux-gnu/libidn.so.11 (0x00007fa5abfe5000) librtmp.so.1 => /usr/lib/x86_64-linux-gnu/librtmp.so.1 (0x00007fa5abdc9000) libgssapi_krb5.so.2 => /usr/lib/x86_64-linux-gnu/libgssapi_krb5.so.2 (0x00007fa5abb7f000) libicudata.so.55 => /usr/lib/x86_64-linux-gnu/libicudata.so.55 (0x00007fa5aa0c8000) libgpg-error.so.0 => /lib/x86_64-linux-gnu/libgpg-error.so.0 (0x00007fa5a9eb4000) libheimntlm.so.0 => /usr/lib/x86_64-linux-gnu/libheimntlm.so.0 (0x00007fa5a9cab000) libkrb5.so.26 => /usr/lib/x86_64-linux-gnu/libkrb5.so.26 (0x00007fa5a9a21000) libasn1.so.8 => /usr/lib/x86_64-linux-gnu/libasn1.so.8 (0x00007fa5a977f000) libcom_err.so.2 => /lib/x86_64-linux-gnu/libcom_err.so.2 (0x00007fa5a957b000) libhcrypto.so.4 => /usr/lib/x86_64-linux-gnu/libhcrypto.so.4 (0x00007fa5a9348000) libroken.so.18 => /usr/lib/x86_64-linux-gnu/libroken.so.18 (0x00007fa5a9132000) libp11-kit.so.0 => /usr/lib/x86_64-linux-gnu/libp11-kit.so.0 (0x00007fa5a8ece000) libtasn1.so.6 => /usr/lib/x86_64-linux-gnu/libtasn1.so.6 (0x00007fa5a8cbb000) libnettle.so.6 => /usr/lib/x86_64-linux-gnu/libnettle.so.6 (0x00007fa5a8a85000) libhogweed.so.4 => /usr/lib/x86_64-linux-gnu/libhogweed.so.4 (0x00007fa5a8852000) libgmp.so.10 => /usr/lib/x86_64-linux-gnu/libgmp.so.10 (0x00007fa5a85d2000) libkrb5.so.3 => /usr/lib/x86_64-linux-gnu/libkrb5.so.3 (0x00007fa5a8300000) libk5crypto.so.3 => /usr/lib/x86_64-linux-gnu/libk5crypto.so.3 (0x00007fa5a80d1000) libkrb5support.so.0 => /usr/lib/x86_64-linux-gnu/libkrb5support.so.0 (0x00007fa5a7ec6000) libwind.so.0 => /usr/lib/x86_64-linux-gnu/libwind.so.0 (0x00007fa5a7c9d000) libheimbase.so.1 => /usr/lib/x86_64-linux-gnu/libheimbase.so.1 (0x00007fa5a7a8e000) libhx509.so.5 => /usr/lib/x86_64-linux-gnu/libhx509.so.5 (0x00007fa5a7843000) libsqlite3.so.0 => /usr/lib/x86_64-linux-gnu/libsqlite3.so.0 (0x00007fa5a756e000) libcrypt.so.1 => /lib/x86_64-linux-gnu/libcrypt.so.1 (0x00007fa5a7336000) libffi.so.6 => /usr/lib/x86_64-linux-gnu/libffi.so.6 (0x00007fa5a712e000) libkeyutils.so.1 => /lib/x86_64-linux-gnu/libkeyutils.so.1 (0x00007fa5a6f2a000) ---- The server is a VM with 16xCPUs and 32 GB RAM. Also, I have set the StartPollers value to 20 and increased default Cache Size values as well, but still has the issue. Appreciate any feedback on this subject.
CacheSize=4096M
HistoryCacheSize={color:#de350b}1024M{color}
HistoryIndexCacheSize={color:#de350b}1024M{color}
TrendCacheSize={color:#de350b}1024M{color}
ValueCacheSize={color:#de350b}1024M{color}
|
| Comments |
| Comment by Andrei Gushchin (Inactive) [ 2020 Jul 22 ] |
|
Thank you for reporting this issue. |
| Comment by Dawood Sajjadi [ 2020 Jul 22 ] |
|
Hi Andrea, I attached the zabbix server log. Also, your requested output is as follows. please let me know if you any further information.
*# ps ax | grep zabbix* 1090 ? Sl 1:08 java -server -Dlogback.configurationFile=/etc/zabbix/zabbix_java_gateway_logback.xml -classpath lib:lib/android-json-4.3_r3.1.jar:lib/logback-classic-0.9.27.jar:lib/logback-core-0.9.27.jar:lib/slf4j-api-1.6.1.jar:bin/zabbix-java-gateway-3.4.4.jar -Dzabbix.pidFile=/var/run/zabbix/zabbix_java_gateway.pid -Dzabbix.timeout=3 -Dsun.rmi.transport.tcp.responseTimeout=3000 com.zabbix.gateway.JavaGateway 1128 ? S 0:00 /usr/sbin/zabbix_agentd -c /etc/zabbix/zabbix_agentd.conf 1137 ? S 0:15 /usr/sbin/zabbix_agentd: collector [idle 1 sec] 1138 ? S 0:24 /usr/sbin/zabbix_agentd: listener #1 [waiting for connection] 1139 ? S 0:22 /usr/sbin/zabbix_agentd: listener #2 [waiting for connection] 1141 ? S 0:22 /usr/sbin/zabbix_agentd: listener #3 [waiting for connection] 1142 ? S 0:04 /usr/sbin/zabbix_agentd: active checks #1 [idle 1 sec] 14890 ? S 0:00 sh -c /usr/bin/fping -C3 -i10 2>&1 </tmp/zabbix_server_29025.pinger;/usr/bin/fping6 -C3 -i10 2>&1 </tmp/zabbix_server_29025.pinger; 14917 ? S 0:00 sh -c /usr/bin/fping -C3 -i10 2>&1 </tmp/zabbix_server_29026.pinger;/usr/bin/fping6 -C3 -i10 2>&1 </tmp/zabbix_server_29026.pinger; 14954 pts/0 S+ 0:00 grep zabbix 28940 ? S 0:00 /usr/sbin/zabbix_server -c /etc/zabbix/zabbix_server.conf 28943 ? S 5:28 /usr/sbin/zabbix_server: configuration syncer [synced configuration in 3.007965 sec, idle 60 sec] 28969 ? S 0:10 /usr/sbin/zabbix_server: housekeeper [deleted 228433 hist/trends, 0 items/triggers, 2499 events, 10 sessions, 0 alarms, 39 audit items, 0 records in 93.268271 sec, idle for 1 hour(s)] 28970 ? S 0:01 /usr/sbin/zabbix_server: timer #1 [updated 0 hosts, suppressed 0 events in 0.000311 sec, idle 59 sec] 28971 ? S 0:01 /usr/sbin/zabbix_server: http poller #1 [got 0 values in 0.000362 sec, idle 5 sec] 28972 ? S 0:00 /usr/sbin/zabbix_server: discoverer #1 [processed 0 rules in 0.000376 sec, idle 60 sec] 28973 ? S 3:08 /usr/sbin/zabbix_server: history syncer #1 [processed 394 values, 565 triggers in 0.041933 sec, idle 1 sec] 28974 ? S 3:07 /usr/sbin/zabbix_server: history syncer #2 [processed 27 values, 14 triggers in 0.003253 sec, idle 1 sec] 28975 ? S 3:10 /usr/sbin/zabbix_server: history syncer #3 [processed 1 values, 0 triggers in 0.006432 sec, idle 1 sec] 28976 ? S 3:10 /usr/sbin/zabbix_server: history syncer #4 [processed 0 values, 0 triggers in 0.000099 sec, idle 1 sec] 28977 ? S 0:04 /usr/sbin/zabbix_server: escalator #1 [processed 0 escalations in 0.000625 sec, idle 3 sec] 28978 ? S 0:00 /usr/sbin/zabbix_server: java poller #1 [got 0 values in 0.000026 sec, idle 5 sec] 28979 ? S 0:00 /usr/sbin/zabbix_server: java poller #2 [got 0 values in 0.000011 sec, idle 5 sec] 28980 ? S 0:00 /usr/sbin/zabbix_server: java poller #3 [got 0 values in 0.000024 sec, idle 5 sec] 28981 ? S 0:00 /usr/sbin/zabbix_server: java poller #4 [got 0 values in 0.000019 sec, idle 5 sec] 28982 ? S 0:00 /usr/sbin/zabbix_server: java poller #5 [got 0 values in 0.000029 sec, idle 5 sec] 28983 ? S 0:00 /usr/sbin/zabbix_server: proxy poller #1 [exchanged data with 0 proxies in 0.000011 sec, idle 5 sec] 28985 ? S 0:01 /usr/sbin/zabbix_server: self-monitoring [processed data in 0.000027 sec, idle 1 sec] 28989 ? S 0:00 /usr/sbin/zabbix_server: task manager [processed 0 task(s) in 0.000285 sec, idle 5 sec] 28990 ? S 2:15 /usr/sbin/zabbix_server: poller #1 [got 197 values in 1.893579 sec, getting values] 28991 ? S 2:14 /usr/sbin/zabbix_server: poller #2 [got 91 values in 0.979638 sec, getting values] 28992 ? S 2:11 /usr/sbin/zabbix_server: poller #3 [got 0 values in 0.000029 sec, getting values] 28995 ? S 2:14 /usr/sbin/zabbix_server: poller #4 [got 84 values in 0.956867 sec, getting values] 28996 ? S 2:14 /usr/sbin/zabbix_server: poller #5 [got 77 values in 1.002101 sec, getting values] 28997 ? S 2:15 /usr/sbin/zabbix_server: poller #6 [got 98 values in 1.894354 sec, getting values] 28998 ? S 2:14 /usr/sbin/zabbix_server: poller #7 [got 84 values in 1.058522 sec, getting values] 28999 ? S 2:15 /usr/sbin/zabbix_server: poller #8 [got 33 values in 1.074356 sec, getting values] 29000 ? S 2:15 /usr/sbin/zabbix_server: poller #9 [got 35 values in 0.976598 sec, getting values] 29001 ? S 2:15 /usr/sbin/zabbix_server: poller #10 [got 37 values in 0.965119 sec, getting values] 29002 ? S 0:04 /usr/sbin/zabbix_server: unreachable poller #1 [got 0 values in 0.000021 sec, idle 5 sec] 29005 ? S 0:49 /usr/sbin/zabbix_server: trapper #1 [processed data in 0.000842 sec, waiting for connection] 29006 ? S 0:52 /usr/sbin/zabbix_server: trapper #2 [processed data in 0.004475 sec, waiting for connection] 29011 ? S 0:49 /usr/sbin/zabbix_server: trapper #3 [processed data in 0.005118 sec, waiting for connection] 29014 ? S 0:49 /usr/sbin/zabbix_server: trapper #4 [processed data in 0.002267 sec, waiting for connection] 29018 ? S 0:51 /usr/sbin/zabbix_server: trapper #5 [processed data in 0.000547 sec, waiting for connection] 29020 ? S 0:09 /usr/sbin/zabbix_server: icmp pinger #1 [got 0 values in 0.000020 sec, idle 1 sec] 29023 ? S 0:09 /usr/sbin/zabbix_server: icmp pinger #2 [got 0 values in 0.000034 sec, idle 1 sec] 29024 ? S 0:09 /usr/sbin/zabbix_server: icmp pinger #3 [got 0 values in 0.000015 sec, idle 1 sec] 29025 ? S 0:09 /usr/sbin/zabbix_server: icmp pinger #4 [pinging hosts] 29026 ? S 0:09 /usr/sbin/zabbix_server: icmp pinger #5 [pinging hosts] 29028 ? S 0:02 /usr/sbin/zabbix_server: alert manager #1 [sent 0, failed 0 alerts, idle 5.002449 sec during 5.002515 sec] 29029 ? S 0:01 /usr/sbin/zabbix_server: alerter #1 [sent 0, failed 0 alerts, idle 51.062953 sec during 51.226172 sec] 29031 ? S 0:01 /usr/sbin/zabbix_server: alerter #2 [sent 0, failed 0 alerts, idle 19.818941 sec during 21.680609 sec] 29032 ? S 0:01 /usr/sbin/zabbix_server: alerter #3 [sent 0, failed 0 alerts, idle 21.498529 sec during 21.718863 sec] 29035 ? S 5:57 /usr/sbin/zabbix_server: preprocessing manager #1 [queued 1, processed 1452 values, idle 4.999917 sec during 5.019544 sec] 29037 ? S 1:06 /usr/sbin/zabbix_server: preprocessing worker #1 started 29038 ? S 0:36 /usr/sbin/zabbix_server: preprocessing worker #2 started 29039 ? S 0:35 /usr/sbin/zabbix_server: preprocessing worker #3 started 29041 ? S 0:04 /usr/sbin/zabbix_server: lld manager #1 [processed 3 LLD rules during 5.147412 sec] 29042 ? S 6:04 /usr/sbin/zabbix_server: lld worker #1 [processed 4 LLD rules, idle 5.579740 sec during 5.861587 sec] 29047 ? S 6:06 /usr/sbin/zabbix_server: lld worker #2 [processed 3 LLD rules, idle 11.600605 sec during 11.824498 sec] 29049 ? S 0:04 /usr/sbin/zabbix_server: alert syncer [queued 0 alerts(s), flushed 0 result(s) in 0.000413 sec, idle 1 sec] |
| Comment by Bartosz Nems [ 2025 Apr 22 ] |
|
Hi, I see that Your ticket was created in 2020, please let Us know if You still require our help. |