ZABBIX 5.0.0 centos8.1.1911

Please provide more information:

• what library are you using - libssh or libssh2
• what authorization method is configured for these items

I deploy zabbix using VMWare (.vmx)
Libssh is already installed

Sometimes it can sometimes fail, unstable

What do I need to fix

arimdjonoks

I cannot currently reproduce the issue using the libssh_0.9.0

1) Could you increase the logging level:

./zabbix_server -R log_level_increase
./zabbix_server -R log_level_increase

,reproduce the issue

then decrease the logging back:

zabbix_server -R log_level_decrease
zabbix_server -R log_level_decrease

and attach the logs?

2) Could you also run the

 ldd  <path to the zabbix_server binary>/zabbix_server

?

[root@appliance ~]# ldd /usr/sbin/zabbix_server
linux-vdso.so.1 (0x00007ffcebaa5000)
libmysqlclient.so.21 => /usr/lib64/mysql/libmysqlclient.so.21 (0x00007f55e6a58000)
libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f55e6838000)
libz.so.1 => /lib64/libz.so.1 (0x00007f55e6621000)
libm.so.6 => /lib64/libm.so.6 (0x00007f55e629f000)
librt.so.1 => /lib64/librt.so.1 (0x00007f55e6096000)
libssl.so.1.1 => /lib64/libssl.so.1.1 (0x00007f55e5e02000)
libcrypto.so.1.1 => /lib64/libcrypto.so.1.1 (0x00007f55e5923000)
libdl.so.2 => /lib64/libdl.so.2 (0x00007f55e571f000)
libxml2.so.2 => /lib64/libxml2.so.2 (0x00007f55e53b7000)
libodbc.so.2 => /lib64/libodbc.so.2 (0x00007f55e5146000)
libnetsnmp.so.35 => /lib64/libnetsnmp.so.35 (0x00007f55e4d9d000)
libssh.so.4 => /lib64/libssh.so.4 (0x00007f55e4b12000)
libOpenIPMI.so.0 => /lib64/libOpenIPMI.so.0 (0x00007f55e47fe000)
libOpenIPMIposix.so.0 => /lib64/libOpenIPMIposix.so.0 (0x00007f55e45f6000)
libevent-2.1.so.6 => /lib64/libevent-2.1.so.6 (0x00007f55e439d000)
libldap-2.4.so.2 => /lib64/libldap-2.4.so.2 (0x00007f55e4150000)
liblber-2.4.so.2 => /lib64/liblber-2.4.so.2 (0x00007f55e3f40000)
libcurl.so.4 => /lib64/libcurl.so.4 (0x00007f55e3cb2000)
libresolv.so.2 => /lib64/libresolv.so.2 (0x00007f55e3a9b000)
libpcre.so.1 => /lib64/libpcre.so.1 (0x00007f55e382a000)
libc.so.6 => /lib64/libc.so.6 (0x00007f55e3467000)
libstdc+.so.6 => /lib64/libstdc+.so.6 (0x00007f55e30d2000)
libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007f55e2eba000)
/lib64/ld-linux-x86-64.so.2 (0x00007f55e78a8000)
liblzma.so.5 => /lib64/liblzma.so.5 (0x00007f55e2c93000)
libltdl.so.7 => /lib64/libltdl.so.7 (0x00007f55e2a89000)
libgssapi_krb5.so.2 => /lib64/libgssapi_krb5.so.2 (0x00007f55e2839000)
libkrb5.so.3 => /lib64/libkrb5.so.3 (0x00007f55e2549000)
libk5crypto.so.3 => /lib64/libk5crypto.so.3 (0x00007f55e232d000)
libcom_err.so.2 => /lib64/libcom_err.so.2 (0x00007f55e2129000)
libOpenIPMIutils.so.0 => /lib64/libOpenIPMIutils.so.0 (0x00007f55e1f1f000)
libgdbm.so.6 => /lib64/libgdbm.so.6 (0x00007f55e1d0f000)
libsasl2.so.3 => /lib64/libsasl2.so.3 (0x00007f55e1af1000)
libnghttp2.so.14 => /lib64/libnghttp2.so.14 (0x00007f55e18cb000)
libidn2.so.0 => /lib64/libidn2.so.0 (0x00007f55e16ad000)
libpsl.so.5 => /lib64/libpsl.so.5 (0x00007f55e149c000)
libbrotlidec.so.1 => /lib64/libbrotlidec.so.1 (0x00007f55e1290000)
libkrb5support.so.0 => /lib64/libkrb5support.so.0 (0x00007f55e107f000)
libkeyutils.so.1 => /lib64/libkeyutils.so.1 (0x00007f55e0e7b000)
libcrypt.so.1 => /lib64/libcrypt.so.1 (0x00007f55e0c52000)
libunistring.so.2 => /lib64/libunistring.so.2 (0x00007f55e08d1000)
libbrotlicommon.so.1 => /lib64/libbrotlicommon.so.1 (0x00007f55e06b1000)
libselinux.so.1 => /lib64/libselinux.so.1 (0x00007f55e0486000)
libpcre2-8.so.0 => /lib64/libpcre2-8.so.0 (0x00007f55e0202000)

zabbix_server.log
192.168.60.133_2020-05-19_23-18-15.zip

It does not look like a Zabbix issue at the moment - it looks like SSH server you are connecting drops the connections.
The exact reason why can only be found by inspecting the SSH server logs.
The most likely reasons are its:
1) "ClientAliveInterval" (or its equivalent) parameter is too small - there seems to be a very large time difference between the machine where SSH server is running(15:18:09) and the machine with Zabbix SSH client (23:22:04). SSH Server may detect the time skew and drop the connection.
2) "ClientAliveCountMax" (or its equivalent) parameter is too small - SSH server may be overwhelmed with SSH connections and it drops the existing sessions.

Could you also verify that you can establish (and hold for some time) several manual SSH connections to the server while this issue. happens ?

I deployed it using the official virtual machine file without making any changes to the SSH configuration

I used SSH. Run to monitor the zabbix server and was not stable myself

Right, I did manage to replicate the issue - I am investigating what could be done about it.

Have you reappeared the problem?If you find a solution, please reply to me, thank you

zabbix_1993
I fixed this issue on my appliance VM after I
1) changed its timezone symlink from the default UTC to my timezone:

ln -s /usr/share/zoneinfo/Europe/Riga /etc/localtime

2) updated the zabbix_server.conf and zabbix_agent.conf Timeout value to 30

Could you try that, restart the VM and checking if issue still persists.

If it still persists - you also try to increase the number of pollers in zabbix_server.conf
StartPollers to 30.
If that also does not work - try to change the interval of SSH checks items to 120 seconds.

UPDATE - Actually those measure above apparently reduces the likelihood but not eliminated the issue (error stopped appearing for like 20 minutes - so I thought it was fixed).
I did deep investigation of the /var/log/secure , /var/log/messages and journalctl and from there is seems like ssh server is processing the requests correctly, however libssh for some reason randomly assumes the channel is closed.
What I experienced is identical to the issue reported on libssh forum here

zabbix server5.0


Zabbix server5.0 added a new ssh.run monitor item, the client 192.168.60.129 environment is centos7, also failed to get the data
zabbix server5.0beta2
but zabbix server5.0 beta2, and the client is 192.168.60.129, get the data normal.

The problem can be reproduced without any involvement of Zabbix on the fresh CentOS 8 installation.
I reported a bug to libssh authors - https://bugs.libssh.org/T231
Essentially there is random chance that libssh will incorrectly report the SSH_ERROR when it shouldn't (as the results of the ssh command execution were returned correctly).
Not much else can be done at this moment, apart from documenting this issue in the Known issues.

Which version of libssh or which version of Zabbix has fixed this problem

zabbix_1993
The raised bug report suggests that libssh team fixed it in their master branch on 23 of june with a commit 750e4f3f. So, you could either use the current libssh master branch or wait for the next 0.9.5 libssh release.

I use this version today : https://launchpad.net/~savoury1/+archive/ubuntu/encryption/+build/20079959/+files/libssh-4_0.9.5-0ubuntu1~20.04.sav0_amd64.deb

I have no more bugs.

Hello, 

I have encountered this issue, the link shared for libssh is not reachable. can you please let me know how I can upgrade to latest libssh version?

Hi,

since I encountered this problem today too I would like to share that I solved it by installing a newer version of libssh (as mentioned by @mma) from here: libssh-releases : David Kedves (launchpad.net).