Steps to reproduce:

1. Upgraded both server and agents from 4.4.x to 5.0.6
2. Using active agent checks and TLS with PSK, on Windows servers

Result:

Server logs occasionally:

10377:20201216:125835.926 failed to accept an incoming connection: from 10.11.22.33: unspecified certificate verification error: TLS handshake set result code to 5:
10374:20201216:125835.928 failed to accept an incoming connection: from 10.22.33.10: unspecified certificate verification error: TLS handshake set result code to 5:
10374:20201216:125907.068 failed to accept an incoming connection: from 10.33.33.8: unspecified certificate verification error: TLS handshake set result code to 5:

At the same time client (10.11.22.33 agent above) logs:

12628:20201216:125833.259 active check data upload to [zabbix-server-ip:10051] started to fail ([connect] TCP successful, cannot establish TLS to [[zabbix-server-ip]:10051]: SSL_connect() timed out)
12628:20201216:125835.938 active check data upload to [zabbix-server-ip:10051] is working again

Expected:
No messages in the logs when agents connect and send data to server.

Other information:

Initially we had server 4.4.10 on Debian Linux 9 (Stretch) and agents 4.4.x, and we didn't have those errors.

Then we first changed the server to a new one with Debian Linux 10 (Buster) with server 4.4.10 (new installation, copied the configurations), and that's when the error messages started.

We then upgraded both server and agents to 5.0.6, but the occasional errors continued. There are less errors though with 5.0.6.

Notable detail is that agents on Linux, on Windows 10 or on Windows Server 2016 do not cause these errors (agents are 4.0.x, 4.4.x or 5.0.6).

Debian 9 server (old server with no problems) openssl version: OpenSSL 1.1.0l 10 Sep 2019

Debian 10 server (current) openssl version: OpenSSL 1.1.1d 10 Sep 2019

Agent TLS configuration:

• TLSConnect=psk
• TLSAccept=psk
• TLSPSKIdentity=XXX
• TLSPSKFile=C:\Program Files\Zabbix Agent\psk.key
• Agents installed with MSI packages from zabbix.com