Steps to reproduce:

We have updated our Zabbix server as well as the agents on different machines from version 3.4 to 4.0.19. Now in the server logs we get the following error messages:

16716:20200401:150916.713 failed to accept an incoming connection: from XXXXXXX: TLS handshake set result code to 1: file ../ssl/record/ssl3_record.c line 677: error:1408F119:SSL routines:ssl3_get_record:decryption failed or bad record mac: TLS write fatal alert "bad record mac"

Correspondingly on the agent we get:

30511:20200401:152818.546 failed to accept an incoming connection: from XXX: TLS handshake set result code to 1: file ../ssl/statem/extensions.c line 1617: error:141FA0FD:SSL routines:tls_psk_do_binder:binder does not verify: TLS write fatal alert "illegal parameter"

We are using PSK encryption and have double checked that incoming and outgoing encryption is checked in the admin frontend. Also we have checked that the TLSPSKIdentity is identical with the PSK identity in the web form and the TLSPSKFile has the same content as the PSK field. The pre shared key is a hex string with 64 characters. In the server and agent configuration we have not settings for TLSCipherPSK or similar.

To reduce potential problems on all systems Ubuntu 18.04 was installed so the Zabbix components are linked with libssl.so.1.1 (/usr/lib/x86_64-linux-gnu/libssl.so.1.1).