[ZBX-18159] TLS handshake fails with 5.-Server after cert was temporary enabled - probably TLSv1.3 session reuse Created: 2020 Jul 29 Updated: 2025 Apr 11 Resolved: 2025 Apr 11 |
|
Status: | Closed |
Project: | ZABBIX BUGS AND ISSUES |
Component/s: | Server (S) |
Affects Version/s: | None |
Fix Version/s: | None |
Type: | Problem report | Priority: | Trivial |
Reporter: | Stefan | Assignee: | Zabbix Support Team |
Resolution: | Fixed | Votes: | 0 |
Labels: | None | ||
Remaining Estimate: | Not Specified | ||
Time Spent: | Not Specified | ||
Original Estimate: | Not Specified |
Description |
Steps to reproduce:
server.conf is
AlertScriptsPath=/usr/lib/zabbix/alertscripts CacheSize= 512M DBHost=localhost DBName=zabbix DBPassword=known DBUser=zabbix ExternalScripts=/storage/externalscripts Fping6Location=/usr/bin/fping6 FpingLocation=/usr/bin/fping HistoryCacheSize=512M HistoryIndexCacheSize=128M LogFileSize=0 LogFile=/var/log/zabbix/zabbix_server.log LogSlowQueries=3000 PidFile=/var/run/zabbix/zabbix_server.pid SSHKeyLocation=/storage/.ssh StartIPMIPollers=20 StartPingers=10 StartPollers=700 StartDiscoverers=20 StartPollersUnreachable=70 StartTrappers=100 Timeout=10 TrendCacheSize=128M ValueCacheSize=1G JavaGateway=127.0.0.1 JavaGatewayPort=10052 StartJavaPollers=5 #TLSCAFile=/etc/zabbix/ca.pem #TLSCertFile=/etc/zabbix/cert.pem #TLSKeyFile=/etc/zabbix/key.pem
TLSAccept=psk
Monitor zabbix_server.log:
8510:20200729:111900.275 failed to accept an incoming connection: from IPV6-ADDRESS: TLS handshake set result code to 1: file ../ssl/statem/extensions.c line 1604: error:141FA0FD:SSL routines:tls_psk_do_binder:binder does not verify: TLS write fatal alert "illegal parameter"
Looks like it has to do with TLSv1.3 session reusage. Both Server and Client is Ubuntu 18. # zabbix_server -V zabbix_server (Zabbix) 5.0.2 Revision 352ca05870 13 July 2020, compilation time: Jul 13 2020 07:00:00Copyright (C) 2020 Zabbix SIA License GPLv2+: GNU GPL version 2 or later <http://gnu.org/licenses/gpl.html>. This is free software: you are free to change and redistribute it according to the license. There is NO WARRANTY, to the extent permitted by law.This product includes software developed by the OpenSSL Project for use in the OpenSSL Toolkit (http://www.openssl.org/).Compiled with OpenSSL 1.1.1 11 Sep 2018 Running with OpenSSL 1.1.1 11 Sep 2018
zabbix_agentd -V zabbix_agentd (daemon) (Zabbix) 4.0.23 Revision 9f3d212e4a 27 July 2020, compilation time: Jul 27 2020 07:00:00Copyright (C) 2020 Zabbix SIA License GPLv2+: GNU GPL version 2 or later <http://gnu.org/licenses/gpl.html>. This is free software: you are free to change and redistribute it according to the license. There is NO WARRANTY, to the extent permitted by law.This product includes software developed by the OpenSSL Project for use in the OpenSSL Toolkit (http://www.openssl.org/).Compiled with OpenSSL 1.1.1 11 Sep 2018 Running with OpenSSL 1.1.1 11 Sep 2018
It started at the time, we accepted for testing purpose and just for a few minutes, on server side also CERT-based authentication in addition to PSK. Before that, we had no log issues like the one above.
Server still receives data from client. It currently "only" floods our log-files massively. |
Comments |
Comment by Eduards Matuls (Inactive) [ 2020 Jul 30 ] |
Hello! If you have an agent with only active checks and you want to use PSK authentication, then you should set TLSConnect=psk in agent configuration. TLSAccept=psk is requesting for PSK encryption only for incoming connections from server side, which are passive checks. Please review your configuration and update the issue. Regards, |
Comment by Stefan [ 2020 Jul 30 ] |
Hi Eduard, client already has TLSConnect=psk. I just skipped this config part when reporting. But we also have TLSAccept=psk because we want to use some passive checks in the future as well. I see nothing wrong in our config so far. |
Comment by Stefan [ 2020 Aug 04 ] |
Eduards,
can i provide more infos for you? It's not only the big amount of logs, spawning our files, quite often, some data is lost due to temporary connection issues. |
Comment by Stefan [ 2020 Aug 06 ] |
Please re-compile the ubuntu 18 server packages with recend openssl-libs. Seems to be the issue: Zabbix 5.0.2-Debian 10 server-packages (no problem): Compiled with OpenSSL 1.1.1d 10 Sep 2019 Zabbix 5.0.2-Ubuntu 18 server-packages (problem): Compiled with OpenSSL 1.1.1 11 Sep 2018 Problem just started when Ubuntu 18 clients have been updated. (see https://support.zabbix.com/browse/ZBX-18005) Thank you. |
Comment by Stefan [ 2020 Aug 12 ] |
Can i provide further infos? Would be happy to get this addresses
Thank you! |
Comment by Stefan [ 2020 Sep 15 ] |
Any news on this? |
Comment by Jurijs Klopovskis [ 2020 Sep 15 ] |
We are building against what's provided by the distribution. Even if we build against 1.1.1d it will still be running with 1.1.1 that is shipped with 18.04. I'll try to get some attention. |
Comment by Stefan [ 2020 Dec 01 ] |
@yurii - do you have any idea, what we can do?
Even after updating to a more recent Version of zabbix-server, the problem persists (now ubuntu 20) # zabbix_server -V zabbix_server (Zabbix) 5.0.6 Revision 93895db26b 30 November 2020, compilation time: Nov 30 2020 08:11:40Copyright (C) 2020 Zabbix SIA License GPLv2+: GNU GPL version 2 or later <http://gnu.org/licenses/gpl.html>. This is free software: you are free to change and redistribute it according to the license. There is NO WARRANTY, to the extent permitted by law.This product includes software developed by the OpenSSL Project for use in the OpenSSL Toolkit (http://www.openssl.org/). Compiled with OpenSSL 1.1.1f 31 Mar 2020 Running with OpenSSL 1.1.1f 31 Mar 2020
Logs still report
23389:20201201:115341.356 failed to accept an incoming connection: from 2003:OUR-IP: TLS handshake set result code to 1: file ../ssl/statem/extensions.c line 1616: error:141FA0FD:SSL routines:tls_psk_do_binder:binder does not verify: TLS write fatal alert "illegal parameter" 23341:20201201:115341.362 failed to accept an incoming connection: from 2003:OUR-IP: TLS handshake set result code to 1: file ../ssl/statem/extensions.c line 1616: error:141FA0FD:SSL routines:tls_psk_do_binder:binder does not verify: TLS write fatal alert "illegal parameter" |
Comment by Stefan [ 2021 Jan 22 ] |
Still an issue. |
Comment by Glebs Ivanovskis [ 2021 Jan 27 ] |
Dear siegmarb, these messages look to me like a generic accepting side errors when TLS 1.3 is involved. $ openssl version OpenSSL 1.1.1f 31 Mar 2020 Client side: $ openssl s_client -psk 0123 CONNECTED(00000003) 140097705981248:error:14094417:SSL routines:ssl3_read_bytes:sslv3 alert illegal parameter:../ssl/record/rec_layer_s3.c:1543:SSL alert number 47 --- no peer certificate available --- No client certificate CA names sent --- SSL handshake has read 7 bytes and written 415 bytes Verification: OK --- New, (NONE), Cipher is (NONE) Secure Renegotiation IS NOT supported Compression: NONE Expansion: NONE No ALPN negotiated Early data was not sent Verify return code: 0 (ok) --- Server side: $ openssl s_server -psk 012345 -nocert Using default temp DH parameters ACCEPT ERROR 140598090466624:error:141FA0FD:SSL routines:tls_psk_do_binder:binder does not verify:../ssl/statem/extensions.c:1616: shutting down SSL CONNECTION CLOSED I think you have a PSK conflict in your configuration, i.e. different PSK values for same PSK identity on different hosts/proxies. This would explain "data loss due to temporary connection issues" you report. Please check Zabbix server's logs for messages like conflicting PSK values for PSK identity "identity" on hosts "host1" and "host2" (and maybe others) You also may be interested in voting for ZBXNEXT-3777 if this turns out the root cause. |
Comment by Anton [ 2021 Jun 19 ] |
Good day! I can confirm that this problem occurs when using the same TLSPSKIdentity for the proxy and the agent on this proxy. P.S. Also because of this, it is possible to regularly trigger the "Zabbix configuration synchronization processes more than 75% of employees"
zabbix-server 5.4.1 - from repository, Ubuntu 20.04 zabbix-proxy 5.4.1 - from repository, Ubuntu 18.04 zabbix-agent 5.4.1 - from repository, Ubuntu 18.04 |
Comment by Frank I [ 2021 Jul 06 ] |
Hello, I had the same error on zabbix proxy (v.5.4), particularly with connection coming from windows server (not sure if this is relevant here). Changing the PSK to 64 bit helped immediately: openssl rand -hex 64 I have seen this issue here: https://www.zabbix.com/forum/zabbix-help/387608-1-ssl-alert-number-47-tls-read-fatal-alert-illegal-parameter If this is true (in my case it was), I think it is worth updating documentation on that regard. Thanks, Franko
|
Comment by Stefan [ 2021 Sep 17 ] |
I might have identified the REAL issue in our case. Zabbix throws above errors, if the same Identity, is used by a one or many hosts with different PSK. As we have thousands of hosts in zabbix, if an administrator sets the wrong PSK in single host, zabbix shows above error for all hosts. e.g: Host1: Identity-> Key*1*, PSK 12345 Host2: Identity-> Key*1*, PSK 67890
Hosts that do not show error:
Host3: Identity-> Key*2*, PSK 55555 |
Comment by Stefan [ 2021 Sep 20 ] |
We could identify the rotten egg with
select h.host, h.tls_accept, h.tls_psk, h.tls_psk_identity from hosts as h where h.tls_psk_identity = 'Key1';
Strangely, even though the PSK setting in Zabbix DID NOT match the PSK on the host itself, communication was working. Even after changing the PSK in Zabbix, still working... |