[ZBX-18545] Quarterly "TLS connection has been closed during handshake" Created: 2020 Oct 22  Updated: 2025 Jun 16  Resolved: 2025 Jun 16

Status: Closed
Project: ZABBIX BUGS AND ISSUES
Component/s: Agent (G), Server (S)
Affects Version/s: 4.0.24
Fix Version/s: None

Type: Problem report Priority: Trivial
Reporter: IBM iX DevOps Assignee: Andrei Gushchin (Inactive)
Resolution: Fixed Votes: 0
Labels: crash, items, performance, tls
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:
  • CentOS 7
  • Postgresql 9.2
  • openssl-1.0.2k-19.el7.x86_64
  • Microsoft Hyper-V Hypervisor
  • VM-Data
    • SSD-Disk
    • 16 CPU Cores
    • 7,5 GiB Memory
    • 1 GiB Swap
  • Most items are active so Zabbix-Server mostly receives only

Perfomance data:

  • 566 hosts
  • 78390 items
  • 24843 triggers
  • ~340 values/sec

Attachments: PNG File zabbix-tls-handshake-tcpdump.png    

 Description   

Steps to reproduce:

  1. Currently no possible as it happens at random times about every 3 months

Result:

  • Lots of messages with 
    21471:20201022:125748.424 failed to accept an incoming connection: from ***.***.***.***: TLS connection has been closed during handshake:
21471:20201022:125748.424 failed to accept an incoming connection: from ***.***.***.***: TLS connection has been closed during handshake:
21471:20201022:125748.424 failed to accept an incoming connection: from ***.***.***.***: TLS connection has been closed during handshake:
  • Test with openssl s_client and given PSK identity and key result in 
    SSL handshake has read 0 bytes and written 289 bytes
  • Higher Debug-Levels reveal nothing specific: 
    End of zbx_tls_accept():FAIL error:'TLS connection has been closed during handshake:'
  • tcpdump + Wireshark show incoming TLS-Handshakes with Client-Hello, no Response from Zabbix-Server except a TCP connection closings with "encrypted alert 21" and lots of FIN_WAIT1 tcp connections, around 600.
  • Code-Part seems to use openssl as Library. Return path for this error gets SSL_ERROR_ZERO_RETURN which obviously seem to suggest handshake problems without any further specifics.
  • Zabbix-Proxy can still receive agent data (passive) but not send it to the Zabbix-Server
  • The only solutions seems to be to do the following steps
    1. Block incoming connections
    2. Restart prozess or server/VM
    3. Increasingly allow more and more clients to connect
  • A different "solution" is to just wait 3-5 hours until the problem magically disappears as fast as it started out nothing.

 Comments   
Comment by Andrei Gushchin (Inactive) [ 2020 Oct 27 ]

Thank you it really weird behaviour.
Can you share the Wireshark and debug level log here?
Does this happens only for zabbix server or some proxy as well?

Comment by IBM iX DevOps [ 2022 Jun 28 ]

Currently the issue is no longer present.

Comment by IBM iX DevOps [ 2023 May 02 ]

Another importand information, for people who found this issue while haveing the same problem. The ultimate cause seems to be a security/vulnerability scan from OpenVAS on the Zabbix-Server port. IT's not clear what a security-scan might cause in the Zabbix code the other connections fail, but it perfectly correlates with our security-scan.

Comment by IBM iX DevOps [ 2023 May 02 ]

Maybe some furthe log messages will help narrowing down the problem:

1752:20230502:164734.578 failed to accept an incoming connection: from 1.2.3.4: reading first byte from connection failed: [104] Connection reset by peer
...
1759:20230502:164757.287 failed to accept an incoming connection: from 1.2.3.4: TLS handshake set result code to 1: file s3_srvr.c line 1435: error:1408A0C1:SSL routines:ssl3_get_client_hello:no shared cipher: TLS write fatal alert "handshake failure"
  1752:20230502:164757.573 failed to accept an incoming connection: from 1.2.3.4: TLS handshake set result code to 1: file s3_srvr.c line 1435: error:1408A0C1:SSL routines:ssl3_get_client_hello:no shared cipher: TLS write fatal alert "handshake failure"

These pop up, when OpenVAS is duing its security-scan.

Comment by IBM iX DevOps [ 2023 May 02 ]

I'm not allowed to share any dumps or log file. I can show you the wireshark view of the tcpdump I took:

Unfortunately there is not much to see, ecnrypted alert is 21 and then a RST is done and agreed upon.

Comment by IBM iX DevOps [ 2023 May 02 ]

We currently still use Zabbix 4.0.45 on this node.

Comment by IBM iX DevOps [ 2023 Jun 12 ]

The issue seems to be, that the OpenVAS security scan "fills" all or certain StartTrappers (used for receiving the data from active agent checks). After reading the Zabbix server configuration documentation, we change the StartTrappers parameter to a higher value and restarted the Zabbix server. Followed by an increase of max_connections in Postgresql. This immediately cleared the error messages and made all agents available again.

Unfortunately the error messages in the Zabbix server log where not very helpful in finding the cause in having to few StartTrappers. It is still not clear, why internal traffic had issues and external traffic did not.

In summary, increasing StartTrappers helped clearing up TLS connections errors.

Comment by Jan Prusinowski [ 2025 May 27 ]

Is this issue still valid? If no actions will be taken this issue will be closed in 2 weeks time due to inactivity.

Generated at Wed Jul 16 09:33:09 EEST 2025 using Jira 9.12.4#9120004-sha1:625303b708afdb767e17cb2838290c41888e9ff0.