[ZBX-15685] zabbix_sender may report successful sending of data while actually there is an error Created: 2019 Feb 19  Updated: 2024 Apr 10  Resolved: 2019 Oct 15

Status: Closed
Project: ZABBIX BUGS AND ISSUES
Component/s: Agent (G)
Affects Version/s: 3.0.25, 4.0.5rc1
Fix Version/s: 4.0.14rc1, 4.2.8rc1, 4.4.1rc1, 5.0.0alpha1, 5.0 (plan)

Type: Problem report Priority: Trivial
Reporter: Andris Mednis Assignee: Artjoms Rimdjonoks
Resolution: Fixed Votes: 0
Labels: openssl, sender, signal, zabbix_sender
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

GNU/Linux

Team: Team A
Sprint: Sprint 56 (Sep 2019), Sprint 57 (Oct 2019)
Story Points: 1

 Description   

If Zabbix server or proxy closes connection during TLS handshake then zabbix_sender may receive SIGPIPE signal.
Although there is a signal handler in data sending thread of zabbix_sender the SIGPIPE is not handled.
zabbix_sender returns 0 (success), but server/proxy has accepted no data at all.

This issue was noticed when working on support of LibreSSL and OpenSSL with no PSK.

Steps to reproduce: (Note: see a comment below for much simpler way to reproduce)

  1. Compile OpenSSL 1.1.1a with no PSK support.
  2. Compile Zabbix server to work with this special OpenSSL. As changes are not yet merged you need to take svn://svn.zabbix.com/branches/dev/ZBX-15552-30 branch which can handle OpenSSL with no PSK support.
  3. Run server with this special OpenSSL.
  4. Run standard zabbix_sender from a machine where zabbix_sender uses usual OpenSSL 1.1.1a (with PSK support enabled). Use connecting with PSK.
  5. Run zabbix_sender multiple times. Results are varying from time to time: 
    $ zabbix_sender -z 192.168.1.2 -s "host1" --tls-connect psk --tls-psk-identity=testid --tls-psk-file=/home/andris/zabbix_agentd.psk -k trapper_test -o abc
Sending failed. Use option -vv for more detailed output.
$ zabbix_sender -z 192.168.1.2 -s "host1" --tls-connect psk --tls-psk-identity=testid --tls-psk-file=/home/andris/zabbix_agentd.psk -k trapper_test -o abc
sent: 1; skipped: 0; total: 1

Expected:
zabbix_sender always reports error.

Could be OpenSSL-specific. It works as expected if zabbix sender uses GnuTLS 3.6.6. Also works if zabbix_sender sends to proxy which uses LibreSSL 2.8.2 (no PSK support).

 Comments   
Comment by Andris Mednis [ 2019 Feb 19 ]

This change DOES NOT help:

Index: src/zabbix_sender/zabbix_sender.c
===================================================================
--- src/zabbix_sender/zabbix_sender.c	(revision 89987)
+++ src/zabbix_sender/zabbix_sender.c	(working copy)
@@ -469,6 +469,7 @@
 	signal(SIGTERM, send_signal_handler);
 	signal(SIGQUIT, send_signal_handler);
 	signal(SIGALRM, send_signal_handler);
+	signal(SIGPIPE, send_signal_handler);
 #endif
 	switch (configured_tls_connect_mode)
 	{
Comment by Andris Mednis [ 2019 Oct 02 ]

There is much simpler way how to reproduce this error.

Preparation:

  1. Compile server and zabbix sender as usual, with system-wide OpenSSL (no need to have PSK disabled in OpenSSL).
  2. Configure a test host in frontend, with PSK encryption. Create a trapper item for testing.
  3. Test with zabbix_sender that sending data for the trapper item works with PSK encryption.

Now invoke zabbix sender with a wrong PSK identity. Observe that sending fails:

$ bin/zabbix_sender -z 192.168.6.68 -p 10091 -s "it00" --tls-connect psk --tls-psk-identity=testid --tls-psk-file=/home/zabbix/zabbix_agentd.psk -k trapper_test -o abc -vv
zabbix_sender [15508]: DEBUG: In zbx_tls_init_child()
zabbix_sender [15508]: DEBUG: OpenSSL library (version OpenSSL 1.1.1d  10 Sep 2019) initialized
zabbix_sender [15508]: DEBUG: zbx_tls_init_child() loaded PSK identity "testid"
zabbix_sender [15508]: DEBUG: zbx_tls_init_child() loaded PSK from file "/home/zabbix/zabbix_agentd.psk"
zabbix_sender [15508]: DEBUG: zbx_tls_init_child() PSK ciphersuites: TLS_CHACHA20_POLY1305_SHA256 TLS_AES_128_GCM_SHA256 ECDHE-PSK-AES128-CBC-SHA256 ECDHE-PSK-AES128-CBC-SHA PSK-AES128-GCM-SHA256 PSK-AES128-CCM8 PSK-AES128-CCM PSK-AES128-CBC-SHA256 PSK-AES128-CBC-SHA
zabbix_sender [15508]: DEBUG: End of zbx_tls_init_child()
zabbix_sender [15509]: DEBUG: In zbx_tls_connect(): psk_identity:"testid"
zabbix_sender [15509]: DEBUG: zbx_psk_client_cb() requested PSK identity "testid"
zabbix_sender [15509]: DEBUG: End of zbx_tls_connect():SUCCEED (established TLSv1.3 TLS_CHACHA20_POLY1305_SHA256)
zabbix_sender [15509]: Warning: SSL_shutdown() with 192.168.6.68 set result code to 1: file ../ssl/ssl_lib.c line 2086: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
zabbix_sender [15509]: DEBUG: send value error: TLS read set result code to 1: file ../ssl/record/rec_layer_s3.c line 1543: error:1409445C:SSL routines:ssl3_read_bytes:tlsv13 alert certificate required: SSL alert number 116: TLS read fatal alert "unknown"
Sending failed.

Modify zabbix_sender by adding an artificial delay:

diff --git a/src/zabbix_sender/zabbix_sender.c b/src/zabbix_sender/zabbix_sender.c
index a59ba4b562..e4ac8369d1 100644
--- a/src/zabbix_sender/zabbix_sender.c
+++ b/src/zabbix_sender/zabbix_sender.c
@@ -504,7 +504,7 @@ static      ZBX_THREAD_ENTRY(send_value, args)
                        zbx_json_adduint64(&sendval_args->json, ZBX_PROTO_TAG_CLOCK, ts.sec);
                        zbx_json_adduint64(&sendval_args->json, ZBX_PROTO_TAG_NS, ts.ns);
                }
-
+zbx_sleep(1);
                if (SUCCEED == (tcp_ret = zbx_tcp_send(&sock, sendval_args->json.buffer)))
                {
                        if (SUCCEED == (tcp_ret = zbx_tcp_recv(&sock)))

Recompile.
Invoke zabbix sender again with a wrong PSK identity. Observe that sending succeeds:

$ bin/zabbix_sender -z 192.168.6.68 -p 10091 -s "it00" --tls-connect psk --tls-psk-identity=testid --tls-psk-file=/home/zabbix/zabbix_agentd.psk -k trapper_test -o abc -vv
zabbix_sender [15580]: DEBUG: In zbx_tls_init_child()
zabbix_sender [15580]: DEBUG: OpenSSL library (version OpenSSL 1.1.1d  10 Sep 2019) initialized
zabbix_sender [15580]: DEBUG: zbx_tls_init_child() loaded PSK identity "testid"
zabbix_sender [15580]: DEBUG: zbx_tls_init_child() loaded PSK from file "/home/zabbix/zabbix_agentd.psk"
zabbix_sender [15580]: DEBUG: zbx_tls_init_child() PSK ciphersuites: TLS_CHACHA20_POLY1305_SHA256 TLS_AES_128_GCM_SHA256 ECDHE-PSK-AES128-CBC-SHA256 ECDHE-PSK-AES128-CBC-SHA PSK-AES128-GCM-SHA256 PSK-AES128-CCM8 PSK-AES128-CCM PSK-AES128-CBC-SHA256 PSK-AES128-CBC-SHA
zabbix_sender [15580]: DEBUG: End of zbx_tls_init_child()
zabbix_sender [15581]: DEBUG: In zbx_tls_connect(): psk_identity:"testid"
zabbix_sender [15581]: DEBUG: zbx_psk_client_cb() requested PSK identity "testid"
zabbix_sender [15581]: DEBUG: End of zbx_tls_connect():SUCCEED (established TLSv1.3 TLS_CHACHA20_POLY1305_SHA256)
sent: 1; skipped: 0; total: 1

The errror is OpenSSL-specific.
If Zabbix server is compiled with GnuTLS, then zabbix_sender works as expected, regardless of using OpenSSL or GnuTLS.

Comment by Artjoms Rimdjonoks [ 2019 Oct 14 ]

Available in versions:

Generated at Wed May 08 20:24:14 EEST 2024 using Jira 9.12.4#9120004-sha1:625303b708afdb767e17cb2838290c41888e9ff0.