[ZBX-15027] Proxy leaves hanging child processes when using invalid certificate Created: 2018 Oct 18  Updated: 2024 Apr 10  Resolved: 2018 Nov 02

Status: Closed
Project: ZABBIX BUGS AND ISSUES
Component/s: Agent (G), Proxy (P), Server (S)
Affects Version/s: 3.0.22, 3.4.14, 4.0.0
Fix Version/s: 3.0.24rc1, 3.4.15rc1, 4.0.2rc1, 4.2.0alpha1, 4.2 (plan)

Type: Problem report Priority: Trivial
Reporter: Anna Kucenko (Inactive) Assignee: Vladislavs Sokurenko
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: File rhelhang.log     HTML File zabbix_ca_file     File zabbix_proxy.log     File zabbix_proxy_dimiru.conf     File zabbix_server.crt     File zabbix_server.key    
Issue Links:
Duplicate
duplicates ZBX-15048 Zombie processes occur when stopping ... Closed
Sub-task
part of ZBX-9867 PidFile location in .conf file ignore... Closed
Team: Team A
Team: Team A
Sprint: Sprint 45, Sprint 46, Nov 2018
Story Points: 3

 Description   

Steps to reproduce: Import incorrect certificate
Result: Proxy process doesn’t start

 Comments   
Comment by richlv [ 2018 Oct 18 ]

What's the issue with the certificate?

Comment by Andris Mednis [ 2018 Oct 18 ]

I think the reason is using encrypted private key, it is not supported. Try unencrypted key.

Comment by Andris Mednis [ 2018 Oct 18 ]

With DebugLevel=3 in server log file:

 Enter PEM pass phrase:
  8118:20181018:122217.052 cannot load private key from file "/home/zabbix30/ZBX-14856/zabbix_server.key": file ../crypto/ui/ui_lib.c line 543: error:2807106B:UI routines:UI_process:processing error: while reading strings file ../crypto/pem/pem_lib.c line 59: error:0906406D:PEM routines:PEM_def_callback:problems getting password file ../crypto/pem/pem_pkey.c line 64: error:0907B068:PEM routines:PEM_read_bio_PrivateKey:bad password read file ../ssl/ssl_rsa.c line 556: error:140B0009:SSL routines:SSL_CTX_use_PrivateKey_file:PEM lib

While it may seem "not user friendly", it shows all technical details from OpenSSL side.
It could be a good idea to add it to
https://www.zabbix.com/documentation/3.0/manual/encryption/troubleshooting/certificate_problems

Comment by dimir [ 2018 Oct 19 ]

I can confirm the issue. Just use attached files for proxy:

TLSAccept=cert
TLSCAFile=/etc/zabbix/certs/zabbix_ca_file
TLSServerCertIssuer=bebebeb
TLSServerCertSubject=bebebebe
TLSCertFile=/etc/zabbix/certs/zabbix_server.crt
TLSKeyFile=/etc/zabbix/certs/zabbix_server.key

And try to start it until it leaves hanging uncontrolled child processes that can only be killed with -KILL.

Reproduced about every 5th time.

The log:

3728:20181019:144915.316 zbx_tls_init_child() loaded CA certificate(s) from file "/etc/zabbix/certs/zabbix_ca_file"
3728:20181019:144915.319 zbx_tls_init_child() loaded certificate(s) from file "/etc/zabbix/certs/zabbix_server.crt"
Enter PEM pass phrase:
3728:20181019:144915.320 cannot load private key from file "/etc/zabbix/certs/zabbix_server.key": file ../crypto/ui/ui_lib.c line 493: error:2807106B:UI routines:UI_process:processing error: while reading strings file ../crypto/pem/pem_lib.c line 64: error:0906406D:PEM routines:PEM_def_callback:problems getting password file ../crypto/pem/pem_pkey.c line 63: error:0907B068:PEM routines:PEM_read_bio_PrivateKey:bad password read file ../ssl/ssl_rsa.c line 550: error:140B0009:SSL routines:SSL_CTX_use_PrivateKey_file:PEM lib
3736:20181019:144915.321 proxy #16 started [poller #7]
3712:20181019:144915.322 One child process died (PID:3728,exitcode/signal:1). Exiting ...
3712:20181019:144915.322 zbx_on_exit() called
3716:20181019:144915.322 Got signal [signal:15(SIGTERM),sender_pid:3712,sender_uid:1000,reason:0]. Exiting ...
3718:20181019:144915.323 Got signal [signal:15(SIGTERM),sender_pid:3712,sender_uid:1000,reason:0]. Exiting ...
3725:20181019:144915.323 Got signal [signal:15(SIGTERM),sender_pid:3712,sender_uid:1000,reason:0]. Exiting ...
3719:20181019:144915.325 Got signal [signal:15(SIGTERM),sender_pid:3712,sender_uid:1000,reason:0]. Exiting ...
3724:20181019:144915.325 Got signal [signal:15(SIGTERM),sender_pid:3712,sender_uid:1000,reason:0]. Exiting ...
3720:20181019:144915.325 Got signal [signal:15(SIGTERM),sender_pid:3712,sender_uid:1000,reason:0]. Exiting ...
3734:20181019:144915.325 proxy #14 started [poller #5]
3717:20181019:144915.326 Got signal [signal:15(SIGTERM),sender_pid:3712,sender_uid:1000,reason:0]. Exiting ...
3727:20181019:144915.326 Got signal [signal:15(SIGTERM),sender_pid:3712,sender_uid:1000,reason:0]. Exiting ...
3732:20181019:144915.326 Got signal [signal:15(SIGTERM),sender_pid:3712,sender_uid:1000,reason:0]. Exiting ...
3741:20181019:144915.326 proxy #21 started [trapper #1]
3738:20181019:144915.326 proxy #18 started [poller #9]
3742:20181019:144915.326 proxy #22 started [trapper #2]
3742:20181019:144915.326 In zbx_tls_init_child()
3730:20181019:144915.327 Got signal [signal:15(SIGTERM),sender_pid:3712,sender_uid:1000,reason:0]. Exiting ...
3733:20181019:144915.327 Got signal [signal:15(SIGTERM),sender_pid:3712,sender_uid:1000,reason:0]. Exiting ...
3731:20181019:144915.327 Got signal [signal:15(SIGTERM),sender_pid:3712,sender_uid:1000,reason:0]. Exiting ...
zabbix_proxy [3712]: Error waiting for process with PID 3728: [10] No child processes
3740:20181019:144915.328 Got signal [signal:15(SIGTERM),sender_pid:3712,sender_uid:1000,reason:0]. Exiting ...
3738:20181019:144915.328 Got signal [signal:15(SIGTERM),sender_pid:3712,sender_uid:1000,reason:0]. Exiting ...
3736:20181019:144915.328 Got signal [signal:15(SIGTERM),sender_pid:3712,sender_uid:1000,reason:0]. Exiting ...

What's left:

$ ps axuwww | grep zabbix_proxy
vl 3712 0.0 0.0 142124 8488 ? S 14:49 0:00 sbin/zabbix_proxy -c /etc/zabbix/zabbix_proxy.conf
vl 3734 0.0 0.0 145092 6980 ? S 14:49 0:00 sbin/zabbix_proxy -c /etc/zabbix/zabbix_proxy.conf
vl 3735 0.0 0.0 145220 7008 ? S 14:49 0:00 sbin/zabbix_proxy -c /etc/zabbix/zabbix_proxy.conf
vl 3736 0.0 0.0 0 0 ? Z 14:49 0:00 [zabbix_proxy] <defunct>
vl 3737 0.0 0.0 142124 2316 ? S 14:49 0:00 sbin/zabbix_proxy -c /etc/zabbix/zabbix_proxy.conf
vl 3738 0.0 0.0 0 0 ? Z 14:49 0:00 [zabbix_proxy] <defunct>
vl 3739 0.0 0.0 142124 2316 ? S 14:49 0:00 sbin/zabbix_proxy -c /etc/zabbix/zabbix_proxy.conf
vl 3740 0.0 0.0 0 0 ? Z 14:49 0:00 [zabbix_proxy] <defunct>
vl 3741 0.0 0.0 142124 2316 ? S 14:49 0:00 sbin/zabbix_proxy -c /etc/zabbix/zabbix_proxy.conf
vl 3742 0.0 0.0 142124 6320 ? S 14:49 0:00 sbin/zabbix_proxy -c /etc/zabbix/zabbix_proxy.conf
vl 3743 0.0 0.0 142124 2316 ? S 14:49 0:00 sbin/zabbix_proxy -c /etc/zabbix/zabbix_proxy.conf
vl 3744 0.0 0.0 142124 2316 ? S 14:49 0:00 sbin/zabbix_proxy -c /etc/zabbix/zabbix_proxy.conf
vl 3745 0.0 0.0 142124 2316 ? S 14:49 0:00 sbin/zabbix_proxy -c /etc/zabbix/zabbix_proxy.conf
vl 3746 0.0 0.0 142124 2316 ? S 14:49 0:00 sbin/zabbix_proxy -c /etc/zabbix/zabbix_proxy.conf
Comment by dimir [ 2018 Oct 19 ]

Assigning to andris

Comment by Vladislavs Sokurenko [ 2018 Oct 23 ]

Issue also reproduced on 3.4 and I believe in all versions, it's not related to certificates but to stopping server during startup, here is what I get on 3.4 error is different though.

11066:20181023:142242.504 Starting Zabbix Proxy (passive) [Zabbix proxy]. Zabbix 3.4.15rc1 (revision {ZABBIX_REVISION}).
 11066:20181023:142242.504 **** Enabled features ****
 11066:20181023:142242.504 SNMP monitoring:       YES
 11066:20181023:142242.504 IPMI monitoring:       YES
 11066:20181023:142242.504 Web monitoring:        YES
 11066:20181023:142242.504 VMware monitoring:     YES
 11066:20181023:142242.504 ODBC:                  YES
 11066:20181023:142242.504 SSH2 support:           NO
 11066:20181023:142242.504 IPv6 support:           NO
 11066:20181023:142242.504 TLS support:           YES
 11066:20181023:142242.504 **************************
 11066:20181023:142242.504 using configuration file: /home/vso/Documents/zabbix_proxy_dimiru.conf
 11066:20181023:142242.506 current database version (mandatory/optional): 03040000/03040007
 11066:20181023:142242.506 required mandatory version: 03040000
 11066:20181023:142242.508 proxy #0 started [main process]
 11068:20181023:142242.508 proxy #1 started [housekeeper #1]
 11069:20181023:142242.508 proxy #2 started [http poller #1]
 11070:20181023:142242.509 proxy #3 started [discoverer #1]
 11071:20181023:142242.509 proxy #4 started [history syncer #1]
 11072:20181023:142242.509 proxy #5 started [history syncer #2]
 11073:20181023:142242.509 proxy #6 started [history syncer #3]
 11074:20181023:142242.510 proxy #7 started [history syncer #4]
 11075:20181023:142242.510 proxy #8 started [java poller #1]
 11076:20181023:142242.510 proxy #9 started [java poller #2]
 11079:20181023:142242.511 proxy #12 started [java poller #5]
Enter PEM pass phrase:
 11075:20181023:142242.512 cannot load private key from file "/etc/zabbix/certs/zabbix_server.key": file crypto/ui/ui_lib.c line 493: error:2807106B:UI routines:UI_process:processing error: while reading strings file crypto/pem/pem_lib.c line 66: error:0906406D:PEM routines:PEM_def_callback:problems getting password file crypto/pem/pem_pkey.c line 63: error:0907B068:PEM routines:PEM_read_bio_PrivateKey:bad password read file ssl/ssl_rsa.c line 550: error:140B0009:SSL routines:SSL_CTX_use_PrivateKey_file:PEM lib
Enter PEM pass phrase:
 11076:20181023:142242.512 cannot load private key from file "/etc/zabbix/certs/zabbix_server.key": file crypto/ui/ui_lib.c line 493: error:2807106B:UI routines:UI_process:processing error: while reading strings file crypto/pem/pem_lib.c line 66: error:0906406D:PEM routines:PEM_def_callback:problems getting password file crypto/pem/pem_pkey.c line 63: error:0907B068:PEM routines:PEM_read_bio_PrivateKey:bad password read file ssl/ssl_rsa.c line 550: error:140B0009:SSL routines:SSL_CTX_use_PrivateKey_file:PEM lib
Enter PEM pass phrase:
 11079:20181023:142242.512 cannot load private key from file "/etc/zabbix/certs/zabbix_server.key": file crypto/ui/ui_lib.c line 493: error:2807106B:UI routines:UI_process:processing error: while reading strings file crypto/pem/pem_lib.c line 66: error:0906406D:PEM routines:PEM_def_callback:problems getting password file crypto/pem/pem_pkey.c line 63: error:0907B068:PEM routines:PEM_read_bio_PrivateKey:bad password read file ssl/ssl_rsa.c line 550: error:140B0009:SSL routines:SSL_CTX_use_PrivateKey_file:PEM lib
 11087:20181023:142242.513 proxy #20 started [poller #6]
 11089:20181023:142242.513 proxy #22 started [poller #8]
 11085:20181023:142242.514 proxy #18 started [poller #4]
 11080:20181023:142242.514 proxy #13 started [self-monitoring #1]
 11081:20181023:142242.514 proxy #14 started [task manager #1]
 11066:20181023:142242.514 One child process died (PID:11075,exitcode/signal:1). Exiting ...
Enter PEM pass phrase:
 11090:20181023:142242.517 proxy #23 started [poller #9]
 11088:20181023:142242.517 proxy #21 started [poller #7]
 11083:20181023:142242.517 proxy #16 started [poller #2]
 11077:20181023:142242.518 proxy #10 started [java poller #3]
 11081:20181023:142242.518 cannot load private key from file "/etc/zabbix/certs/zabbix_server.key": file crypto/ui/ui_lib.c line 493: error:2807106B:UI routines:UI_process:processing error: while reading strings file crypto/pem/pem_lib.c line 66: error:0906406D:PEM routines:PEM_def_callback:problems getting password file crypto/pem/pem_pkey.c line 63: error:0907B068:PEM routines:PEM_read_bio_PrivateKey:bad password read file ssl/ssl_rsa.c line 550: error:140B0009:SSL routines:SSL_CTX_use_PrivateKey_file:PEM lib
Enter PEM pass phrase:
 11090:20181023:142242.533 cannot load private key from file "/etc/zabbix/certs/zabbix_server.key": file crypto/ui/ui_lib.c line 493: error:2807106B:UI routines:UI_process:processing error: while reading strings file crypto/pem/pem_lib.c line 66: error:0906406D:PEM routines:PEM_def_callback:problems getting password file crypto/pem/pem_pkey.c line 63: error:0907B068:PEM routines:PEM_read_bio_PrivateKey:bad password read file ssl/ssl_rsa.c line 550: error:140B0009:SSL routines:SSL_CTX_use_PrivateKey_file:PEM lib
 11066:20181023:142244.514 syncing history data...
 11066:20181023:142244.514 syncing history data done
 11066:20181023:142244.514 Zabbix Proxy stopped. Zabbix 3.4.15rc1 (revision {ZABBIX_REVISION}).
 11095:20181023:142245.512 Starting Zabbix Proxy (passive) [Zabbix proxy]. Zabbix 3.4.15rc1 (revision {ZABBIX_REVISION}).
 11095:20181023:142245.512 **** Enabled features ****
 11095:20181023:142245.512 SNMP monitoring:       YES
 11095:20181023:142245.512 IPMI monitoring:       YES
 11095:20181023:142245.512 Web monitoring:        YES
 11095:20181023:142245.512 VMware monitoring:     YES
 11095:20181023:142245.512 ODBC:                  YES
 11095:20181023:142245.512 SSH2 support:           NO
 11095:20181023:142245.512 IPv6 support:           NO
 11095:20181023:142245.512 TLS support:           YES
 11095:20181023:142245.512 **************************
 11095:20181023:142245.512 using configuration file: /home/vso/Documents/zabbix_proxy_dimiru.conf
 11095:20181023:142245.514 current database version (mandatory/optional): 03040000/03040007
 11095:20181023:142245.514 required mandatory version: 03040000
 11095:20181023:142245.516 listener failed: bind() for [[-]:10051] failed: [98] Address already in use
 11099:20181023:142248.520 Starting Zabbix Proxy (passive) [Zabbix proxy]. Zabbix 3.4.15rc1 (revision {ZABBIX_REVISION}).
 11099:20181023:142248.520 **** Enabled features ****
 11099:20181023:142248.520 SNMP monitoring:       YES
 11099:20181023:142248.520 IPMI monitoring:       YES
 11099:20181023:142248.520 Web monitoring:        YES
 11099:20181023:142248.520 VMware monitoring:     YES
 11099:20181023:142248.520 ODBC:                  YES
 11099:20181023:142248.520 SSH2 support:           NO
 11099:20181023:142248.520 IPv6 support:           NO
 11099:20181023:142248.520 TLS support:           YES
 11099:20181023:142248.520 **************************
 11099:20181023:142248.520 using configuration file: /home/vso/Documents/zabbix_proxy_dimiru.conf
 11099:20181023:142248.522 current database version (mandatory/optional): 03040000/03040007
 11099:20181023:142248.522 required mandatory version: 03040000
 11099:20181023:142248.524 listener failed: bind() for [[-]:10051] failed: [98] Address already in use

11085 ?        S      0:00 ./sbin/zabbix_proxy -c /home/vso/Documents/zabbix_proxy_dimiru.conf

Related issue:
ZBX-9867

Comment by Vladislavs Sokurenko [ 2018 Oct 24 ]

Currently this solves the issue for me:

Index: src/zabbix_server/poller/checks_snmp.c
===================================================================
--- src/zabbix_server/poller/checks_snmp.c	(revision 86000)
+++ src/zabbix_server/poller/checks_snmp.c	(working copy)
@@ -2151,7 +2151,15 @@
 
 void	zbx_init_snmp(void)
 {
+	sigset_t	mask, orig_mask;
+
+	sigemptyset(&mask);
+	sigaddset(&mask, SIGTERM);
+	sigprocmask(SIG_BLOCK, &mask, &orig_mask);
+
 	init_snmp(progname);
+
+	sigprocmask(SIG_SETMASK, &orig_mask, NULL);
 }

It helps to avoid following error:

#0  0x00007f000bca963f in pthread_rwlock_wrlock () from /lib64/libpthread.so.0
#1  0x00007f0009d566cd in CRYPTO_THREAD_write_lock () from /lib64/libcrypto.so.1.1
#2  0x00007f0009d0b0c4 in OBJ_NAME_remove () from /lib64/libcrypto.so.1.1
#3  0x00007f0009cff2d9 in OPENSSL_LH_doall () from /lib64/libcrypto.so.1.1
#4  0x00007f0009d0b31b in OBJ_NAME_cleanup () from /lib64/libcrypto.so.1.1
#5  0x00007f0009ceeac2 in evp_cleanup_int () from /lib64/libcrypto.so.1.1
#6  0x00007f0009cfd307 in OPENSSL_cleanup () from /lib64/libcrypto.so.1.1
#7  0x00007f0007f80b27 in __cxa_finalize () from /lib64/libc.so.6
#8  0x00007f0009c08187 in __do_global_dtors_aux () from /lib64/libcrypto.so.1.1
#9  0x00007ffe6c0509a0 in ?? ()
#10 0x00007f000c4e88e6 in _dl_fini () from /lib64/ld-linux-x86-64.so.2
Backtrace stopped: frame did not save the PC

#0  0x00007f5a2c06f26c in __lll_lock_wait_private () from /lib64/libc.so.6
#1  0x00007f5a2bfe60c1 in _IO_flush_all_lockp () from /lib64/libc.so.6
#2  0x00007f5a2bfe6345 in _IO_cleanup () from /lib64/libc.so.6
#3  0x00007f5a2bfa16da in __run_exit_handlers () from /lib64/libc.so.6
#4  0x00007f5a2bfa171c in exit () from /lib64/libc.so.6
#5  0x00000000005e82cf in exit_with_failure () at sighandler.c:46
#6  0x00000000005e8673 in terminate_signal_handler (sig=15, siginfo=0x7ffedff603f0, 
    context=0x7ffedff602c0) at sighandler.c:122
#7  <signal handler called>
#8  0x00007f5a2bfe4e6e in __GI__IO_link_in () from /lib64/libc.so.6
#9  0x00007f5a2bfe3c12 in _IO_new_file_init_internal () from /lib64/libc.so.6
#10 0x00007f5a2bfd83a8 in __fopen_internal () from /lib64/libc.so.6
#11 0x00007f5a2ec76be8 in ?? () from /lib64/libnetsnmp.so.30
#12 0x00007f5a2ec751bd in netsnmp_read_module () from /lib64/libnetsnmp.so.30
#13 0x00007f5a2ec638b5 in netsnmp_init_mib () from /lib64/libnetsnmp.so.30
#14 0x00007f5a2ec89a96 in init_snmp () from /lib64/libnetsnmp.so.30
#15 0x0000000000441b9b in zbx_init_snmp () at checks_snmp.c:2154
#16 0x0000000000453dad in poller_thread (args=0x7ffedff60bc0) at poller.c:901
#17 0x00000000005f11f5 in zbx_thread_start (handler=0x453bc1 <poller_thread>, 
    thread_args=0x7ffedff60bc0) at threads.c:133
#18 0x000000000041cb7e in MAIN_ZABBIX_ENTRY (flags=0) at proxy.c:1086
#19 0x00000000005e6ddb in daemon_start (allow_root=0, user=0x0, flags=0) at daemon.c:392
#20 0x000000000041bfa3 in main (argc=3, argv=0x2970370) at proxy.c:870

#0  0x00007fb2d3cb226c in __lll_lock_wait_private () from /lib64/libc.so.6
#1  0x00007fb2d3be450b in __run_exit_handlers () from /lib64/libc.so.6
#2  0x00007fb2d3be471c in exit () from /lib64/libc.so.6
#3  0x0000000000631c06 in exit_with_failure () at sighandler.c:46
#4  0x0000000000631faa in terminate_signal_handler (sig=15, siginfo=0x7ffd42648ef0, context=0x7ffd42648dc0)
    at sighandler.c:122
#5  <signal handler called>
#6  0x00007fb2d3be4a90 in __cxa_finalize () from /lib64/libc.so.6
#7  0x00007fb2d240d107 in __do_global_dtors_aux () from /lib64/libnss3.so
#8  0x00007ffd426496b0 in ?? ()
#9  0x00007fb2d814c8e6 in _dl_fini () from /lib64/ld-linux-x86-64.so.2
Backtrace stopped: frame did not save the PC
Comment by Vladislavs Sokurenko [ 2018 Oct 25 ]

This might also be related to OpenSSL 1.1.0 release.

Comment by Vladislavs Sokurenko [ 2018 Oct 30 ]

Fixed in:

  • 3.0.24rc1 r86131
  • 3.4.15rc1 r86132
  • 4.0.2rc1 r86133
  • 4.2.0alpha1 (trunk) r86134
Generated at Fri Apr 19 19:52:08 EEST 2024 using Jira 9.12.4#9120004-sha1:625303b708afdb767e17cb2838290c41888e9ff0.