[#ZBX-11101] CA certificate with empty Subject & Issuer make Zabbix server fail

[ZBX-11101] CA certificate with empty Subject & Issuer make Zabbix server fail Created: 2016 Aug 16 Updated: 2019 May 27 Resolved: 2016 Dec 23
Status:	Closed
Project:	ZABBIX BUGS AND ISSUES
Component/s:	Proxy (P), Server (S)
Affects Version/s:	3.0.4
Fix Version/s:	3.0.8rc1, 3.2.4rc1, 3.4.0alpha1

Type:

Incident report

Priority:

Major

Reporter:

Jean Baptiste Favre

Assignee:

Unassigned

Resolution:

Fixed

Votes:

Labels:

tls, trapper

Remaining Estimate:

Not Specified

Time Spent:

Not Specified

Original Estimate:

Not Specified

Environment:

Debian Jessie, zabbix-server-mysql with configure option --with-openssl

Attachments:

protobix-test-ca-certs.tar.bz2

Issue Links:

Duplicate
is duplicated by	~~ZBX-10970~~	Poller process crash if no certificat...	Closed

Description

While testing Zabbix Server TLS support, I noticed strange logs when connecting to it:

zabbix_server [32180]: ERROR [file:proxy.c,line:2039] Something impossible has just happened.

Items are well received, Zabbix server always answer succees, but all items are trashed:

{'info': 'processed: 0; failed: 1; total: 1; seconds spent: 0.000117', 'response': 'success'}

With DebugLevel >=4 it's even worse. Connection is interrupted without any answer from Zabbix server. Here are the Zabbix Server logs:

32438:20160816:134050.267 log level has been increased to 4 (debug)
32438:20160816:134050.268 __zbx_zbx_setproctitle() title:'trapper #1 [processed data in 0.000000 sec, waiting for connection]'
32438:20160816:134054.865 In zbx_tls_accept()
32438:20160816:134054.886 End of zbx_tls_accept():FAIL error:''
32438:20160816:134054.886 failed to accept an incoming connection: from 127.0.0.1: (null)
32438:20160816:134054.886 __zbx_zbx_setproctitle() title:'trapper #1 [processed data in 0.000000 sec, waiting for connection]'

I tried with another CA+certificate (home made): everything worked well.

The only difference is that my first set of certificate used a home made CA which has an empty issuer & subject.
That sounds weird, but can be easily reproduced with openssl.

Zabbix Server doesn't complain about this strange CA certificate during startup (not sure it should though)
Zabbix Server behave differently between DebugLevel 3 & 4 (it clearly shouldn't)

How to reproduce it

openssl.cnf

dir = ~/test_ca
default_days = 3650

[ req_distinguished_name ]
countryName_default =
stateOrProvinceName_default =
localityName_default =
0.organizationName_default =
organizationalUnitName_default =
commonName_default =
emailAddress_default =

[req]
distinguished_name = req_distinguished_name
x509_extensions	= v3_ca

[ v3_ca ]
subjectKeyIdentifier=hash
authorityKeyIdentifier=keyid:always,issuer
basicConstraints = CA:true

Create the CA certificate with:

openssl req -new -x509 -days 3650 -extensions v3_ca -nodes -newkey rsa:4096 -subj '/' -keyout ~/test_ca/private/cakey.pem -out ~/test_ca/cacert.pem -config ~/test_ca/openssl.cnf

Then create one certificate for Zabbix Server signed with this CA and another one for the client, configure both server and client, use them.
You're done

Comments

Comment by Aleksandrs Saveljevs [ 2016 Aug 17 ]

For completeness, could you please show how you create server and agent certificates using this CA?

Comment by Jean Baptiste Favre [ 2016 Aug 17 ]

I used XCA, which is a QT based software to ease CA management, and created by mistake a CA certificate with empty Subject. Since root CA certificates are self-signed, Issuer is empty too.

Once you created CA certificate with the openssl command I provided in the ticket, I guess you can create any certificate you want the usual way.
I'll check that asap

Regards,
Jean Baptiste Favre

Comment by Jean Baptiste Favre [ 2016 Aug 17 ]

For certificate signature by CA to work, you need to add following lines in openssl.cnf

[ ca ]
default_ca      = CA_default            # The default ca section

[ CA_default ]
dir             = ~/test_ca
certs           = $dir/certs
crl_dir         = $dir/crl
database        = $dir/index.txt
new_certs_dir   = $dir/newcerts
certificate     = $dir/cacert.pem
serial          = $dir/serial
crlnumber       = $dir/crlnumber
crl             = $dir/crl.pem
private_key     = $dir/private/cakey.pem
RANDFILE        = $dir/private/.rand
x509_extensions = usr_cert
name_opt        = ca_default            # Subject Name options
cert_opt        = ca_default            # Certificate field options
default_days    = 365                   # how long to certify for
default_crl_days= 30                    # how long before next CRL
default_md      = default               # use public key default MD
preserve        = no                    # keep passed DN ordering
policy          = policy_anything

[ usr_cert ]
basicConstraints=CA:FALSE
nsComment                       = "OpenSSL Generated Certificate"
subjectKeyIdentifier=hash
authorityKeyIdentifier=keyid,issuer

[ policy_anything ]
countryName             = optional
stateOrProvinceName     = optional
localityName            = optional
organizationName        = optional
organizationalUnitName  = optional
commonName              = supplied
emailAddress            = optional

Then create Zabbix server certificate request with

openssl req -new -nodes -out ./newcerts/zabbix-server.csr.pem -keyout private/zabbix-server.key.pem -config ./zabbix-server.domain.tld.conf

and configuration:

oid_section = xca_oids

[ xca_oids ]
dom = 1.3.6.1.4.1.311.20.2
MsCaV = 1.3.6.1.4.1.311.21.1
msEFSFR = 1.3.6.1.4.1.311.10.3.4.1
iKEIntermediate = 1.3.6.1.5.5.8.2.2
nameDistinguisher = 0.2.262.1.10.7.20
id-kp-eapOverPPP = 1.3.6.1.5.5.7.3.13
id-kp-eapOverLAN = 1.3.6.1.5.5.7.3.14
id-pkkdcekuoid = 1.3.6.1.5.2.3.5

[ req ]
default_bits = 1024
default_keyfile = privkey.pem
distinguished_name = xca_dn
x509_extensions = xca_extensions
req_extensions = xca_extensions
string_mask = MASK:0x2002
utf8 = yes
prompt = no

[ xca_dn ]
0.C=FR
1.ST=Ile de France
2.L=Paris
3.O=protobix
4.OU=monitoring
5.CN=zabbix-server.domain.tld
[email protected]

[ xca_extensions ]
nsComment=xca certificate
nsCertType=server
keyUsage=digitalSignature, nonRepudiation, keyEncipherment
subjectKeyIdentifier=hash
basicConstraints=critical,CA:FALSE

And sign it with your CA

openssl ca -config ./openssl.cnf -out certs/zabbix-server.cert.pem -infiles ./newcerts/zabbix-server.csr.pem

As you can see, Issue field is empty:

> openssl x509 -in certs/zabbix-server.cert.pem -noout -textCertificate:
    Data:
        Version: 3 (0x2)
        Serial Number: 4096 (0x1000)
    Signature Algorithm: sha256WithRSAEncryption
        Issuer: 
        Validity
            Not Before: Aug 17 08:55:46 2016 GMT
            Not After : Aug 17 08:55:46 2017 GMT
        Subject: C=FR, ST=Ile de France, L=Paris, O=protobix, OU=monitoring, CN=zabbix-server.domain.tld/[email protected]
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                Public-Key: (1024 bit)
                Modulus:
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            X509v3 Basic Constraints: 
                CA:FALSE
            Netscape Comment: 
                OpenSSL Generated Certificate
            X509v3 Subject Key Identifier: 
                BF:1C:E0:C9:7C:1E:A6:87:D9:C5:F7:4F:AE:EF:90:89:E2:66:70:4F
            X509v3 Authority Key Identifier: 
                keyid:38:1B:CB:72:07:90:6E:C1:6A:C9:EF:A7:CB:61:A1:D4:B4:B4:8D:EA

Same goes for client with following configuration

oid_section = xca_oids

[ xca_oids ]
dom = 1.3.6.1.4.1.311.20.2
MsCaV = 1.3.6.1.4.1.311.21.1
msEFSFR = 1.3.6.1.4.1.311.10.3.4.1
iKEIntermediate = 1.3.6.1.5.5.8.2.2
nameDistinguisher = 0.2.262.1.10.7.20
id-kp-eapOverPPP = 1.3.6.1.5.5.7.3.13
id-kp-eapOverLAN = 1.3.6.1.5.5.7.3.14
id-pkkdcekuoid = 1.3.6.1.5.2.3.5

[ req ]
default_bits = 1024
default_keyfile = privkey.pem
distinguished_name = xca_dn
x509_extensions = xca_extensions
req_extensions = xca_extensions
string_mask = MASK:0x2002
utf8 = yes
prompt = no

[ xca_dn ]
0.C=FR
1.ST=Ile de France
2.L=Paris
3.O=protobix
4.OU=monitoring
5.CN=protobix-client.domain.tld
[email protected]

[ xca_extensions ]
nsComment=xca certificate
nsCertType=client, email
keyUsage=digitalSignature, keyEncipherment, dataEncipherment
subjectKeyIdentifier=hash
basicConstraints=critical,CA:FALSE

And command

openssl req -new -nodes -out ./newcerts/zabbix-client.csr.pem -keyout private/zabbix-client.key.pem -config ./zabbix-client.conf

Once signed with

openssl ca -config ./openssl.cnf -out certs/zabbix-client.cert.pem -infiles ./newcerts/zabbix-client.csr.pem

you can check that Issuer field is empty

> openssl x509 -in certs/zabbix-client.cert.pem -noout -text
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number: 4097 (0x1001)
    Signature Algorithm: sha256WithRSAEncryption
        Issuer: 
        Validity
            Not Before: Aug 17 09:06:18 2016 GMT
            Not After : Aug 17 09:06:18 2017 GMT
        Subject: C=FR, ST=Ile de France, L=Paris, O=protobix, OU=monitoring, CN=protobix-client.domain.tld/[email protected]
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                Public-Key: (1024 bit)
                Modulus:
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            X509v3 Basic Constraints: 
                CA:FALSE
            Netscape Comment: 
                OpenSSL Generated Certificate
            X509v3 Subject Key Identifier: 
                3D:FF:A6:E0:E1:D9:8F:04:7D:58:77:D8:8B:DE:44:BD:6E:DF:12:51
            X509v3 Authority Key Identifier: 
                keyid:38:1B:CB:72:07:90:6E:C1:6A:C9:EF:A7:CB:61:A1:D4:B4:B4:8D:EA

Comment by Jean Baptiste Favre [ 2016 Aug 17 ]

For the record, I've absolutely no idea whether empty Subject and/or Issuer fields are to be considered as valid or not from an RFC point of view.

Comment by Aleksandrs Saveljevs [ 2016 Aug 17 ]

Thank you for the detailed instructions! I have managed to create all three certificates (CA, server, client) using these commands.

However, I do not seem to get any errors when a zabbix_sender using the generated client certificate is connecting to Zabbix server using the generated server certificate. What GnuTLS version do you use? Are both zabbix_sender and Zabbix server using GnuTLS? What encryption settings did you use for the host in Zabbix frontend? How do you run zabbix_sender?

According to https://tools.ietf.org/html/rfc5280#section-4.1.2.6 , the "Subject" field for a CA should be non-empty.

Comment by Jean Baptiste Favre [ 2016 Aug 17 ]

I just reproduced the problem with zabbix_sender 3.0.4
Both zabbix_server & zabbix_sender are in version 3.0.4

Both are build against libgnutls28-dev (3.3.8-6+deb8u3) from Debian Jessie
Both depends on libgnutls (3.3.8-6+deb8u3) from Debian Jessie

But my tests were done with Python 2.7 & 3.5 using ssl module which use, I guess, Openssl.

Maybe I can provide you with my certificates (which are made fore testing purpose) so that you can check ?

Comment by Jean Baptiste Favre [ 2016 Aug 17 ]

Here are all the faulty certificate I use: https://www.jbfavre.org/assets/protobix-test-ca-certs.tar.bz2

Comment by Aleksandrs Saveljevs [ 2016 Aug 17 ]

Thank you! I have downloaded the certificates. Will try to check them next week on Monday.

Comment by Jean Baptiste Favre [ 2016 Aug 17 ]

I'm thinking you might be interested in config files:

zabbix_server.conf

LogType=console
DebugLevel = 3
PidFile=/tmp/zabbix_server.pid
DBHost=localhost
DBName=zabbix
DBUser=zabbix
DBPassword=zabbix
StartPollers=0
StartIPMIPollers=0
StartPollersUnreachable=1
StartTrappers=1
StartPingers=0
StartDiscoverers=0
StartHTTPPollers=0
StartTimers=1
StartEscalators=1
StartJavaPollers=0
StartVMwareCollectors=0
StartSNMPTrapper=0
StartDBSyncers=4
StartProxyPollers=0
ListenIP=127.0.0.1
Timeout=3
TrapperTimeout=300
TLSCAFile=tests/tls_ca/protobix-ca.cert.pem
TLSCRLFile=tests/tls_ca/protobix.crl
TLSCertFile=tests/tls_ca/protobix-zabbix-server.cert.pem
TLSKeyFile=tests/tls_ca/protobix-zabbix-server.key.pem

Launching zabbix_server with:

/usr/sbin/zabbix_server -c tests/zabbix/zabbix_server.conf -f

Trying to send items stored in zabbix_sender.list file:

protobix.host1 my.protobix.item.int 0
protobix.host1 my.protobix.item.string "item string"

Both host and items are configured in Zabbix.

Executing zabbix_sender with:

zabbix_sender -z 127.0.0.1 --tls-connect cert --tls-ca-file tests/tls_ca/protobix-ca.cert.pem  --tls-cert-file tests/tls_ca/protobix-client.cert.pem --tls-key-file tests/tls_ca/protobix-client.key.pem -i zabbix_sender.list

Got output from zabbix_sender:

info from server: "processed: 0; failed: 2; total: 2; seconds spent: 0.000133"

And from Zabbix server, on console:

zabbix_server [25610]: ERROR [file:proxy.c,line:2039] Something impossible has just happened.

Comment by Jean Baptiste Favre [ 2016 Aug 17 ]

Archive with certificates with which you can reproduce the problem

Comment by Jean Baptiste Favre [ 2016 Aug 17 ]

I just figure out that my zabbix_server use OpenSSL and not gnutls as wrongly stated earlier :-/
I built a new version which is linked against gnutls, that's why I mixed both libs.

I'll make some more tests and update the ticket

Comment by Jean Baptiste Favre [ 2016 Aug 17 ]

Just tested with 2 fresh compiled zabbix_server & zabbix_sender, first ones with gnutls, second with openssl
That give me 2 sets of binaries:
1. zabbix_server_gnutls_sqlite3 & zabbix_sender_gnutls_sqlite3
2. zabbix_server_openssl_sqlite3 & zabbix_sender_openssl_sqlite3

gnutls

Configuration:

  Detected OS:           linux-gnu
  Install path:          /usr
  Compilation arch:      linux

  Compiler:              x86_64-linux-gnu-gcc
  Compiler flags:         -g -O2 -fPIE -fstack-protector-strong -Wformat -Werror=format-security

  Library-specific flags:
    database:                  -I/usr/include
    libXML2:               -I/usr/include/libxml2
    OpenIPMI:              -I/usr/include
    libssh2:               -I/usr/include
    TLS:                   -I/usr/include
    LDAP:                  -I/usr/include

  Enable server:         yes
  Server details:
    With database:         SQLite v3.x
    WEB Monitoring:        cURL
    Native Jabber:         yes
    SNMP:                  no
    IPMI:                  yes
    SSH:                   yes
    TLS:                   GnuTLS
    ODBC:                  yes
    Linker flags:               -L/usr/lib     -L/usr/lib -L/usr/lib -L/usr/lib -L/usr/lib  -L/usr/lib/x86_64-linux-gnu -rdynamic -fPIE -pie -Wl,-z,relro -Wl,-z,now -Wl,--as-needed 
    Libraries:                  -lsqlite3 -liksemel -lxml2  -lodbc  -lssh2 -lOpenIPMI -lOpenIPMIposix -lgnutls -lldap -llber   -lcurl -lm -ldl  -lresolv 

  Enable proxy:          yes
  Proxy details:
    With database:         SQLite v3.x
    WEB Monitoring:        cURL
    SNMP:                  no
    IPMI:                  yes
    SSH:                   yes
    TLS:                   GnuTLS
    ODBC:                  yes
    Linker flags:               -L/usr/lib    -L/usr/lib -L/usr/lib -L/usr/lib -L/usr/lib  -L/usr/lib/x86_64-linux-gnu -rdynamic -fPIE -pie -Wl,-z,relro -Wl,-z,now -Wl,--as-needed 
    Libraries:                  -lsqlite3 -lxml2  -lodbc  -lssh2 -lOpenIPMI -lOpenIPMIposix -lgnutls -lldap -llber   -lcurl -lm -ldl  -lresolv 

  Enable agent:          yes
  Agent details:
    TLS:                   GnuTLS
    Linker flags:           -L/usr/lib -L/usr/lib  -L/usr/lib/x86_64-linux-gnu -rdynamic -fPIE -pie -Wl,-z,relro -Wl,-z,now -Wl,--as-needed 
    Libraries:              -lgnutls -lldap -llber   -lcurl -lm -ldl  -lresolv 

  Enable Java gateway:   no

  LDAP support:          yes
  IPv6 support:          yes

***********************************************************
*            Now run 'make install'                       *
*                                                         *
*            Thank you for using Zabbix!                  *
*              <http://www.zabbix.com>                    *
***********************************************************

test:

 ./tests/zabbix/zabbix_sender_gnutls_sqlite3 -z 127.0.0.1 --tls-connect cert --tls-ca-file tests/tls_ca/protobix-ca.cert.pem  --tls-cert-file tests/tls_ca/protobix-client.cert.pem --tls-key-file tests/tls_ca/protobix-client.key.pem -i zabbix_sender.list
info from server: "processed: 2; failed: 0; total: 2; seconds spent: 0.000145"

> ./tests/zabbix/zabbix_server_gnutls_sqlite3 -c tests/zabbix/zabbix_server.conf -f
Starting Zabbix Server. Zabbix 3.0.4 (revision 61185).
Press Ctrl+C to exit.

 23032:20160817:195225.630 Starting Zabbix Server. Zabbix 3.0.4 (revision 61185).
 23032:20160817:195225.630 ****** Enabled features ******
 23032:20160817:195225.630 SNMP monitoring:            NO
 23032:20160817:195225.630 IPMI monitoring:           YES
 23032:20160817:195225.630 Web monitoring:            YES
 23032:20160817:195225.630 VMware monitoring:         YES
 23032:20160817:195225.630 SMTP authentication:       YES
 23032:20160817:195225.630 Jabber notifications:      YES
 23032:20160817:195225.630 Ez Texting notifications:  YES
 23032:20160817:195225.630 ODBC:                      YES
 23032:20160817:195225.630 SSH2 support:              YES
 23032:20160817:195225.630 IPv6 support:              YES
 23032:20160817:195225.630 TLS support:               YES
 23032:20160817:195225.630 ******************************
 23032:20160817:195225.630 using configuration file: tests/zabbix/zabbix_server.conf
 23032:20160817:195225.637 current database version (mandatory/optional): 03000000/03000000
 23032:20160817:195225.637 required mandatory version: 03000000
 23032:20160817:195225.641 server #0 started [main process]
 23033:20160817:195225.641 server #1 started [configuration syncer #1]
 23034:20160817:195225.642 server #2 started [db watchdog #1]
 23035:20160817:195225.644 server #3 started [unreachable poller #1]
 23036:20160817:195225.644 server #4 started [trapper #1]
 23037:20160817:195225.645 server #5 started [alerter #1]
 23039:20160817:195225.646 server #7 started [timer #1]
 23040:20160817:195225.646 server #8 started [history syncer #1]
 23041:20160817:195225.646 server #9 started [history syncer #2]
 23042:20160817:195225.646 server #10 started [history syncer #3]
 23043:20160817:195225.647 server #11 started [history syncer #4]
 23038:20160817:195225.647 server #6 started [housekeeper #1]
 23044:20160817:195225.647 server #12 started [escalator #1]
 23045:20160817:195225.648 server #13 started [self-monitoring #1]
^C 23032:20160817:195329.502 Got signal [signal:2(SIGINT),sender_pid:0,sender_uid:0,reason:128]. Exiting ...
 23032:20160817:195331.505 syncing history data...
 23032:20160817:195331.505 syncing history data done
 23032:20160817:195331.505 syncing trends data...
 23032:20160817:195331.505 syncing trends data done
 23032:20160817:195331.506 Zabbix Server stopped. Zabbix 3.0.4 (revision 61185).

openssl

Configuration:

  Detected OS:           linux-gnu
  Install path:          /usr
  Compilation arch:      linux

  Compiler:              x86_64-linux-gnu-gcc
  Compiler flags:         -g -O2 -fPIE -fstack-protector-strong -Wformat -Werror=format-security

  Library-specific flags:
    database:                  -I/usr/include
    libXML2:               -I/usr/include/libxml2
    OpenIPMI:              -I/usr/include
    libssh2:               -I/usr/include
    TLS:                   -I/usr/include
    LDAP:                  -I/usr/include

  Enable server:         yes
  Server details:
    With database:         SQLite v3.x
    WEB Monitoring:        cURL
    Native Jabber:         yes
    SNMP:                  no
    IPMI:                  yes
    SSH:                   yes
    TLS:                   OpenSSL
    ODBC:                  yes
    Linker flags:               -L/usr/lib     -L/usr/lib -L/usr/lib -L/usr/lib -L/usr/lib  -L/usr/lib/x86_64-linux-gnu -rdynamic -fPIE -pie -Wl,-z,relro -Wl,-z,now -Wl,--as-needed 
    Libraries:                  -lsqlite3 -liksemel -lxml2  -lodbc  -lssh2 -lOpenIPMI -lOpenIPMIposix -lssl -lcrypto -lldap -llber   -lcurl -lm -ldl  -lresolv 

  Enable proxy:          yes
  Proxy details:
    With database:         SQLite v3.x
    WEB Monitoring:        cURL
    SNMP:                  no
    IPMI:                  yes
    SSH:                   yes
    TLS:                   OpenSSL
    ODBC:                  yes
    Linker flags:               -L/usr/lib    -L/usr/lib -L/usr/lib -L/usr/lib -L/usr/lib  -L/usr/lib/x86_64-linux-gnu -rdynamic -fPIE -pie -Wl,-z,relro -Wl,-z,now -Wl,--as-needed 
    Libraries:                  -lsqlite3 -lxml2  -lodbc  -lssh2 -lOpenIPMI -lOpenIPMIposix -lssl -lcrypto -lldap -llber   -lcurl -lm -ldl  -lresolv 

  Enable agent:          yes
  Agent details:
    TLS:                   OpenSSL
    Linker flags:           -L/usr/lib -L/usr/lib  -L/usr/lib/x86_64-linux-gnu -rdynamic -fPIE -pie -Wl,-z,relro -Wl,-z,now -Wl,--as-needed 
    Libraries:              -lssl -lcrypto -lldap -llber   -lcurl -lm -ldl  -lresolv 

  Enable Java gateway:   no

  LDAP support:          yes
  IPv6 support:          yes

***********************************************************
*            Now run 'make install'                       *
*                                                         *
*            Thank you for using Zabbix!                  *
*              <http://www.zabbix.com>                    *
***********************************************************

test

> ./tests/zabbix/zabbix_sender_openssl_sqlite3 -z 127.0.0.1 --tls-connect cert --tls-ca-file tests/tls_ca/protobix-ca.cert.pem  --tls-cert-file tests/tls_ca/protobix-client.cert.pem --tls-key-file tests/tls_ca/protobix-client.key.pem -i zabbix_sender.list
info from server: "processed: 0; failed: 2; total: 2; seconds spent: 0.000047"

> ./tests/zabbix/zabbix_server_openssl_sqlite3 -c tests/zabbix/zabbix_server.conf -f
Starting Zabbix Server. Zabbix 3.0.4 (revision 61185).
Press Ctrl+C to exit.

 23206:20160817:195341.185 Starting Zabbix Server. Zabbix 3.0.4 (revision 61185).
 23206:20160817:195341.185 ****** Enabled features ******
 23206:20160817:195341.185 SNMP monitoring:            NO
 23206:20160817:195341.185 IPMI monitoring:           YES
 23206:20160817:195341.185 Web monitoring:            YES
 23206:20160817:195341.185 VMware monitoring:         YES
 23206:20160817:195341.185 SMTP authentication:       YES
 23206:20160817:195341.185 Jabber notifications:      YES
 23206:20160817:195341.185 Ez Texting notifications:  YES
 23206:20160817:195341.185 ODBC:                      YES
 23206:20160817:195341.185 SSH2 support:              YES
 23206:20160817:195341.185 IPv6 support:              YES
 23206:20160817:195341.185 TLS support:               YES
 23206:20160817:195341.185 ******************************
 23206:20160817:195341.185 using configuration file: tests/zabbix/zabbix_server.conf
 23206:20160817:195341.193 current database version (mandatory/optional): 03000000/03000000
 23206:20160817:195341.193 required mandatory version: 03000000
 23206:20160817:195341.199 server #0 started [main process]
 23207:20160817:195341.199 server #1 started [configuration syncer #1]
 23208:20160817:195341.199 server #2 started [db watchdog #1]
 23209:20160817:195341.200 server #3 started [unreachable poller #1]
 23210:20160817:195341.200 server #4 started [trapper #1]
 23211:20160817:195341.201 server #5 started [alerter #1]
 23212:20160817:195341.201 server #6 started [housekeeper #1]
 23213:20160817:195341.202 server #7 started [timer #1]
 23214:20160817:195341.203 server #8 started [history syncer #1]
 23215:20160817:195341.204 server #9 started [history syncer #2]
 23216:20160817:195341.204 server #10 started [history syncer #3]
 23217:20160817:195341.205 server #11 started [history syncer #4]
 23218:20160817:195341.206 server #12 started [escalator #1]
 23219:20160817:195341.208 server #13 started [self-monitoring #1]
zabbix_server_openssl_sqlite3 [23210]: ERROR [file:proxy.c,line:2039] Something impossible has just happened.
^C 23206:20160817:201413.439 Got signal [signal:2(SIGINT),sender_pid:0,sender_uid:0,reason:128]. Exiting ...
^L 23206:20160817:201415.444 syncing history data...
 23206:20160817:201415.444 syncing history data done
 23206:20160817:201415.444 syncing trends data...
 23206:20160817:201415.444 syncing trends data done
 23206:20160817:201415.444 Zabbix Server stopped. Zabbix 3.0.4 (revision 61185).

Regards,
JB

Comment by Aleksandrs Saveljevs [ 2016 Aug 22 ]

Thanks for the detailed information! It is a great pleasure to work with it.

For completeness, which version of OpenSSL do you use? Do you have multiple versions of OpenSSL libraries on the system by any chance?

Comment by Jean Baptiste Favre [ 2016 Aug 22 ]

I tested on a Jessie docker container with OpenSSL 1.0.1k-3+deb8u5 from Jessie security updates.
Due to debian packaging, it's very unlikely to have multiple versions of OpenSSL installed (and I'm absolutely sure I didn't install it manually)

Comment by Aleksandrs Saveljevs [ 2016 Aug 23 ]

Thank you! I have finally managed to reproduce the problem. The behavior indeed depends on DebugLevel=3/4 setting and whether verbose flag is used for zabbix_sender. More details to follow later.

Comment by Jean Baptiste Favre [ 2016 Aug 23 ]

IMHO there are 2 problems:

First one occurs in DebugLevel=3 and produce following log:

ERROR [file:proxy.c,line:2039] Something impossible has just happened.

In this case, TLS connection is established but items are all rejected by the serverr.

Second one occurs with DebugLevel>=4.
My guess is that Zabbix try to log/whatever certificate issuer and/or subject, which is empty, and this generates an error.
In this case, TLS connection isn't even establish, item are not sent.

BTW, I'm happy you manage to reproduce the problem. I was thinking I did something wrong and was alone to have this issue

Comment by Aleksandrs Saveljevs [ 2016 Sep 05 ]

It seems that there are multiple considerations to take into account in order to approach this task. Let's try to gradually document them one by one.

First, let's start with the RFC. Section https://tools.ietf.org/html/rfc5280#section-4.1.2.4 says that "Issuer" field must have a non-empty name:

The issuer field MUST contain a non-empty distinguished name (DN).

Section https://tools.ietf.org/html/rfc5280#section-4.1.2.6 says the following about "Subject" (slightly filtered for readability):

The subject name MAY be carried in the subject field and/or the subjectAltName extension. If the subject is a CA, then the subject field MUST be populated with a non-empty distinguished name matching the contents of the issuer field in all certificates issued by the subject CA. If subject naming information is present only in the subjectAltName extension, then the subject name MUST be an empty sequence and the subjectAltName extension MUST be critical.

Thus, according to the RFC, the "Issuer" field must be non-empty and "Subject" field may be empty (although currently it may not work perfectly in Zabbix anyway, see ZBXNEXT-3145), so Zabbix should be prepared to handle at least an empty "Subject".

Comment by Aleksandrs Saveljevs [ 2016 Sep 05 ]

The second consideration is that in host configuration in Zabbix frontend there are "Issuer" and "Subject" fields, which are meant to specify exact values that these fields should have in host certificates. There, empty "Issuer" and "Subject" have a special meaning that no restrictions should be imposed. So, if we allow "Issuer" and "Subject" fields in certificates to be empty, there will be a slightly discrepancy here. On the other hand, trying to restrict these fields to empty values may not be very useful, even with ZBXNEXT-3144.

Similarly, Zabbix agent, for instance, has TLSServerCertIssuer and TLSServerCertSubject in its configuration file and, currently, we restrict them to non-empty values in zbx_tls_validate_config() function, so this will have to be changed.

Comment by Aleksandrs Saveljevs [ 2016 Oct 04 ]

The third consideration is that we should check all three libraries (OpenSSL, GnuTLS, and mbed TLS) to see how they handle empty values. It would be nice if our behavior with all three libraries would be the same. Also, real practical implementations may influence the decision how to handle an empty "Issuer" - RFC says it is invalid, but if all libraries handle it and it is possible to create such certificates, should we support them anyway?

Comment by Aleksandrs Saveljevs [ 2016 Oct 25 ]

Regarding the third consideration, it seems that mbed TLS (1.3.x versions at least) does not like empty issuers. However, GnuTLS accepts them, so for consistency decided to make OpenSSL accept them as well (it did not, but that was because of our code, not OpenSSL).

Comment by Aleksandrs Saveljevs [ 2016 Oct 25 ]

Fixed in development branch svn://svn.zabbix.com/branches/dev/ZBX-11101 . In summary, OpenSSL version of Zabbix now works with empty issuers and subjects, and Zabbix behavior should be consistent between different logging levels. Semantics of "Issuer" and "Subject" fields in the frontend (and TLSServerCertIssuer and TLSServerCertSubject parameters in configuration files) was not changed.

Comment by Glebs Ivanovskis (Inactive) [ 2016 Nov 21 ]

(1) With OpenSSL in zbx_tls_connect() at LOG_LEVEL_DEBUG function zbx_get_issuer_subject() can be called second time which I guess will leak storage allocated for issuer and subject in the first call.

asaveljevs RESOLVED in r63899.

glebs.ivanovskis This solves the problem.
CLOSED

Comment by Glebs Ivanovskis (Inactive) [ 2016 Nov 21 ]

(2) With mbed TLS in zbx_log_peer_cert() we pass NULL as error to zabbix_log() if we failed to get serial and it's going to be a crash.

asaveljevs RESOLVED in r63900.

glebs.ivanovskis I wasn't entirely sure about ZBX_NULL2STR over ZBX_NULL2EMPTY_STR preference and decided to make messages more verbose. Also seems that 0 is a valid return for x509_serial_gets(). Please have a look at r63915. Still RESOLVED

asaveljevs Looks good. CLOSED.

Comment by Glebs Ivanovskis (Inactive) [ 2016 Nov 21 ]

(3) GnuTLS edition of zbx_log_peer_cert() logs "zbx_log_peer_cert" as __function_name while mbed TLS's zbx_log_peer_cert() logs caller's __function_name which would make its messages more like OpenSSL's ones if they contained function name, because OpenSSL currently does not have zbx_log_peer_cert(). (Probably it deserves one?) Let's make logging consistent across libraries.

asaveljevs Added "function_name" parameter to GnuTLS version of zbx_log_peer_cert() in r63901. RESOLVED.

glebs.ivanovskis There is odd-one-out message among "cannot obtain peer certificate ...":

 		zabbix_log(LOG_LEVEL_DEBUG, "%s(): gnutls_x509_crt_print() failed: %d %s",
 				__function_name function_name, res, gnutls_strerror(res));

REOPENED

asaveljevs It may have a different style from other libraries, but at least it is consistent with other GnuTLS messages. As discussed, CLOSED.

Comment by Glebs Ivanovskis (Inactive) [ 2016 Nov 21 ]

(4) This piece of code does not feel right:

		if ((NULL != tls_arg2 && '\0' != *tls_arg2) || (NULL != tls_arg1 && '\0' != *tls_arg1))
		{
			if (NULL == (peer_cert = zbx_get_peer_cert(s->tls_ctx->ctx, error)))
			{
				zbx_tls_close(s);
				goto out1;
			}
		}

If it's possible to establish TLS connection with both tls_arg1 and tls_arg2 equal to NULL or empty and logging is less verbose than LOG_LEVEL_DEBUG then we will pass peer_cert equal to NULL to zbx_verify_issuer_subject(). Either if ((NULL != tls_arg2 && '\0' != *tls_arg2) || (NULL != tls_arg1 && '\0' != *tls_arg1)) check is redundant or we need to change the logic later on.

asaveljevs It does not seem to cause a problem, because we only use the certificate inside zbx_verify_issuer_subject() if at least one of tls_arg1 or tls_arg2 is non-empty.

glebs.ivanovskis You're right, but it was non-obvious, because part of issuer/subject verification logic was in zbx_verify_issuer_subject() and part of it was tightly incorporated into zbx_tls_connect() and zbx_tls_accept() (6 places altogether thanks to 3 libraries). I tried to improve the situation in r64145, r64146, r64147, r64150. RESOLVED

asaveljevs CLOSED

Comment by Glebs Ivanovskis (Inactive) [ 2016 Nov 22 ]

(5) [D] Host configuration docs explicitly says that with empty Subject field any valid certificate is accepted. For Issuer field it just says that this field is optional. To verify that the behaviour is the same one needs to dig as deep as point 3. here. Would be nice to mention it on the page describing host configuration.

glebs.ivanovskis Please review my changes in:

asaveljevs Perfect! CLOSED.

Comment by Glebs Ivanovskis (Inactive) [ 2016 Nov 23 ]

(6) Probably introduced in ~~ZBX-4148~~ or latest ~~ZBXNEXT-1263~~ merge. If zbx_tls_connect() fails for mbed TLS in the very beginning we should not free context. RESOLVED in r63958.

asaveljevs CLOSED

Comment by Glebs Ivanovskis (Inactive) [ 2016 Nov 29 ]

(7) One more bug was introduced in ~~ZBX-4148~~ and is a most likely reason of ~~ZBX-10970~~. RESOLVED in r64105.

asaveljevs Well spotted! CLOSED.

Comment by Glebs Ivanovskis (Inactive) [ 2016 Nov 29 ]

(8) In zbx_get_issuer_subject() we create a new BIO, get a pointer to its data buffer and then free the BIO. Then outside of zbx_get_issuer_subject() we access the aforementioned pointer to copy or print buffer contents and finally free that pointer. I wondered why this proved to work.

According to OpenSSL documentation BIO_free() should free BIO and associated buffer unless BIO_set_close(..., BIO_NOCLOSE) is set. We don't call BIO_set_close(), so we must be either double-freeing BIO's data buffer after reading from dangling pointer (if default is BIO_CLOSE) or leaking BUF_MEM structure associated with BIO (if default is BIO_NOCLOSE). Neither is good.

What actually saves us is BIO_set_flags(..., BIO_FLAGS_MEM_RDONLY), in OpenSSL sources I've found this:

static int mem_free(BIO *a)
{
    if (a == NULL)
        return (0);
    if (a->shutdown) {
        if ((a->init) && (a->ptr != NULL)) {
            BUF_MEM *b;
            b = (BUF_MEM *)a->ptr;
            if (a->flags & BIO_FLAGS_MEM_RDONLY)
                b->data = NULL;
            BUF_MEM_free(b);
            a->ptr = NULL;
        }
    }
    return (1);
}

But comments in our source explain this call with "performance" considerations...

Even though we seem to be OK for now, I feel that this solution is so much not supported...

glebs.ivanovskis RESOLVED in r64141.

asaveljevs In case X509_NAME_print_ex() fails, we say that we are "out of memory". Are we sure about that? Also, are we sure that X509_NAME_print_ex() always NUL-terminates on success? Previous code used to write NUL explicitly.

glebs.ivanovskis We ask X509_NAME_print_ex() to write into memory BIO and man 3 BIO_s_mem says that

Writes to memory BIOs will always succeed if memory is available: that is their size can grow indefinitely.

...from which I draw a conclusion that X509_NAME_print_ex() failure is caused by memory shortage. But we can change error message to something like "X509_NAME_print_ex() failed" for consistency.

zbx_strlcpy() promises to NULL terminate in any case. Feel free to add some comments!

asaveljevs X509_NAME_print_ex() is a pretty complicated function, so it may be a good idea not to make this conclusion.

Regarding zbx_strlcpy(), it does NUL-terminate, but the character sequence it will terminate may be longer than needed (consider actual, valuable data being shorter than buffer size and followed by some garbage). REOPENED.

asaveljevs Please take a look at typo fixes in r64574.

glebs.ivanovskis Thank you for these fixes, accepted. Message and buffer termination issues RESOLVED in r64583.

asaveljevs Wonderful! CLOSED

Comment by Glebs Ivanovskis (Inactive) [ 2016 Nov 30 ]

(9) Please review my changes:

r63959 - purely stylistic and readability improvement;
r63965, r63966, r63967 - preprocessor directives;
r64029, r64060, r64077, r64081, r64083 - refactoring of zbx_tls_read() (ideally it would be nice to make zbx_tls_read() mimic ZBX_TCP_READ() and move timekeeping to zbx_tcp_read() completely, but it's a bit problematic since ZBX_PROTO_ERROR and ZBX_PROTO_AGAIN are passed around in different ways...);
r64084 - carbon copy for zbx_tls_write().

asaveljevs Looks wonderful, but please take a look at a minor consistency fix in r64568.

glebs.ivanovskis Thank you for being alert!
CLOSED

Comment by Glebs Ivanovskis (Inactive) [ 2016 Dec 01 ]

(10) Warnings with GnuTLS:

Making all in zbxcrypto
tls.c: In function ‘zbx_tls_connect’:
tls.c:3973:44: warning: cast to pointer from integer of different size [-Wint-to-pointer-cast]
  gnutls_transport_set_ptr(s->tls_ctx->ctx, (gnutls_transport_ptr_t)s->socket);
                                            ^
tls.c: In function ‘zbx_tls_accept’:
tls.c:4688:44: warning: cast to pointer from integer of different size [-Wint-to-pointer-cast]
  gnutls_transport_set_ptr(s->tls_ctx->ctx, (gnutls_transport_ptr_t)s->socket);
                                            ^

Seem to be legitimate and also easily fixable since GnuTLS provides gnutls_transport_set_int(). What do you think?

Although I would consider setting push/pull functions explicitly a better solution. After a brief investigation of this aspect I'm a bit puzzled how send() and recv() (which have 4 parameters) work as GnuTLS transport push/pull functions (which must have just 3).

asaveljevs gnutls_transport_set_int() is a nice discovery. I like it.

asaveljevs According to https://www.zabbix.com/documentation/3.0/manual/encryption , we support GnuTLS 3.1.18 and above, and gnutls_transport_set_int() is available since 3.1.9.

glebs.ivanovskis Seems like GnuTLS always stores "transport layer descriptor" as pointer and has a full arsenal of int to pointer and pointer to int conversion tools, so let's delegate the responsibility (and warnings) to them. RESOLVED in r64581.

asaveljevs Looks good, but please see r64588 - it moves ZBX_SOCKET_TO_INT() to a place that is slightly more proper.

glebs.ivanovskis Conditional #if defined(SOCKET) || defined(_WINDOWS) dates back to a very distant past (r4623 adds _WINDOWS part without any explanation, SOCKET part existed from the genesis). Modern socket-ologists believe that SOCKET can't exist without Windows and therefore SOCKET check can be dropped and ZBX_SOCKET_TO_INT() can be placed wherever it matches formatting. RESOLVED in r64591 and r64592.

asaveljevs Thank you for useful discoveries in Zabbix archaeology! CLOSED

Comment by Glebs Ivanovskis (Inactive) [ 2016 Dec 20 ]

Regarding (10) there was a bit of concern about converting SOCKET to int on 64 bit Windows to pass it to gnutls_transport_set_int(). There SOCKET is a 64 bit unsigned integer and the only guarantee provided by Microsoft is that valid socket value is not equal to INVALID_SOCKET constant. Converting SOCKET to int may lead to a loss of significant bits in top 4 bytes. Judging by GnuTLS code no matter how we pass socket descriptor it will be passed through (int) cast at some point:

#define GNUTLS_POINTER_TO_INT(_) ((int) GNUTLS_POINTER_TO_INT_CAST (_))

Here GNUTLS_POINTER_TO_INT_CAST() is a clever cast macro set by configure script which converts void * to an integer type of appropriate size to avoid -Wint-to-pointer-cast warnings.

Plus OpenSSL leaves us no choice at all: SSL_set_fd() only accepts int.

Given that so many cross-platform libraries and applications rely on SOCKET to int conversion it is reasonably safe to use it in Zabbix too.

Comment by Glebs Ivanovskis (Inactive) [ 2016 Dec 20 ]

(11) On the final stages of zbx_tls_connect() we verify peer certificate and log its details, but these actions come in different order for GnuTLS and OpenSSL. For troubleshooting it makes sense to do the logging first, this way certificate details will be seen in the log file even if it's rejected by the library. RESOLVED r64596

asaveljevs It seems the same was needed in zbx_tls_accept(). Please take a look at r64618. Still RESOLVED.

glebs.ivanovskis Absolutely! Regarding comments, I had a go in r64630. Now they make sense, at least to me. Please have a look. Still RESOLVED

asaveljevs Great! Finally CLOSED.

Comment by Aleksandrs Saveljevs [ 2016 Dec 21 ]

Fixed in pre-3.0.8rc1 r64639, pre-3.2.4rc1 r64640, pre-3.3.0 (trunk) r64641.

Generated at Thu Apr 25 12:24:48 EEST 2024 using Jira 9.12.4#9120004-sha1:625303b708afdb767e17cb2838290c41888e9ff0.