[ZBX-17604] Zabbix TLS "unknown CA" with connections agent-active cert signed by intermediate CA Created: 2020 Apr 19  Updated: 2022 Jan 19

Status: Open
Project: ZABBIX BUGS AND ISSUES
Component/s: None
Affects Version/s: 4.4.7
Fix Version/s: None

Type: Problem report Priority: Trivial
Reporter: Jad Assignee: Renats Valiahmetovs (Inactive)
Resolution: Unresolved Votes: 1
Labels: agent, certificates, encryption
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

Docker

Attachments: Zip Archive zabbix-tls-unknown-ca-issue-reproducer-enc-intermediate.zip     Zip Archive zabbix-tls-unknown-ca-issue-reproducer-enc-root.zip    
Issue Links:
Duplicate

 Description   

**All the details of issue reproduction are explained in the issue reproducer I made for this issue: github.com/jadbaz/zabbix-tls-unknown-ca-issue-reproducer

I've put a lot of effort into making this MCVE because I think this is a non-trivial issue  and that this will speed up debugging

Check out the instructions in the repo and run with "--inter" and without
Result
Agent:

active check configuration update from [zabbix-server:10051] started to fail (TCP successful, cannot establish TLS to [[zabbix-server]:10051]: invalid CA certificate: SSL_connect() set result code to SSL_ERROR_SSL: file ssl/statem/statem_clnt.c line 1913: error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed: TLS write fatal alert "unknown CA")

Server

failed to accept an incoming connection: from <IP>: TLS handshake set result code to 1: file ssl/record/rec_layer_s3.c line 1543: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca: SSL alert number 48: TLS read fatal alert "unknown CA"

 
Expected
**Expected for the agent to connect to the server without any errors

Issue description
I have an agent connected to a zabbix server with 1 active check. This agent is configured to use encryption for "connections from host". 

In the "root CA" mode, I am doing the following:

  • Generating a root CA key and self-signed cert
  • Generating a server key
  • Creating a server CSR
  • Signing the CSR by the root CA
  • Generating an agent key
  • Creating an agent CSR
  • Signing the CSR by the root CA

For cert files

  • In Zabbix CA file, I include
    • root CA cert
  • In zabbix_server.crt I include in order
    • server cert
    • root CA cert
  • In zabbix_server.crt I include in order
    • agent cert
    • root CA cert

In the "intermediate CA" mode, I am doing the following (bold parts are different or added):

  • Generating a root CA key and self-signed cert
  • Creating an intermediate CA key
  • Creating an intermediate CA CSR
  • Signing the CSR by the root CA
  • Generating a server key
  • Creating a server CSR
  • Signing the CSR by the intermediate CA
  • Generating an agent key
  • Creating an agent CSR
  • Signing the CSR by the intermediate CA

For cert files

  • In Zabbix CA file, I include
    • root CA cert
    • intermediate CA cert
  • In zabbix_server.crt I include in order
    • server cert
    • intermediate CA cert
    • root CA cert
  • In zabbix_server.crt I include in order
    • agent cert
    • intermediate CA cert
    • root CA cert

Moreover, I am attaching both "/var/lib/zabbix/enc" directories when using root only and when using intermediate CA (they are mounted by agent and server).

 Comments   
Comment by Jad [ 2020 Apr 27 ]

Hello,

Can I kindly ask about any updates on this issue? (I know it's only been a week but this is a huge blocker)

And let me know if you have any issues running the reproducer

Thank you

Comment by dimir [ 2020 Apr 28 ]

Cross-reference: https://www.zabbix.com/forum/zabbix-troubleshooting-and-problems/400194-zabbix-tls-unknown-ca-with-agent-active-cert-signed-by-intermediate-ca

Comment by Jad [ 2020 Jun 01 ]

Hello, can I ask again about an update on this issue?

Comment by Orlop [ 2020 Jul 20 ]

I have the same issue (with proxy)

Comment by Aaron C. de Bruyn [ 2021 Feb 12 ]

I'm running into the same issue.  Swapping the certs out for one without an intermediate CA in the chain fixes it.

I'm wondering if it's because the intermediate cert doesn't have a CRL set.

Comment by Oleksii Zagorskyi [ 2022 Jan 13 ]

Documentation has a note bottom of this page:
https://www.zabbix.com/documentation/5.0/en/manual/encryption/using_certificates

If Zabbix component is compiled with OpenSSL and CRLs are used then each top and intermediate level CA in certificate chains must have a corresponding CRL (it can be empty) in TLSCRLFile.

I'm wondering was this requirement satisfied or not?

Comment by Andris Mednis [ 2022 Jan 17 ]

I tested with zabbix-tls-unknown-ca-issue-reproducer-enc-root.zip (works) and with zabbix-tls-unknown-ca-issue-reproducer-enc-intermediate.zip (fails) - as described above. Removing the inter-CA certificate from zabbix_ca_file and root-CA certificate from zabbix_agentd.crt and zabbix_server.crt did not help.
But generally it works with intermediate CA.

Here is a demo walk-through for a Zabbix server and agent on the same Linux machine.
Let's use recommendations from https://pki-tutorial.readthedocs.io/en/latest/simple/index.html.

1. Create Root CA.
1.1 Create 4 directories:

$ cd
$ mkdir ca1
$ cd ca1
$ mkdir -p ca/root-ca/private ca/root-ca/db crl certs etc
$ chmod 700 ca/root-ca/private

1.2 Create database:

$ cp /dev/null ca/root-ca/db/root-ca.db
$ cp /dev/null ca/root-ca/db/root-ca.db.attr
$ echo 01 > ca/root-ca/db/root-ca.crt.srl
$ echo 01 > ca/root-ca/db/root-ca.crl.srl

Get config file sample from https://pki-tutorial.readthedocs.io/en/latest/simple/root-ca.conf.html
and put it into file etc/root-ca.conf. Replace sha1 -> sha256 in it.

1.3 Create CA certificate request

$ openssl req -new -config etc/root-ca.conf -out ca/root-ca.csr -keyout ca/root-ca/private/root-ca.key
Generating a RSA private key
...................................................................................................................+++++
...................................................................+++++
writing new private key to 'ca/root-ca/private/root-ca.key'
Enter PEM pass phrase:
Verifying - Enter PEM pass phrase:
-----

1.4 Create CA certificate

$ openssl ca -selfsign  -config etc/root-ca.conf -in ca/root-ca.csr -out ca/root-ca.crt -extensions root_ca_ext
Using configuration from etc/root-ca.conf
Enter pass phrase for ./ca/root-ca/private/root-ca.key:
Check that the request matches the signature
Signature ok
Certificate Details:
        Serial Number: 1 (0x1)
        Validity
            Not Before: Jan 17 19:10:57 2022 GMT
            Not After : Jan 17 19:10:57 2032 GMT
        Subject:
            domainComponent           = org
            domainComponent           = simple
            organizationName          = Simple Inc
            organizationalUnitName    = Simple Root CA
            commonName                = Simple Root CA
        X509v3 extensions:
            X509v3 Key Usage: critical
                Certificate Sign, CRL Sign
            X509v3 Basic Constraints: critical
                CA:TRUE
            X509v3 Subject Key Identifier: 
                6C:D2:09:FF:5E:78:E6:F6:A3:7C:B6:9C:99:24:81:63:F2:DE:89:86
            X509v3 Authority Key Identifier: 
                keyid:6C:D2:09:FF:5E:78:E6:F6:A3:7C:B6:9C:99:24:81:63:F2:DE:89:86

Certificate is to be certified until Jan 17 19:10:57 2032 GMT (3652 days)
Sign the certificate? [y/n]:y

1 out of 1 certificate requests certified, commit? [y/n]y
...

2. Create Signing CA

2.1 Create directories:

$ mkdir -p ca/signing-ca/private ca/signing-ca/db crl certs
$ chmod 700 ca/signing-ca/private

2.2 Create database:

$ cp /dev/null ca/signing-ca/db/signing-ca.db
$ cp /dev/null ca/signing-ca/db/signing-ca.db.attr
$ echo 01 > ca/signing-ca/db/signing-ca.crt.srl
$ echo 01 > ca/signing-ca/db/signing-ca.crl.srl

2.3 Create Signing CA certificate request
Get config file sample from https://pki-tutorial.readthedocs.io/en/latest/simple/signing-ca.conf.html.
and put it into etc/signing-ca.conf. Replace sha1 -> sha256 in it.

$ openssl req -new -config etc/signing-ca.conf -out ca/signing-ca.csr -keyout ca/signing-ca/private/signing-ca.key
Generating a RSA private key
.........................................+++++
...+++++
writing new private key to 'ca/signing-ca/private/signing-ca.key'
Enter PEM pass phrase:
Verifying - Enter PEM pass phrase:
-----

2.4 Sign "Signing CA" certificate request with "Root CA" certificate

$ openssl ca -config etc/root-ca.conf -in ca/signing-ca.csr -out ca/signing-ca.crt -extensions signing_ca_ext
Using configuration from etc/root-ca.conf
Enter pass phrase for ./ca/root-ca/private/root-ca.key:
Check that the request matches the signature
Signature ok
Certificate Details:
        Serial Number: 2 (0x2)
        Validity
            Not Before: Jan 17 19:22:54 2022 GMT
            Not After : Jan 17 19:22:54 2032 GMT
        Subject:
            domainComponent           = org
            domainComponent           = simple
            organizationName          = Simple Inc
            organizationalUnitName    = Simple Signing CA
            commonName                = Simple Signing CA
        X509v3 extensions:
            X509v3 Key Usage: critical
                Certificate Sign, CRL Sign
            X509v3 Basic Constraints: critical
                CA:TRUE, pathlen:0
            X509v3 Subject Key Identifier: 
                AE:79:CD:D2:2C:64:52:AB:15:B9:10:1D:73:CC:32:48:4A:B3:8C:AA
            X509v3 Authority Key Identifier: 
                keyid:6C:D2:09:FF:5E:78:E6:F6:A3:7C:B6:9C:99:24:81:63:F2:DE:89:86

Certificate is to be certified until Jan 17 19:22:54 2032 GMT (3652 days)
Sign the certificate? [y/n]:y

1 out of 1 certificate requests certified, commit? [y/n]y
...

3. Create Zabbix server certificate request

Get server config file sample from https://pki-tutorial.readthedocs.io/en/latest/simple/server.conf.html
and put it in etc/server.conf. Replace sha1 -> sha256 in it.

$ SAN=DNS:localhost openssl req -new -config etc/server.conf -out certs/zabbix_server.csr -keyout certs/zabbix_server.key
Generating a RSA private key
..................................................+++++
....................................................+++++
writing new private key to 'certs/zabbix_server.key'
-----
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
1. Domain Component         (eg, com)       []:org
2. Domain Component         (eg, company)   []:simple
3. Domain Component         (eg, pki)       []:
4. Organization Name        (eg, company)   []:Simple Inc
5. Organizational Unit Name (eg, section)   []:
6. Common Name              (eg, FQDN)      []:Zabbix server

3.2 Sign Zabbix server certificate request

$ openssl ca -config etc/signing-ca.conf -in certs/zabbix_server.csr -out certs/zabbix_server.crt -extensions server_ext
Using configuration from etc/signing-ca.conf
Enter pass phrase for ./ca/signing-ca/private/signing-ca.key:
Check that the request matches the signature
Signature ok
Certificate Details:
        Serial Number: 1 (0x1)
        Validity
            Not Before: Jan 17 19:42:54 2022 GMT
            Not After : Jan 17 19:42:54 2024 GMT
        Subject:
            domainComponent           = org
            domainComponent           = simple
            organizationName          = Simple Inc
            commonName                = Zabbix server
        X509v3 extensions:
            X509v3 Key Usage: critical
                Digital Signature, Key Encipherment
            X509v3 Basic Constraints: 
                CA:FALSE
            X509v3 Extended Key Usage: 
                TLS Web Server Authentication, TLS Web Client Authentication
            X509v3 Subject Key Identifier: 
                8E:85:3A:48:C0:8C:BD:6B:B1:71:DD:04:FE:F1:AA:84:99:4E:45:63
            X509v3 Authority Key Identifier: 
                keyid:AE:79:CD:D2:2C:64:52:AB:15:B9:10:1D:73:CC:32:48:4A:B3:8C:AA

            X509v3 Subject Alternative Name: 
                DNS:localhost
Certificate is to be certified until Jan 17 19:42:54 2024 GMT (730 days)
Sign the certificate? [y/n]:y

1 out of 1 certificate requests certified, commit? [y/n]y
...

4. Create Zabbix agentd certificate
4.1 Create certificate request

$ SAN=DNS:localhost openssl req -new -config etc/server.conf -out certs/zabbix_agentd.csr -keyout certs/zabbix_agentd.key
Generating a RSA private key
....................+++++
................................+++++
writing new private key to 'certs/zabbix_agentd.key'
-----
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
1. Domain Component         (eg, com)       []:org
2. Domain Component         (eg, company)   []:simple
3. Domain Component         (eg, pki)       []:
4. Organization Name        (eg, company)   []:Simple Inc
5. Organizational Unit Name (eg, section)   []:
6. Common Name              (eg, FQDN)      []:Zabbix agentd

4.2 Sign Zabbix agent certificate request

$ openssl ca -config etc/signing-ca.conf -in certs/zabbix_agentd.csr -out certs/zabbix_agentd.crt -extensions server_ext
Using configuration from etc/signing-ca.conf
Enter pass phrase for ./ca/signing-ca/private/signing-ca.key:
Check that the request matches the signature
Signature ok
Certificate Details:
        Serial Number: 2 (0x2)
        Validity
            Not Before: Jan 17 19:48:28 2022 GMT
            Not After : Jan 17 19:48:28 2024 GMT
        Subject:
            domainComponent           = org
            domainComponent           = simple
            organizationName          = Simple Inc
            commonName                = Zabbix agentd
        X509v3 extensions:
            X509v3 Key Usage: critical
                Digital Signature, Key Encipherment
            X509v3 Basic Constraints: 
                CA:FALSE
            X509v3 Extended Key Usage: 
                TLS Web Server Authentication, TLS Web Client Authentication
            X509v3 Subject Key Identifier: 
                DC:1C:8D:DE:2C:DF:5F:B8:52:03:8A:57:3B:F6:66:4B:9C:51:7F:38
            X509v3 Authority Key Identifier: 
                keyid:AE:79:CD:D2:2C:64:52:AB:15:B9:10:1D:73:CC:32:48:4A:B3:8C:AA

            X509v3 Subject Alternative Name: 
                DNS:localhost
Certificate is to be certified until Jan 17 19:48:28 2024 GMT (730 days)
Sign the certificate? [y/n]:y

1 out of 1 certificate requests certified, commit? [y/n]y
...

5. Create directory where to put generated certificates:

$ cd
$ mkdir certs

Create zabbix_ca_file:

$ cp ca1/ca/root-ca.crt certs/zabbix_ca_file

Copy private keys:

$ cp ca1/certs/zabbix_server.key ca1/certs/zabbix_agentd.key certs/

Compose certificate files:

$ cat ca1/certs/zabbix_server.crt ca1/ca/signing-ca.crt >certs/zabbix_server.crt
$ cat ca1/certs/zabbix_agentd.crt ca1/ca/signing-ca.crt >certs/zabbix_agentd.crt

6. Configure these files in Zabbix server and agentd config files and test.

Comment by Andris Mednis [ 2022 Jan 17 ]

Note that in the demo walk-through above the cert files are simpler:

  • In Zabbix CA file:
    • root CA cert
  • In zabbix_server.crt - in order
    • server cert
    • intermediate CA cert
  • In zabbix_agentd.crt - in order
    • agent cert
    • intermediate CA cert

Also note that both Zabbix server and zabbix agents certificates have "TLS Web Server Authentication, TLS Web Client Authentication":

        X509v3 extensions:
            X509v3 Key Usage: critical
                Digital Signature, Key Encipherment
            X509v3 Basic Constraints: 
                CA:FALSE
            X509v3 Extended Key Usage: 
                TLS Web Server Authentication, TLS Web Client Authentication

because the agent acts as a TLS client during active checks and as a TLS server during passive checks. In all connections both parties (Zabbix server and agent) verify each other certificates.

Generated at Fri Jun 27 06:51:05 EEST 2025 using Jira 9.12.4#9120004-sha1:625303b708afdb767e17cb2838290c41888e9ff0.