Steps to reproduce

Compile the latest stable zabbix with openssl support on 

We configured TLS certificate-based encryption consulting the manuals. Generated a CA certificate, a server certificate, and a client certificate. All three are RSA-based certificates (as the blog indicated that the initial encryption implementation does not support ECC). We used a 4096-bit key for the CA, and 2048-bit keys for the client and server. 

Openssl certificates were generated with: 

cd /home/zabbix/.ssl
openssl genrsa -out ca.key 4096
openssl req -x509 -new -sha256 -nodes -key ca.key -days 1095 -out ca.crt
openssl genrsa -out server.key 2048
openssl req -new -sha256 -key server.key -nodes -out server.csr 
openssl x509 -req -sha256 -days 1095 -in server.csr -CA ca.crt -CAkey ca.key -CAcreateserial -out server.crt
chown zabbix.zabbix *

The client certificate was created identically to the server certificate. The server and CA cert have identical Issuer/Subject strings, but the client cert has a different value for the CN field. 

The server was configured (in the database or web client) to accept only TLS connections and send only TLS connections to a test zabbix proxy. The same was done for the test proxy in its config file. 

Result:

The client keeps trying and failing to establish a TLS handshake. The message appears very strange: 

8052:20181106:114804.883 Unable to connect to the server [HOSTNAME]:10051 [TCP successful, cannot establish TLS to [[HOSTNAME]:10051]: self signed certificate: SSL_connect() set result code to SSL_ERROR_SSL: file ../ssl/statem/statem_clnt.c line 1230: error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed: TLS write fatal alert "unknown CA"]

The interesting thing to note: 'self signed certificate'. Yet our cert is signed by an (admittedly custom) CA. 

openssl verify -verbose -CAfile ca.crt client.crt

Produces an 'OK' result. 

Our symptoms appear very similar to https://support.zabbix.com/browse/ZBX-14386?attachmentOrder=desc. Unfortunately this user did not indicate what their configuration problem was or how they solved it, so it's still an unknown to everyone else. 

There's no useful information in the manual, or anywhere else on the internet for that matter, to be found about this particular exact type of certificate issue though. 

Expected:
The error message is rather cryptic. We expect some useful pointer, but this indicates nothing but 'something is wrong', as clearly 'unknown CA' is a bogus message: there is a CA. Perhaps Zabbix fails to load our ca.crt file, which begs the question: are there additional requirements beyond just generating a basic certifictate. The manual pages are certainly no help; they don't indicate clearly at all which kinds/types of certificates are required.   

While randomly trying things has changed the error message a few times, here we are at a point where there's a huge search space of setting combinations and poor error communication. The proverbial needle in the haystack problem. 

This by the way is true of any number of certificate issues. The messages are cryptic, unclear, and altogether mislead you around the true problem. It would be so much nicer if zabbix could at least verify various basic things that can be wrong: Examples:

1) Supported/unsupported algorithms (e.g. the ECC issue). 
2) Whether it can find the specified cert and key files. 
3) Whether the zabbix user can read the specified cert and key files
4) Whether the cert and key files are of a supported format. 

As right now, there's too many different ways this can go wrong, and a poor error reporting, which makes debugging any TLS problems very hard. (Part of the problem is admittedly openssl itself and the prolification of hundreds of standards that just do the same thing slightly differently).