Current solution - https://www.zabbix.com/documentation/current/manual/encryption:

Currently each encrypted connection opens with a full TLS handshake, no session caching and tickets are implemented
Adding encryption increases the time for item checks and actions, depending on network latency:
For example, if packet delay is 100ms then opening a TCP connection and sending unencrypted request takes around 200ms. With encryption about 1000 ms are added for establishing the TLS connection;
Timeouts may need to be increased, otherwise some items and actions running remote scripts on agents may work with unencrypted connections, but fail with timeout with encrypted.

For example web servers has different options how to deal with SSL/TLS cache:
https://httpd.apache.org/docs/2.4/mod/mod_ssl.html#sslsessioncache
http://nginx.org/en/docs/http/ngx_http_ssl_module.html#ssl_session_cache