Steps to reproduce:

1. Enable SSL in a remote Java server that you wish to monitor with JMX from Zabbix. You can use these system properties:
   1. -Dcom.sun.management.jmxremote.port=12345
   2. -Dcom.sun.management.jmxremote.authenticate=false
   3. -Dcom.sun.management.jmxremote.ssl=true
   4. -Dcom.sun.management.jmxremote.registry.ssl=true
   5. -Djavax.net.ssl.keyStore=/path/to/keystore
   6. -Djavax.net.ssl.keyStorePassword=mypasswd
   7. -Djavax.net.ssl.trustStore=/path/to/truststore
   8. -Djavax.net.ssl.trustStorePassword=mypasswd
2. Configure the Zabbix Java Gateway's truststore so it can trust the remote Java server.
3. Create a JMX connection from Zabbix to the remove Java server (service:jmx:rmi:///jndi/rmi://<host>:12345/jmxrmi) and try to connect.

Result:

Error:

non-JRMP server at remote endpoint: service:jmx:rmi:///jndi/rmi://<host>:12345/jmxrmi

Workaround:

If you set -Dcom.sun.management.jmxremote.registry.ssl=false (or if you remove the property, because this is the default value for this system property) in the remote Java server, then the connection from Zabbix is successful. Note that this setting is insecure and not recommended. In the JMX documentation (https://docs.oracle.com/en/java/javase/11/management/monitoring-and-management-using-jmx-technology.html#GUID-BA10AD87-78E8-4248-B648-D02284D21626):

[...]To protect the RMI registry using SSL, you must set the following system property:

com.sun.management.jmxremote.registry.ssl=true

When this property is set to true, an RMI registry protected by SSL will be created and configured by the ready-to-use management agent when the Java VM is started. The default value of this property is false. However, it is recommended that you set this property to true.[...]

Expected:

No errors.