Moreover, I am sure it would be handy to have some sort of way of allowing SSL connections to servers using a self-signed certificate. (e.g. DEV, SIT, UAT Servers)

After some digging, I added this to my own version of the java gateway to skip SSL cert verification

System.setProperty("com.sun.net.ssl.checkRevocation", "false");

        TrustManager[] trustAllCerts = new TrustManager[]{new X509TrustManager() {
            public java.security.cert.X509Certificate[] getAcceptedIssuers() {
                return null;
            }

            public void checkClientTrusted(X509Certificate[] certs, String authType) {
            }

            public void checkServerTrusted(X509Certificate[] certs, String authType) {
            }
        }
        };

        try {
            logger.info("Hacking SSL Validation");
            SSLContext sc = SSLContext.getInstance("SSL");
            sc.init(null, trustAllCerts, new java.security.SecureRandom());            
            SSLContext.setDefault(sc);
            logger.info("SSL Validation hacked");
        } catch (Exception e) {
            throw new RuntimeException(e);
        }

If you set up SSL command-line parameters as described at https://www.zabbix.com/documentation/2.4/manual/config/items/itemtypes/jmx_monitoring , does it start working?

Possibly like that or with some specific Glassfish properties to disable JMX auth and JMX SSL....

But we wouldn't want to connect to prod servers without SSL or without credentials either. Why are you asking?

The issue description suggests modifying Java gateway code. However, the page at https://www.zabbix.com/documentation/2.4/manual/config/items/itemtypes/jmx_monitoring suggests adding command-line parameters like the following to get SSL working:

java \
-Djava.rmi.server.hostname=192.168.3.14 \
-Dcom.sun.management.jmxremote \
-Dcom.sun.management.jmxremote.port=12345 \
-Dcom.sun.management.jmxremote.authenticate=true \
-Dcom.sun.management.jmxremote.password.file=/etc/java-6-openjdk/management/jmxremote.password \
-Dcom.sun.management.jmxremote.access.file=/etc/java-6-openjdk/management/jmxremote.access \
-Dcom.sun.management.jmxremote.ssl=true \
-Djavax.net.ssl.keyStore=$YOUR_KEY_STORE \
-Djavax.net.ssl.keyStorePassword=$YOUR_KEY_STORE_PASSWORD \
-Djavax.net.ssl.trustStore=$YOUR_TRUST_STORE \
-Djavax.net.ssl.trustStorePassword=$YOUR_TRUST_STORE_PASSWORD \
-Dcom.sun.management.jmxremote.ssl.need.client.auth=true \
-jar /usr/share/doc/openjdk-6-jre-headless/demo/jfc/Notepad/Notepad.jar

In your case, only a subset may be needed or maybe one should use other properties specific to Glassfish. So I was wondering whether you can achieve the same effect (get SSL working) by modifying the way Java gateway is started, but not changing the code itself?

Ok, I got you now.

With that bit of code there, you can enable SSL on the target java application and you can also protect the jmx connection with user name and password (glassfish has both SSL + Auth enabled by default)

With the keyStore settings on that bit of code are for the the application to find an appropiate SSL certificate.

But that bit of code is for the application the gateway will be connecting to (NotePad, GlassFish, etc)

The problem on the java gateway itself is that it can not open a SSL connection to GlassFish unless you use specify the SSL socket factory SslRMIClientSocketFactory(). Not sure how it would work when connecting to the sample NotePad application but when attempting to do so against glassfish it says: Failing to do this returns in IOEXceptiom: non JRMP endpoint on remote server.

I also looked into how to achieve this through JVM command line parameters passed to the java gateway on startup but I couldn't figure out how.

Another reason that I am asking is that if it is possible to make Java gateway connect to Glassfish using SSL using command-line parameters, then this is the preferred way and this issue should be closed as "Won't fix". We should only leave this issue open if the code change is really necessary for it to work. If you are not sure, it would be wonderful if you would discuss that at https://www.zabbix.org/wiki/Getting_help .

Hey Alek,

I understand what you mean. I am not aware of any command line parameters to specify the socket type. It may work with -Dcom.sun.jndi.rmi.factory.socket= fully classified class name. Not sure, problem being is that the gateway may only work in SSL mode by doing so?

My skype id is pablo.anahata-it.com.au if you want to chat.

Alex,

I am on the zabbix IRC channel now.

We created a fork of the java gateway using maven with the workaround for self-signed SSL certificates and SSL over JMX. It is here https://bitbucket.org/anahata/anahata-zabbix-java-gateway

I think it needs a bit of improvement around pooling JMX connections.

We are happy to help with the java side of things. You can email me on [email protected] if you need a hand on the java side. Otherwise skype / irc good too.

We seem to have it documented at https://www.zabbix.com/documentation/2.4/manual/config/items/itemtypes/jmx_monitoring how to enable SSL for a basic monitored application, but we do not seem to give an example on how to enable SSL for Java gateway. I should probably remember how it is done and then we will decide what to do with this issue - will it only require a documentation update or some code changes are necessary, too.

Alek, what is documented here https://www.zabbix.com/documentation/2.4/manual/config/items/itemtypes/jmx_monitoring is the process to launch a java application (like the NotePad example) so it can be monitored with and without SSL.

Glassfish or Java EE application servers (what is more often monitored) the JMX configuration is different, for example in GlassFish, to lookup a JMX connection from a remote client (such as the zabbix java gateway) one needs to specify the SSLRmi socket factory as Glassfish exposes JMX through RMI.

Ok, so what I would do if i was you Alek, is try to see how many java ee servers with SSL zabbix can monitor. The most common Java EE Servers are: GlasFish/Payara, Tomcat/TomEE, WebLogic/WildFly, Geronimo, WebSphere

quite offtopic, but in the region where zabbix is developed, that shortening is not common - you might just go with the full form of "Aleksandrs" (although the correct language form would be "Aleksandr", we can leave that topic for the zabbix conference or a similar event )

I am from Spain but here in Australia everything gets shortened, I am not aware of any other country in the world which shortens names more than Australia, but yes, I agree with you, does need further discussion.

pranahata, I will not have time to research this issue on this occasion. For now, you might wish to consult with https://www.zabbix.org/wiki/Getting_help if you run into any SSL issues with Java gateway.

Having just run into this issue on Tomcat, I think I can clarify the confusion a bit.

The problem stems from the fact that two different services/ports/connections are involved when using Java RMI:

• first a connection/request to the RMI registry on the server is made in order to look up the (JMX management) object,
• and then a connection/request to the object's port in order to make the actual RMI (JMX) calls.

Now, enabling SSL as described in the zabbix docs will enable SSL only for the object connection, but the registry connection will still be insecure (non-SSL). This used to be a common setup.

However, it appears that Glassfish (as inferred from the above comments), and newer versions of Tomcat (specifically versions > 7.0.40 of the tomcat-catalina-jmx-remote lib) also run the registry on SSL, and that is not currently supported by zabbix and would require the above rmi-connection-factory changes.

A custom trust-manager as described in the first comment is not necessary (nor desired) if the javax.net.ssl.trustStore system property has been properly set as described in the docs.

To emulate those troubles for a java app it's possible to force SSL for RMI registry by this option for JVM:

-Dcom.sun.management.jmxremote.registry.ssl=true

To test such connection you can use jmxterm with option "-s" when connecting to remote host. From its help:

-s --sslrmiregistry Whether the server's RMI registry is protected with SSL/TLS

I can confirm for sure that it's not possible to do something like that in current java gateway.

Here I found a fork of gateway and it looks like modification is simple and possible.
https://github.com/apizone-projects/zab_java/blob/master/src/com/zabbix/gateway/JMXItemChecker.java#L111
Yes, it uses the same technic as mentioned initially.

jmxterm does following if -s option used:

          if (options.isSecureRmiRegistry()) {
            // Required to prevent "java.rmi.ConnectIOException: non-JRMP server at remote endpoint"
            // error
            env.put("com.sun.jndi.rmi.factory.socket", new SslRMIClientSocketFactory());
          }

For example currently it's not possible to monitor Cassandra (v3.11.9) because if forcibly enables SSL for RMI registry (somewhere internally, not on JVM level) and I could not find a way to disable it.
I've tried to set "-Dcom.sun.management.jmxremote.registry.ssl=false" for JVM but looks like it's ignored, as expected.

Produced error:

non-JRMP server at remote endpoint

So we really need such a feature in zabbix.

Simple way to check that the JMX agent requires SSL for RMI registry is to check from command line this way, select any:

curl -vvv -k https://127.0.0.1:7199
openssl s_client -showcerts -connect 127.0.0.1:7199

if you see that certificate is received when opening connection - SSL for RMI registry must be used.

Available in versions:

• pre-4.0.30rc1 20bf125a83
• pre-5.0.10rc1 b445fac545
• pre-5.2.6rc1 ccc249c5f8
• pre-5.4.0beta1 3a9d3250ac

Updated documentation:

• "JMX monitoring" for 5.4, 5.2, 5.0 and 4.0

Technical detail from developers:

What this change does is that it tries to make a regular non-SSL connection to the registry SSL and if it does not work - it then tries to use SSL connection.
It caches which method (non-SSL or SSL) was successful the last time, (cache is cleared every 24 hours).
It caches it for every JMX endpoint separately.