Zabbix Server to monitor Java applications on two remote hosts behind a strict firewall via JMX

note that proxy can be used with sqlite, which is very, very simple to set up.

also, what's "blackboard thread"?

sry, must be "bulletin board", not "blackboard"

Proxy isn't to set up at all, because for the given SLES 11.2 some mandatory devel packages are not available. Static linked compiling doesn't work properly either. Someone "solved" the problem switching over to dynamically linking, see: http://www.zabbix.com/forum/showthread.php?t=21100 It took me a while to understand...

Please don't mind, developing such a great project is something that I definitely appreciate.

I hereby only want to point out that in situations when you aren't in control of the environment for some reason it would be great to stay as simple as possible.

There is an add-on for newer tomcat versions that fixes the issue by fixing the ports, btw, but I have to upgrade the 4-years-old Tomcat before; I hope I will become allowed to. See "catalina-jmx-remote.jar" for details.

The things are slightly different from what I thought they were. Connecting to a remote server using JMX DOES NOT REQUIRE the server to be able to CONNECT BACK on an arbitrary port. It is the case that the server opens an arbitrary port for INCOMING communication.

However, if there is a firewall in between, this still won't work as the port is not predictable.

There is a WORKAROUND if the application you like to monitor is Apache Tomcat ≥ 6.0.24

1.) Download the file http://archive.apache.org/dist/tomcat/tomcat-6/v

/bin/extras/catalina-jmx-remote.jar into Tomcat’s /lib folder.

2.) Add the following line to servers.xml, where port1 and port2 are the ports that should be used. I read that it is possible to set both ports the same value but didn’t try it by now:
<Listener className="org.apache.catalina.mbeans.JmxRemoteLifecycleListener" rmiRegistryPortPlatform="

" rmiServerPortPlatform="

:localhost:

-N
port 1 and port 2 are the ports you set in your server.xml. They should be unused on your zabbix machine, too!

Without SSH, open both ports in your firewall.

9.) With SSH, set Configuration → Hosts → JMX interfaces to point to localhost, IP 127.0.0.1, Port = port 1
Without SSH, set JMX interface to your host and port 1

Thanks to

• http://gabenell.blogspot.de/2010/04/connecting-to-jmx-on-tomcat-6-through.html and
• http://www.liferay.com/web/thiago.moreira/blog/-/blogs/11229431
  where I found this mainly.

You can also use the add-on jar for Tomcat (e.g. catalina-jmx-remote.jar - tomcat7 - extras) that enables the JMX Remote Lifecycle Listener.

Once the lib is in place, (tomcat_home/lib) you can put the following in your server.xml

<Listener className="org.apache.catalina.mbeans.JmxRemoteLifecycleListener"
rmiRegistryPortPlatform="10001" rmiServerPortPlatform="10002" />

Now only ports 10001 and 10002 need to be set on the firewall. (of course either of these ports can be set to anything)

Any ideas when we could have a workable solution on this? It's really annoying in our environment.

BR,
Janis

it's not on the roadmap currently

I recompiled the Java GW myself to fix this issue:

vi zabbix-2.2.1/src/zabbix_java/src/com/zabbix/gateway/JMXItemChecker.java
// url = new JMXServiceURL("service:jmx:rmi:///jndi/rmi://" + conn + ":" + port + "/jmxrmi");
url = new JMXServiceURL("service:jmx:rmi://" + conn + ":" + port + "/jndi/rmi://" + conn + ":" + port + "/jmxrmi");

The URL then becomes of the form service:jmx:rmi://$HOSTNAME:$PORT/jndi/rmi://$HOSTNAME:$PORT/jmxrmi

Then make sure the Java arguments reflect the same port:
-Dcom.sun.management.jmxremote.port=$PORT
-Dcom.sun.management.jmxremote.rmi.port=$PORT

I am also voting for this bug to be resolved soon.
There can be a workaround for Tomcat, but what about the other applications ?
Until this bug gets fixed, Zabbix Java Gateway cannot be used in production environment without getting our hands dirty.

@ Jamshid Damanpour

How to configure the complex connection parameter in Zabbix UI

@Raghu GS
There's nothing complicated in the UI. The host just needs one JMX interface address/port and that will be passed to both instances of conn and port in that Java call (JMXServiceURL). If you have multiple java executables with listeners on multiple ports, define them all and reference the appropriate one in each item.
e.g.
Item Key = jmx["java.lang:type=Memory",HeapMemoryUsage.used]

This is not a Zabbix issue, it is a JMX issue. com.sun.management.jmxremote.port and com.sun.management.jmxremote.rmi.port may also be set to the same port without issue so only one port is used and needs to be opened in any firewalls (com.sun.management.jmxremote.rmi.port is only available in 1.7u25 and later). My stock java gateway has no issues retrieving JMX data in this manner.

I recommend closing this ticket.

Please can you test the patch service_jmx.patch in https://support.zabbix.com/browse/ZBXNEXT-1274

You must also change the db.

In case you use mysql you can execute this

mysql zabbix

ALTER TABLE interface CHANGE dns dns varchar(255);

This is an old post, but this problem affected me this week. So, I want to share my solution.

First of all: this isn't a zabbix's bug nor a zabbix-java-gateway's bug.

How RMI works?

Read this post: http://www.netcluesoft.com/rmi-through-a-firewall.html.

How Zabbix monitors JMX?

1. Zabbix connects to java gateway by tcp port 10052 (default port), which can be configured in /etc/zabbix/zabbix_java_gateway.conf (if you have installed it through apt-get). The parameter is LISTEN_PORT
2. Java gateway will try to access JMX on remote host using tcp port 12345 (default), which needs to be set on remote host using this java command line argument "-Dcom.sun.management.jmxremote.port". Normally it is configured in JAVA_OPTS environment variable.
3. The remote host will answer to zabbix-java-gateway suggesting to it to use another random port to access RMI information. This occurs because RMI was implemented to use RPC (Remote Procedure Call). This is the problematic step when you use a firewall between zabbix-java-gateway and the remote host running java, because you don't know which tcp port will be negociated.

How to solve this problem?

1. On the remote machine, add these two paramters in JAVA_OPTS along with the other JMX options:
   -Djava.rmi.server.hostname=$REMOTE_HOST_IP_ADDR
   -Dcom.sun.management.jmxremote.rmi.port=12346.
   • The first one will help to solve the bellow problem, as explained in details here. $REMOTE_HOST_IP_ADDR is the IP address that was configured in "JMX interfaces" of host definition in Zabbix web interface.

     com.zabbix.gateway.ZabbixException: java.rmi.ConnectException: Connection refused to host: 127.0.0.1; nested exception is: 
             java.net.ConnectException: Connection refused
     

   • The second parameter will tell to java to use only that defined port to make RMI connections. So, when java on remote host answers to a zabbix-java-getway JMX request, it will not try to negociate a random tcp port. It will tell to zabbix-java-getway to connect to 12346 instead.
2. Go into your firewall and open these two ports: 12345 (for JMX) and 12346 (for RMI)
3. Restart you java application on remote host
4. Download this script and make a test with a JMX item in Zabbix server's shell

   $ zabbix_get_jmx 127.0.0.1 10052 $REMOTE_HOST_IP_ADDR 12345 'jmx[\"java.lang:type=Memory\",HeapMemoryUsage.max]'
   

5. You must see an answer like this:

   {"response":"success","data":[{"value":"2863661056"}]}
   

Now, be happy and go drink a beer

bruno.galindro,

How about moving your great investigation to the official Wiki and remaining a hyper-link to there instead?

Marc

In which topic could I put this info?

bruno.galindro,

Just create a new site and maybe consider to add respective category tags for 'Bash', 'Howto‏‎' and 'JMX'.

Done:

http://zabbix.org/wiki/ConfigureJMX

As it turned out not to be a bug in Zabbix, closing as Won't Fix.