[ZBX-6815] improve doc regarding using JMX and firewalls Created: 2013 Jul 23  Updated: 2024 Apr 10  Resolved: 2017 Oct 31

Status: Closed
Project: ZABBIX BUGS AND ISSUES
Component/s: Documentation (D)
Affects Version/s: 2.0.6, 2.1.1
Fix Version/s: 4.0 (plan)

Type: Documentation task Priority: Minor
Reporter: Oleksii Zagorskyi Assignee: Martins Valkovskis
Resolution: Fixed Votes: 1
Labels: java, javagateway, jmx, timeout
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Duplicate
duplicates ZBX-5326 JMX cannot connect through firewalls Closed
is duplicated by ZBX-10709 zabbix_java try to connect to localho... Closed
Team: Team A
Sprint: Sprint 17, Sprint 18, Sprint 19
Story Points: 0.25

 Description   

Communication between java proxy and monitored JMX application should not be firewalled, proofs:

https://github.com/gehel/jmx-rmi-agent <- just read description
http://serverfault.com/questions/308662/how-do-i-fix-a-failed-to-retrieve-rmiserver-stub-jmx-error
http://stackoverflow.com/questions/8734981/java-jconsole-jmx-connection-failure
http://olegz.wordpress.com/2009/03/23/jmx-connectivity-through-the-firewall/ <- wery nice one
https://blogs.oracle.com/jmxetc/entry/connecting_through_firewall_using_jmx

Also, if on a server hosted a java app are several network interfaces, there is some requirement to specify "java.rmi.server.hostname" (see olegz.wordpress.com article)
I think we could use Oracle's blog article hyperlink in our doc.

Our doc should be improved (by adding one note) because users have troubles with this.

 Comments   
Comment by Filipp Sudanov (Inactive) [ 2015 Mar 12 ]

Or should we add to the docs recommendation to use com.sun.management.jmxremote.rmi.port as described in this comment:
https://support.zabbix.com/browse/ZBX-5326?focusedCommentId=117571&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-117571

Comment by Oleksii Zagorskyi [ 2015 Aug 18 ]

Not sure it's a good place to leave it, but it still better to finally leave it somewhere than keep it indefinitely in internal notes
The test done 2014-05-16 using corresponding latest zabbix components version.

When java app is stopped:

latest value:
2014.May.16 14:03:08 64093224

zabbix_server.log:

 30214:20140516:140408.729 JMX agent item "jmx["java.lang:type=Memory","HeapMemoryUsage.used"]" on host "it0" failed: first network error, wait for 15 seconds
 30192:20140516:140423.809 JMX agent item "jmx["java.lang:type=Memory","HeapMemoryUsage.used"]" on host "it0" failed: another network error, wait for 15 seconds
 30192:20140516:140438.818 JMX agent item "jmx["java.lang:type=Memory","HeapMemoryUsage.used"]" on host "it0" failed: another network error, wait for 15 seconds
 30192:20140516:140453.828 temporarily disabling JMX agent checks on host "it0": host unavailable

jmx-interface error tool-tip:

"java.io.IOException: Failed to retrieve RMIServer stub: javax.naming.ServiceUnavailableException [Root exception is java.rmi.Co

note that the tool-tip it's a truncated part of zabbix_java.log (here two lines only):

2014-05-16 14:04:08.715 [pool-1-thread-3] WARN com.zabbix.gateway.SocketProcessor - error processing request
com.zabbix.gateway.ZabbixException: java.io.IOException: Failed to retrieve RMIServer stub: javax.naming.ServiceUnavailableException [Root exception is java.rmi.ConnectException: Connection refused to host: 127.0.0.1; nested exception is:

When java-gateway is stopped:

latest value:
2014.May.16 14:12:08 64411856

zabbix_server.log:

 30214:20140516:141308.808 JMX agent item "jmx["java.lang:type=Memory","HeapMemoryUsage.used"]" on host "it0" failed: first network error, wait for 15 seconds
 30192:20140516:141323.986 JMX agent item "jmx["java.lang:type=Memory","HeapMemoryUsage.used"]" on host "it0" failed: another network error, wait for 15 seconds
 30192:20140516:141338.998 JMX agent item "jmx["java.lang:type=Memory","HeapMemoryUsage.used"]" on host "it0" failed: another network error, wait for 15 seconds
 30192:20140516:141354.011 temporarily disabling JMX agent checks on host "it0": host unavailable

jmx-interface error tool-tip:

cannot connect to [[localhost]:10052]: [111] Connection refused

Conclusions:
1) it's not possible to distinguish what's wrong (gateway or java app) using zabbix_server.log only.
2) if jmx-interface error tool-tip contains "java-style" garbage - then jateway is running, but java application probably is not.
3) jmx-interface error tool-tip is too small to include end of error, which is the most useful. NOTE - it was, as for v3.0 we have changed error field lengths from 128 to 2048.
4) jmx-key and TPC port is possible to see only with "debug" level of lib/logback.xml

Comment by Oleksii Zagorskyi [ 2015 Nov 03 ]

A bit more details.
A test with a dummy IP in the internet (which doesn't respond by ICMP too):

2015-11-03 00:29:25.205 [pool-1-thread-1] DEBUG com.zabbix.gateway.SocketProcessor - starting to process incoming connection
2015-11-03 00:29:25.208 [pool-1-thread-1] DEBUG c.z.gateway.BinaryProtocolSpeaker - reading Zabbix protocol header
2015-11-03 00:29:25.208 [pool-1-thread-1] DEBUG c.z.gateway.BinaryProtocolSpeaker - reading 8 bytes of data length
2015-11-03 00:29:25.208 [pool-1-thread-1] DEBUG c.z.gateway.BinaryProtocolSpeaker - reading 130 bytes of request data
2015-11-03 00:29:25.209 [pool-1-thread-1] DEBUG c.z.gateway.BinaryProtocolSpeaker - received the following data in request: {"request":"java gateway jmx","conn":"123.123.123.123","port":12345,"keys":["jmx[\"java.lang:type=Memory\",HeapMemoryUsage.max]"]}
2015-11-03 00:29:25.219 [pool-1-thread-1] DEBUG com.zabbix.gateway.SocketProcessor - dispatched request to class com.zabbix.gateway.JMXItemChecker
2015-11-03 00:29:25.223 [pool-1-thread-1] DEBUG c.z.g.ZabbixJMXConnectorFactory - connecting to JMX agent at 'service:jmx:rmi:///jndi/rmi://123.123.123.123:12345/jmxrmi'
2015-11-03 00:29:28.233 [pool-1-thread-1] WARN  com.zabbix.gateway.SocketProcessor - error processing request
com.zabbix.gateway.ZabbixException: java.net.SocketTimeoutException: connection timed out: service:jmx:rmi:///jndi/rmi://123.123.123.123:12345/jmxrmi
	at com.zabbix.gateway.JMXItemChecker.getValues(JMXItemChecker.java:97) ~[zabbix-java-gateway.jar:na]
	at com.zabbix.gateway.SocketProcessor.run(SocketProcessor.java:63) ~[zabbix-java-gateway.jar:na]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_79]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_79]
	at java.lang.Thread.run(Thread.java:745) [na:1.7.0_79]
Caused by: java.net.SocketTimeoutException: connection timed out: service:jmx:rmi:///jndi/rmi://123.123.123.123:12345/jmxrmi
	at com.zabbix.gateway.ZabbixJMXConnectorFactory.connect(ZabbixJMXConnectorFactory.java:123) ~[zabbix-java-gateway.jar:na]
	at com.zabbix.gateway.JMXItemChecker.getValues(JMXItemChecker.java:89) ~[zabbix-java-gateway.jar:na]
	... 4 common frames omitted
2015-11-03 00:29:28.237 [pool-1-thread-1] DEBUG c.z.gateway.BinaryProtocolSpeaker - sending the following data in response: {"response":"failed","error":"java.net.SocketTimeoutException: connection timed out: service:jmx:rmi:\/\/\/jndi\/rmi:\/\/123.123.123.123:12345\/jmxrmi"}
2015-11-03 00:29:28.238 [pool-1-thread-1] DEBUG com.zabbix.gateway.SocketProcessor - finished processing incoming connection

If server's timeout is 3 seconds, interface error:

ZBX_TCP_READ() failed: [4] Interrupted system call

if server's timeout is 4 seconds, interface error is different:

java.net.SocketTimeoutException: connection timed out: service:jmx:rmi:///jndi/rmi://123.123.123.123:12345/jmxrmi

So in zabbix 3.0 it's nice that default timeout is 4 seconds https://www.zabbix.com/documentation/3.0/manual/introduction/whatsnew300#default_configuration_file_improvements

Comment by Sandeep Singhal [ 2017 Sep 06 ]

We have already increased the timeout from 3 to 4 but still its giving the same error as per below logs.
mx_available,jmx_disable_until,status,name from hosts where status in (0,1,5,6) and flags<>2]
29553:20170906:163029.463 DCsync_configuration() jmxitems : 1 (1009 slots)
29561:20170906:163040.865 In substitute_key_macros() data:'jmx[java.lang:type=Memory,HeapMemoryUsage.used]'
29561:20170906:163040.865 End of substitute_key_macros():SUCCEED data:'jmx[java.lang:type=Memory,HeapMemoryUsage.used]'
29561:20170906:163040.866 JSON before sending [

{"request":"java gateway jmx","conn":"or1010050158074.corp.adobe.com","port":9990,"username":"wflyadmin","password":"wflyadmin@123","keys":["jmx[java.lang:type=Memory,HeapMemoryUsage.used]"]}

]
29561:20170906:163043.867 query [txnlev:1] [update hosts set jmx_disable_until=1504715503,jmx_error='ZBX_TCP_READ() failed: [4] Interrupted system call' where hostid=11736]

Please suggest.

Regards

Comment by Oleksii Zagorskyi [ 2018 May 11 ]

Collecting links... additional nice reading about JMX and ports http://blog.zabbix.com/new-monitoring-possibilities-for-java-applications-in-zabbix-3-4/5972/

Let me quote an interesting part here:

It should be noted that the port in this case will be randomly generated and there may be problems with configuring the firewall. Such cases is where we should use the previous version of the JMX endpoint record, because it allows you to explicitly specify the port.

Generated at Fri Apr 26 18:46:33 EEST 2024 using Jira 9.12.4#9120004-sha1:625303b708afdb767e17cb2838290c41888e9ff0.