[ZBX-5271] JMX interface never gets to state up again after it failed Created: 2012 Jul 04  Updated: 2017 May 30  Resolved: 2015 Mar 13

Status: Closed
Project: ZABBIX BUGS AND ISSUES
Component/s: Server (S)
Affects Version/s: 2.0.1
Fix Version/s: None

Type: Incident report Priority: Critical
Reporter: Simon Balz Assignee: Unassigned
Resolution: Cannot Reproduce Votes: 4
Labels: interface, javagateway, jmx
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

CentOS 5.7 x86_64, both Zabbix server and Java bridge are 2.0.1


Issue Links:
Duplicate
is duplicated by ZBX-5306 Java Gateway blocks and doesn't gathe... Closed
is duplicated by ZBX-5593 Zabbix Java Gateway needs to be resta... Closed

 Description   

If your JMX interface of a monitored host or the Java bridge was down when Zabbix server tried to poll JMX items, the interface get marked as unavailable, with a message like
cannot connect to [[127.0.0.1]:10052]: [111] Connection refused

After bringing up the Java bridge again, Zabbix server doesn't restart the polling for items belonging to this interface.



 Comments   
Comment by Simon Balz [ 2012 Jul 04 ]

Is there a workaround to re-enable the interface manually?

Comment by richlv [ 2012 Jul 04 ]

similar issue ZBX-5226 turned out to be a misconfiguration

Comment by Simon Balz [ 2012 Jul 04 ]

I've checked ZBX-5226 already before, but I didn't touch the configuration before or after stopping/starting my Java gateway.
Further I have some hosts with JMX interfaces, where it still works fine, only interfaces where Zabbix tried to pull exactly at the time the Gateway was down, don't get green again.
When I track the JMX pulls in zabbix_server.log, I can see log entries like

5397:20120704:160254.989 In get_values_java() host:'myhost1' addr:'192.168.2.68' num:4

for hosts/interfaces working but I miss a log line for the the host/interface which is marked down. So I have the impression, Zabbix server doesn't even try to reach it.

Comment by Simon Balz [ 2012 Jul 23 ]

Hi

Is there any progress on this issue?
This issue is pretty blocking for me. I have several hosts with a JMX interface and items which aren't polled anymore. The items itself are active but the interface is marked down. The java gateway doesn't report any connect attempts.
Is there any way to force re-enable the JMX interface?

Thanks

Comment by richlv [ 2012 Jul 23 ]

this issue not scheduled for short term currently. depending on where the interfaces are disabled, you can try restarting server/gateway, or looking at the database

Comment by richlv [ 2012 Jul 24 ]

can't reproduce with 2.0.2rc1 and openjdk 1.6.0_24

if i shut down the monitored java process, it shows up as down and then recovers properly when process is started up again.
if i shut down java gateway, it shows up as down and then recovers properly when gateway is started up again.

Comment by Tim McCune [ 2013 May 02 ]

I found a workaround. I found the host's entry in the "hosts" table in the zabbix database, and changed jmx_disable_until to 0, changed jmx_available to 1, and jmx_error_from to 0. Restarted zabbix-server. Stuff started working again.

Comment by Anna Somova [ 2013 Jun 26 ]

In my case bundles have become available, but on the monitor I see only "no data". In the database jmx_disable_until and jmx_error fields updates with new error values. Any ideas?

Comment by Aleksandrs Saveljevs [ 2014 Feb 17 ]

We have tried various timeout scenarios, but cannot reproduce the problem at the moment.

Those of you who still experience the problem with the latest versions, do you have a reliable way of reproducing it?

Would it be possible for you to run Java gateway with a "trace" debug level?

In order to increase the debugging level, change "info" to "trace" in lib/logback.xml:

<root level="info">
	<appender-ref ref="FILE" />
</root>

Similarly, in order to increase the maximum log file size, change "maxFileSize" parameter:

<triggeringPolicy class="ch.qos.logback.core.rolling.SizeBasedTriggeringPolicy">
	<maxFileSize>5MB</maxFileSize>
</triggeringPolicy>

When the problem happens, is it possible to connect to the Java gateway? For instance, by telnetting to its port or running a debugging utility from Zabbix source code like follows:

echo '{"request":"java gateway internal","keys":["zabbix[java,,ping]","zabbix[java,,version]"]}' | misc/debug/sender.pl -h 127.0.0.1 -p 10052 -i -
Comment by Aleksandrs Saveljevs [ 2014 Aug 26 ]

If there is anybody that still experiences this problem with the latest gateway versions, please tell us.

For those of you having this problem with JBoss, please check whether ZBX-8579 solves your problem.

Comment by Aleksandrs Saveljevs [ 2014 Oct 01 ]

Looks like ZBX-8839 might explain the problem people were having here.

Comment by Aleksandrs Saveljevs [ 2015 Mar 13 ]

There is no response, so closing the issue as "Cannot Reproduce". Fix in ZBX-8839 has just been merged, too. Please reopen if the issue appears again with ZBX-8839 fix in place.

Comment by Neiro [ 2015 Apr 14 ]

Hi,

Confirmed the bug

OS: Red Hat Enterprise Linux Server release 6.6 (Santiago)
Zabbix-server: 2.0.1
java-gateway: 2.0.0


After this (see near) works fine!

mysql> select host, jmx_disable_until, jmx_available,  jmx_errors_from from hosts where host="srv14";
+-----------+-------------------+---------------+-----------------+
| host      | jmx_disable_until | jmx_available | jmx_errors_from |
+-----------+-------------------+---------------+-----------------+
| srv14 |        1583430260 |             1 |      1426054531 |
+-----------+-------------------+---------------+-----------------+
1 row in set (0.00 sec)

mysql> update hosts set jmx_disable_until=0, jmx_available=1, jmx_errors_from=0 where  host="srv14"; 
Query OK, 1 row affected (0.00 sec)
Rows matched: 1  Changed: 1  Warnings: 0


mysql> select host, jmx_disable_until, jmx_available,  jmx_errors_from from hosts where host="srv14";
+-----------+-------------------+---------------+-----------------+
| host      | jmx_disable_until | jmx_available | jmx_errors_from |
+-----------+-------------------+---------------+-----------------+
| srv14 |                 0 |             1 |               0 |
+-----------+-------------------+---------------+-----------------+
1 row in set (0.00 sec)

& RESTART zabbix_server and java_gateway

Comment by Aleksandrs Saveljevs [ 2015 Apr 15 ]

Neiro, as suggested in my last comment above, please try the latest version with ZBX-8839 fix included. Versions 2.0.0 and 2.0.1 are too old.

Generated at Mon Apr 28 09:17:58 EEST 2025 using Jira 9.12.4#9120004-sha1:625303b708afdb767e17cb2838290c41888e9ff0.