[ZBXNEXT-2231] [Patch] Implement bulk retrieval for JMX items Created: 2014 Mar 29  Updated: 2018 Jan 20

Status: Open
Project: ZABBIX FEATURE REQUESTS
Component/s: Java gateway (J)
Affects Version/s: 2.2.2
Fix Version/s: None

Type: Change Request Priority: Major
Reporter: A B Assignee: Unassigned
Resolution: Unresolved Votes: 1
Labels: bulk, javagateway, patch, unsquashable
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: File bulk-jmx.patch     File zbxnext-2231-v34.patch     File zbxnext-2231-v345.patch     File zbxnext-2231.patch    

 Description   

Currently the Java gateway sends a separate JMX getAttribute() request for each item. This is quite wasteful, as JMX supports bulk retrieval via getAttributes(). Furthermore, for items from composite objects the gateway currently ends up requesting the exact same object multiple times (in our case we literally have about 100 items in one composite object, so the object is being fetched 100 times, forcing us to add special caching on the server for this object). The result of this is a noticeable load on our monitored servers.

The attached patch implements bulk retrieval in the gateway, including for discovered items. The approach is basically:
1. group all requested items by jmx-object (so we can do a bulk attribute retrieval for each), and for each jmx-object collect the list of (top-level) attributes (so composite objects are only retrieved once)
2. if discovery is requested, do discovery and add those items to the groupings from the previous step
3. for each jmx-object do a bulk retrieval of all requested attributes
4. generate the response by extracting the value for each item from the retrieved data in the previous step

This is working in our production and has noticeably lowered the load on both the monitored servers and zabbix.

A further enhancement would be improve the zabbix poller to submit requests for items in the same jmx-object in one go, similar to what was discussed for SNMP in ZBXNEXT-98 .

The patch is against branches/2.2 r43648.

 Comments   
Comment by Oleksii Zagorskyi [ 2014 Mar 30 ]

https://www.zabbix.com/documentation/2.2/manual/concepts/java
says:

Zabbix server or proxy will try to pool requests to a single JMX target together as much as possible (affected by item intervals) and send them to the Java Gateway in a single connection for better performance.

For me it doesn't match a bit what AB said about further enhancement.
Would be good to clarify.

Comment by Marc [ 2014 Mar 30 ]

Haven't investigated the code yet.

But from my understanding requests are currently pooled together to send them over a single tcp connection and this patch should enable to receive multiple values of the same MBean by one request.

Comment by A B [ 2014 Apr 03 ]

Marc is exactly correct. I'm sorry if I wasn't clear before, so let me try again. Currently the zabbix poller will collect a bunch of items (often 30 or 40) to be updated and send that as a single request to the java gateway. The java gateway will then open a single JMX/TCP connection, but will then send a separate request (MBeanServerConnection.getAttribute(jmx-object, attribute)) for each item. Now, often there are several items that correspond to different attributes of the same jmx-object (e.g., in the "Template JMX Generic" you'll find jmx["java.lang:type=ClassLoading",LoadedClassCount], jmx["java.lang:type=ClassLoading",TotalLoadedClassCount], and jmx["java.lang:type=ClassLoading",UnloadedClassCount]). So instead of doing three separate requests for these this patch will issue a single MBeanServerConnection.getAttributes("java.lang:type=ClassLoading", [LoadedClassCount, TotalLoadedClassCount, UnloadedClassCount]) request.

That is the first optimization. The second one is even bigger in our case: suppose one of the attributes returns a composite object. So, say, you have items jmx[obj,attr.a1.b1], jmx[obj,attr.a1.b2], jmx[obj,attr.a2.b1], jmx[obj,attr.a2.b2], etc. These are all "independent" items in zabbix's view, but they are all values from the same composite object. And currently the java gateway will end up issuing a separate getAttribute() request for each of these items, even though each request is exactly the same (namely MBeanServerConnection.getAttribute(obj, attr)). So the second optimization is to only issue a single request in this case. As I said in the description, we have composite objects with around 100 items (think a1...a20, and b1...b5), so this optimization is quite significant in our case.

I hope this clarifies things a bit.

Comment by Aleksandrs Saveljevs [ 2014 Apr 10 ]

roadrunner, this would be a great improvement to the Java gateway. It is out of scope for today's bugsquash though.

Comment by A B [ 2014 Apr 12 ]

I cleaned up the patch a little bit after realizing it didn't compile under java 6 (it was using the diamond operator) - not sure if java 6 compatibility is really required, but at least the ubuntu package appears to have a dependency on 6 instead of java 7.

Anyway, besides removing the use of the diamond operator, I also added some comments on the new datastructures and created essentially a type alias for one map to simplify and clarify things a little.

In short, the functionality and behaviour of this new patch is identical to the previous, which the changes being mostly syntactic to (hopefully) improve readability.

Comment by A B [ 2017 Dec 16 ]

Updated patch for zabbix 3.4 (in particular due to the new jmx discovery items).

Comment by A B [ 2018 Jan 20 ]

Updated patch for zabbix 3.4.5.

Generated at Fri Apr 19 17:07:45 EEST 2024 using Jira 9.12.4#9120004-sha1:625303b708afdb767e17cb2838290c41888e9ff0.