ZABBIX BUGS AND ISSUES

JMX monitoring doesn't work for MBeans with attributes which names contain dots

Details

  • Type: Bug Bug
  • Status: Closed Closed
  • Priority: Major Major
  • Resolution: Fixed
  • Affects Version/s: 1.9.9 (beta)
  • Fix Version/s: 2.0.4rc1, 2.1.0
  • Component/s: Java gateway (J)
  • Labels:
  • Environment:
    CentOS, jrockit
  • Zabbix ID:
    RTF

Description

It looks like JMX monitoring doesn't work for MBeans with attributes which names contain dots.

For example attribute "jrockit.gc.pool.heap.used" - full Item key string "jmx["oracle.jrockit.management:type=PerfCounters","jrockit.gc.pool.heap.used"])".

Please see attached screenshot which illustrates the situation.

Issue looks like a major because the most important counters for jrockit contain dots.

Activity

Hide
Vladimir K added a comment -

I would be really glad for any workaround suggestion.

Show
Vladimir K added a comment - I would be really glad for any workaround suggestion.
Hide
Benjamin Goodrich added a comment -

We have some custom mbeans with dots in their names. Seeing the following error in the server log

23768:20120828:091903.317 Item [server-1_application:jmx["app.devops.buildinfo:name=secure","app.name"]] became not supported: No such property: app

Show
Benjamin Goodrich added a comment - We have some custom mbeans with dots in their names. Seeing the following error in the server log 23768:20120828:091903.317 Item [server-1_application:jmx["app.devops.buildinfo:name=secure","app.name"]] became not supported: No such property: app
Hide
Benjamin Goodrich added a comment -

(workaround is to escape () the dot "." character in the string)

Show
Benjamin Goodrich added a comment - (workaround is to escape () the dot "." character in the string)
Hide
Benjamin Goodrich added a comment -

Spoke too soon, looked like it worked when I made the change, but still reporting an error in the log and later turned back to "unsupported"

23767:20120828:102903.954 Item [server-1_application:jmx["app.devops.buildinfo:name=secure","app\.name"]] became not supported: No such property: app\

Show
Benjamin Goodrich added a comment - Spoke too soon, looked like it worked when I made the change, but still reporting an error in the log and later turned back to "unsupported" 23767:20120828:102903.954 Item [server-1_application:jmx["app.devops.buildinfo:name=secure","app\.name"]] became not supported: No such property: app\
Hide
Vladimir K added a comment -

Escaping didn't work.
Went to code and disabled dot related logic inside.

Show
Vladimir K added a comment - Escaping didn't work. Went to code and disabled dot related logic inside.
Hide
Benjamin Goodrich added a comment -

I think "." dot is being used for attributes that return a composite data value. I wonder if an optional third field could be used instead in the JMX definition for composite data types.

Show
Benjamin Goodrich added a comment - I think "." dot is being used for attributes that return a composite data value. I wonder if an optional third field could be used instead in the JMX definition for composite data types.
Hide
dimir added a comment -

In case of "jrockit.gc.pool.heap.used" the returned value isn't composite, isn't it just "java.lang.Long"?

Show
dimir added a comment - In case of "jrockit.gc.pool.heap.used" the returned value isn't composite, isn't it just "java.lang.Long"?
Hide
dimir added a comment - - edited

Any simple example to reproduce it? I do not want to setup Oracle or whatever it takes to use that MBeans and I couldn't find an attribute with dots in MXBean:

http://docs.oracle.com/javase/6/docs/api/java/lang/management/MemoryMXBean.html

I'm using OpenJDK by the way.

Show
dimir added a comment - - edited Any simple example to reproduce it? I do not want to setup Oracle or whatever it takes to use that MBeans and I couldn't find an attribute with dots in MXBean: http://docs.oracle.com/javase/6/docs/api/java/lang/management/MemoryMXBean.html I'm using OpenJDK by the way.
Hide
dimir added a comment -

That's true. Currently Zabbix Java gateway works so that if attribute name contains dot it treats it as a separator between an attribute name with composite data and the field name of that data. We would need to add a way to destinguish an attribute containing composite data from an attribute with simple data that just contains dots.

Show
dimir added a comment - That's true. Currently Zabbix Java gateway works so that if attribute name contains dot it treats it as a separator between an attribute name with composite data and the field name of that data. We would need to add a way to destinguish an attribute containing composite data from an attribute with simple data that just contains dots.
Hide
Benjamin Goodrich added a comment -

I believe that even with a dot "." in the string, it is still a valid JMX attribute?

Show
Benjamin Goodrich added a comment - I believe that even with a dot "." in the string, it is still a valid JMX attribute?
Hide
dimir added a comment - - edited

Sure. We can fix the issue by handling an exceptions when handling attributes with dots. But I was thinking if it would be more obvious to introduce optional 3rd parameter to indicate attributes with composite data:

jmx[oracle.jrockit.management:type=PerfCounters,jrockit.gc.pool.heap,used] (3 parameters, composite data)
jmx[com.example:type=Hello,test.my.attribute] (2 parameters, primitive data)

This would introduce a regression though.

Show
dimir added a comment - - edited Sure. We can fix the issue by handling an exceptions when handling attributes with dots. But I was thinking if it would be more obvious to introduce optional 3rd parameter to indicate attributes with composite data: jmx[oracle.jrockit.management:type=PerfCounters,jrockit.gc.pool.heap,used] (3 parameters, composite data) jmx[com.example:type=Hello,test.my.attribute] (2 parameters, primitive data) This would introduce a regression though.
Hide
Benjamin Goodrich added a comment -

I think it is more clear for composite types than a dot, regardless. But any method that would resolve the issue (including exception handling) is preferable to not handling JMX attributes with dots in them.

Show
Benjamin Goodrich added a comment - I think it is more clear for composite types than a dot, regardless. But any method that would resolve the issue (including exception handling) is preferable to not handling JMX attributes with dots in them.
Hide
dimir added a comment - - edited

We handle next situations:

  • primitive type attribute without dots (simple)
  • primitive type attribute with dots (s.i.m.p.l.e)
  • composite type attribute without dots (fruits.apple)
  • composite type attribute with dots (f.r.u.i.t.s.apple)
  • nested composite type attribute without dots (cars.audi.weight)
  • nested composite type attribute with dots (c.a.r.s.audi.weight)

In order to identify those we process the provided attribute name as follows:

  • if the name contains dots
    • find out the attribute name by starting from left most and appending the right parts
    • if nothing left
      • it's primitive type attribute with dots
    • else
      • what's left is the key of composite data, process it recursively

Let's see the example of processing the last option "c.a.r.s.audi.weight" where the attribute name is "c.a.r.s" which contains composite data "audi" with nested composite data "weight":

  • key with dots
  • "c", "c.a", "c.a.r" will throw the exception, handle it
  • attribute = "c.a.r.s" (this will work)
  • there is non-empty part left ("audi.weight"), get the value by recursively processing each part as composite data
Show
dimir added a comment - - edited We handle next situations:
  • primitive type attribute without dots (simple)
  • primitive type attribute with dots (s.i.m.p.l.e)
  • composite type attribute without dots (fruits.apple)
  • composite type attribute with dots (f.r.u.i.t.s.apple)
  • nested composite type attribute without dots (cars.audi.weight)
  • nested composite type attribute with dots (c.a.r.s.audi.weight)
In order to identify those we process the provided attribute name as follows:
  • if the name contains dots
    • find out the attribute name by starting from left most and appending the right parts
    • if nothing left
      • it's primitive type attribute with dots
    • else
      • what's left is the key of composite data, process it recursively
Let's see the example of processing the last option "c.a.r.s.audi.weight" where the attribute name is "c.a.r.s" which contains composite data "audi" with nested composite data "weight":
  • key with dots
  • "c", "c.a", "c.a.r" will throw the exception, handle it
  • attribute = "c.a.r.s" (this will work)
  • there is non-empty part left ("audi.weight"), get the value by recursively processing each part as composite data
Hide
Benjamin Goodrich added a comment -

So recursively evaluate attribute from left to right, based on if it throws an exception...

  • Evaluate left field, if it does throw an exception, increase left (by one right field) and try again.
  • If it does not throw an exception left part of argument is stored as attribute, right part of argument is further recursively processed based on previous attributes.
  • If down to only "left" field (no more right fields) and there is an exception after processing, throw an error.

In the end you would have a list of attributes and sub-attributes either with or without a dot or an error.

Is that summary correct?

Show
Benjamin Goodrich added a comment - So recursively evaluate attribute from left to right, based on if it throws an exception...
  • Evaluate left field, if it does throw an exception, increase left (by one right field) and try again.
  • If it does not throw an exception left part of argument is stored as attribute, right part of argument is further recursively processed based on previous attributes.
  • If down to only "left" field (no more right fields) and there is an exception after processing, throw an error.
In the end you would have a list of attributes and sub-attributes either with or without a dot or an error. Is that summary correct?
Hide
dimir added a comment - - edited

Yep, that is correct. And what's left on the right side (after we find the attribute part on the left) is the key(s) of the composite data.

Show
dimir added a comment - - edited Yep, that is correct. And what's left on the right side (after we find the attribute part on the left) is the key(s) of the composite data.
Hide
Benjamin Goodrich added a comment -

I see this as a possibility, (attribute.1.attribute.2.key.1)

attribute1=attribute.1
attribute2=attribute.2
key=key.1

Would the algorithm be able to handle this? Could the key value also contain a dot? As long as it can find every possible attribute, then I assume whatever is "left over" on the right side would be the key value?

Show
Benjamin Goodrich added a comment - I see this as a possibility, (attribute.1.attribute.2.key.1) attribute1=attribute.1 attribute2=attribute.2 key=key.1 Would the algorithm be able to handle this? Could the key value also contain a dot? As long as it can find every possible attribute, then I assume whatever is "left over" on the right side would be the key value?
Hide
dimir added a comment -

Please see the update description of the flow above. I hope it's more clear this way. And I have an example application with exactly the attributes like mentioned there.

Show
dimir added a comment - Please see the update description of the flow above. I hope it's more clear this way. And I have an example application with exactly the attributes like mentioned there.
Hide
Benjamin Goodrich added a comment -

What if there was a value 'c.a.r.s.audi.r8.weight.kg' where "c.a.r.s" is the attribute name, with composite data "audi.r8", with nested data "weight.kg"? Is that a possible valid combination?

Show
Benjamin Goodrich added a comment - What if there was a value 'c.a.r.s.audi.r8.weight.kg' where "c.a.r.s" is the attribute name, with composite data "audi.r8", with nested data "weight.kg"? Is that a possible valid combination?
Hide
dimir added a comment - - edited

Good point. Actually this is quite a valid combination. The field name of the composite data can be of any type, including String, which, of course, can contain dots. Such a situation was not handled before. But how to handle it?

I can get all the keys of composite data and then start guessing which part of the name did the user want to use. This would not help for next case though (attribute = "cars"):

  • cars:
    • audi.r8 = "The ultimate car"
    • audi:
      • r8:
        • weight = 123

User provided attribute name "cars.audi.r8" would bring in the confusion. So I guess we have to drop the support for field names with dots.

Show
dimir added a comment - - edited Good point. Actually this is quite a valid combination. The field name of the composite data can be of any type, including String, which, of course, can contain dots. Such a situation was not handled before. But how to handle it? I can get all the keys of composite data and then start guessing which part of the name did the user want to use. This would not help for next case though (attribute = "cars"):
  • cars:
    • audi.r8 = "The ultimate car"
    • audi:
      • r8:
        • weight = 123
User provided attribute name "cars.audi.r8" would bring in the confusion. So I guess we have to drop the support for field names with dots.
Hide
dimir added a comment -

We are currently discussing how to handle such a situation. One way is escape the dots that are not acting as delimiters. Introducing third parameter will not probably suffice, as in case of nested composite data we would need a separate parameter for each one. In the latter case it would be not possible to add a parameter at a later time without introducing a regression.

Show
dimir added a comment - We are currently discussing how to handle such a situation. One way is escape the dots that are not acting as delimiters. Introducing third parameter will not probably suffice, as in case of nested composite data we would need a separate parameter for each one. In the latter case it would be not possible to add a parameter at a later time without introducing a regression.
Hide
Benjamin Goodrich added a comment -

Actually, Vladimir, that sounds like a much simpler and better solution. Nice.

Show
Benjamin Goodrich added a comment - Actually, Vladimir, that sounds like a much simpler and better solution. Nice.
Hide
dimir added a comment -

You mean a separate parameter for each field of composite data? Like:

jmx[objectName:type=Cars, audi, weight]
jmx[objectName:type=Cars, the.trucks, volvo, model.x, weight]

? Yes, this is clean solution but the problem is that if we decide to add an additional parameter at a later time it would introduce a regression, as we would need to add it before the variable-length list.

Show
dimir added a comment - You mean a separate parameter for each field of composite data? Like: jmx[objectName:type=Cars, audi, weight] jmx[objectName:type=Cars, the.trucks, volvo, model.x, weight] ? Yes, this is clean solution but the problem is that if we decide to add an additional parameter at a later time it would introduce a regression, as we would need to add it before the variable-length list.
Hide
Benjamin Goodrich added a comment -

Actually, I was commenting that escaping the dots was a simpler and better solution.

jmx[ObjectName:type=Cars\.model,audi\.r8.weight\.kg]

You are correct that three fields could not definitively work, for instance:(java.lang:type=GarbageCollector,name=ConcurrentMarkSweep,LastGcInfo,memoryUsageAfterGc,CMS Old Gen,committed)

Show
Benjamin Goodrich added a comment - Actually, I was commenting that escaping the dots was a simpler and better solution. jmx[ObjectName:type=Cars\.model,audi\.r8.weight\.kg] You are correct that three fields could not definitively work, for instance:(java.lang:type=GarbageCollector,name=ConcurrentMarkSweep,LastGcInfo,memoryUsageAfterGc,CMS Old Gen,committed)
Hide
dimir added a comment - - edited

Right. So this is how it was done. We added a support for escaping the dot that is not a separator by using a backslash. In case of a backslash symbol in the attribute name user would need to escape it with another backslash. The development branch will be ready a bit later today. This will be fixed for both 2.0 and trunk. Deploying the fix will require recompiling and restarting Zabbix java gateway. No regression introduced.

Show
dimir added a comment - - edited Right. So this is how it was done. We added a support for escaping the dot that is not a separator by using a backslash. In case of a backslash symbol in the attribute name user would need to escape it with another backslash. The development branch will be ready a bit later today. This will be fixed for both 2.0 and trunk. Deploying the fix will require recompiling and restarting Zabbix java gateway. No regression introduced.
Hide
Benjamin Goodrich added a comment -

That's great, thanks for finding a solution for this.

Show
Benjamin Goodrich added a comment - That's great, thanks for finding a solution for this.
Hide
dimir added a comment -

Fixed in development branch: svn://svn.zabbix.com/branches/dev/ZBX-4663

Show
dimir added a comment - Fixed in development branch: svn://svn.zabbix.com/branches/dev/ZBX-4663
Hide
Alexander Vladishev added a comment - - edited

(1) Please review my changes in r30236.

<dimir> Thanks, looks great! CLOSED

Show
Alexander Vladishev added a comment - - edited (1) Please review my changes in r30236. <dimir> Thanks, looks great! CLOSED
Hide
Alexander Vladishev added a comment - - edited

(2) attributes or keys with a composite data doesn't work with trailing backslashes.

Key to reproduce it: jmx[com.example:type=Hello,backslashend\\.a]
where: "backslashend\" - attribute name
       "a" - key

<dimir> RESOLVED in r30243
<Sasha> CLOSED

Show
Alexander Vladishev added a comment - - edited (2) attributes or keys with a composite data doesn't work with trailing backslashes.
Key to reproduce it: jmx[com.example:type=Hello,backslashend\\.a]
where: "backslashend\" - attribute name
       "a" - key
<dimir> RESOLVED in r30243 <Sasha> CLOSED
Hide
dimir added a comment -

Merging this to 2.0 and trunk will have to wait till 2.0 is unfrozen.

Show
dimir added a comment - Merging this to 2.0 and trunk will have to wait till 2.0 is unfrozen.
Hide
Alexander Vladishev added a comment -

Fixed in pre-2.0.4 r30620 and pre-2.1.0 (trunk) r30621.

Show
Alexander Vladishev added a comment - Fixed in pre-2.0.4 r30620 and pre-2.1.0 (trunk) r30621.
Hide
David added a comment -

Is this actually working?

Redacted logs from our 2.0.6 installation:
17801:20130509:213853.364 item [hostname:jmx["Metrics:context=apiCounters",ExceptionCounts.com\.DeviceDisconnectedException"]] became not supported: Argument key="com\" is not an existing item name for this CompositeData instance.

Item key: jmx["Metrics:context=apiCounters",ExceptionCounts.com\.DeviceDisconnectedException"]

It seems to cut off after any dot, no matter what escaping I try. ExceptionCounts is a composite item, the attribute name is "com.DeviceDisconnectedException".

Show
David added a comment - Is this actually working? Redacted logs from our 2.0.6 installation: 17801:20130509:213853.364 item [hostname:jmx["Metrics:context=apiCounters",ExceptionCounts.com\.DeviceDisconnectedException"]] became not supported: Argument key="com\" is not an existing item name for this CompositeData instance. Item key: jmx["Metrics:context=apiCounters",ExceptionCounts.com\.DeviceDisconnectedException"] It seems to cut off after any dot, no matter what escaping I try. ExceptionCounts is a composite item, the attribute name is "com.DeviceDisconnectedException".
Hide
David added a comment -

Apologies, our Java Gateway was still running 2.0.3. This is working for us now. Please disregard the above comment.

Show
David added a comment - Apologies, our Java Gateway was still running 2.0.3. This is working for us now. Please disregard the above comment.

People

Vote (1)
Watch (3)

Dates

  • Created:
    Updated:
    Resolved: