ZABBIX BUGS AND ISSUES
  1. ZABBIX BUGS AND ISSUES
  2. ZBX-4663

JMX monitoring doesn't work for MBeans with attributes which names contain dots

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 1.9.9 (beta)
    • Fix Version/s: 2.0.4rc1, 2.1.0
    • Component/s: Java gateway (J)
    • Labels:
    • Environment:
      CentOS, jrockit

      Description

      It looks like JMX monitoring doesn't work for MBeans with attributes which names contain dots.

      For example attribute "jrockit.gc.pool.heap.used" - full Item key string "jmx["oracle.jrockit.management:type=PerfCounters","jrockit.gc.pool.heap.used"])".

      Please see attached screenshot which illustrates the situation.

      Issue looks like a major because the most important counters for jrockit contain dots.

        Activity

        Hide
        Vladimir K added a comment -

        I would be really glad for any workaround suggestion.

        Show
        Vladimir K added a comment - I would be really glad for any workaround suggestion.
        Hide
        Benjamin Goodrich added a comment -

        We have some custom mbeans with dots in their names. Seeing the following error in the server log

        23768:20120828:091903.317 Item [server-1_application:jmx["app.devops.buildinfo:name=secure","app.name"]] became not supported: No such property: app

        Show
        Benjamin Goodrich added a comment - We have some custom mbeans with dots in their names. Seeing the following error in the server log 23768:20120828:091903.317 Item [server-1_application:jmx ["app.devops.buildinfo:name=secure","app.name"] ] became not supported: No such property: app
        Hide
        Benjamin Goodrich added a comment -

        (workaround is to escape () the dot "." character in the string)

        Show
        Benjamin Goodrich added a comment - (workaround is to escape () the dot "." character in the string)
        Hide
        Benjamin Goodrich added a comment -

        Spoke too soon, looked like it worked when I made the change, but still reporting an error in the log and later turned back to "unsupported"

        23767:20120828:102903.954 Item [server-1_application:jmx["app.devops.buildinfo:name=secure","app\.name"]] became not supported: No such property: app\

        Show
        Benjamin Goodrich added a comment - Spoke too soon, looked like it worked when I made the change, but still reporting an error in the log and later turned back to "unsupported" 23767:20120828:102903.954 Item [server-1_application:jmx ["app.devops.buildinfo:name=secure","app\.name"] ] became not supported: No such property: app\
        Hide
        Vladimir K added a comment -

        Escaping didn't work.
        Went to code and disabled dot related logic inside.

        Show
        Vladimir K added a comment - Escaping didn't work. Went to code and disabled dot related logic inside.
        Hide
        Benjamin Goodrich added a comment -

        I think "." dot is being used for attributes that return a composite data value. I wonder if an optional third field could be used instead in the JMX definition for composite data types.

        Show
        Benjamin Goodrich added a comment - I think "." dot is being used for attributes that return a composite data value. I wonder if an optional third field could be used instead in the JMX definition for composite data types.
        Hide
        dimir added a comment -

        In case of "jrockit.gc.pool.heap.used" the returned value isn't composite, isn't it just "java.lang.Long"?

        Show
        dimir added a comment - In case of "jrockit.gc.pool.heap.used" the returned value isn't composite, isn't it just "java.lang.Long"?
        Hide
        dimir added a comment - - edited

        Any simple example to reproduce it? I do not want to setup Oracle or whatever it takes to use that MBeans and I couldn't find an attribute with dots in MXBean:

        http://docs.oracle.com/javase/6/docs/api/java/lang/management/MemoryMXBean.html

        I'm using OpenJDK by the way.

        Show
        dimir added a comment - - edited Any simple example to reproduce it? I do not want to setup Oracle or whatever it takes to use that MBeans and I couldn't find an attribute with dots in MXBean: http://docs.oracle.com/javase/6/docs/api/java/lang/management/MemoryMXBean.html I'm using OpenJDK by the way.
        Hide
        dimir added a comment -

        That's true. Currently Zabbix Java gateway works so that if attribute name contains dot it treats it as a separator between an attribute name with composite data and the field name of that data. We would need to add a way to destinguish an attribute containing composite data from an attribute with simple data that just contains dots.

        Show
        dimir added a comment - That's true. Currently Zabbix Java gateway works so that if attribute name contains dot it treats it as a separator between an attribute name with composite data and the field name of that data. We would need to add a way to destinguish an attribute containing composite data from an attribute with simple data that just contains dots.
        Hide
        Benjamin Goodrich added a comment -

        I believe that even with a dot "." in the string, it is still a valid JMX attribute?

        Show
        Benjamin Goodrich added a comment - I believe that even with a dot "." in the string, it is still a valid JMX attribute?
        Hide
        dimir added a comment - - edited

        Sure. We can fix the issue by handling an exceptions when handling attributes with dots. But I was thinking if it would be more obvious to introduce optional 3rd parameter to indicate attributes with composite data:

        jmx[oracle.jrockit.management:type=PerfCounters,jrockit.gc.pool.heap,used] (3 parameters, composite data)
        jmx[com.example:type=Hello,test.my.attribute] (2 parameters, primitive data)

        This would introduce a regression though.

        Show
        dimir added a comment - - edited Sure. We can fix the issue by handling an exceptions when handling attributes with dots. But I was thinking if it would be more obvious to introduce optional 3rd parameter to indicate attributes with composite data: jmx [oracle.jrockit.management:type=PerfCounters,jrockit.gc.pool.heap,used] (3 parameters, composite data) jmx [com.example:type=Hello,test.my.attribute] (2 parameters, primitive data) This would introduce a regression though.
        Hide
        Benjamin Goodrich added a comment -

        I think it is more clear for composite types than a dot, regardless. But any method that would resolve the issue (including exception handling) is preferable to not handling JMX attributes with dots in them.

        Show
        Benjamin Goodrich added a comment - I think it is more clear for composite types than a dot, regardless. But any method that would resolve the issue (including exception handling) is preferable to not handling JMX attributes with dots in them.
        Hide
        dimir added a comment - - edited

        We handle next situations:

        • primitive type attribute without dots (simple)
        • primitive type attribute with dots (s.i.m.p.l.e)
        • composite type attribute without dots (fruits.apple)
        • composite type attribute with dots (f.r.u.i.t.s.apple)
        • nested composite type attribute without dots (cars.audi.weight)
        • nested composite type attribute with dots (c.a.r.s.audi.weight)

        In order to identify those we process the provided attribute name as follows:

        • if the name contains dots
          • find out the attribute name by starting from left most and appending the right parts
          • if nothing left
            • it's primitive type attribute with dots
          • else
            • what's left is the key of composite data, process it recursively

        Let's see the example of processing the last option "c.a.r.s.audi.weight" where the attribute name is "c.a.r.s" which contains composite data "audi" with nested composite data "weight":

        • key with dots
        • "c", "c.a", "c.a.r" will throw the exception, handle it
        • attribute = "c.a.r.s" (this will work)
        • there is non-empty part left ("audi.weight"), get the value by recursively processing each part as composite data
        Show
        dimir added a comment - - edited We handle next situations: primitive type attribute without dots (simple) primitive type attribute with dots (s.i.m.p.l.e) composite type attribute without dots (fruits.apple) composite type attribute with dots (f.r.u.i.t.s.apple) nested composite type attribute without dots (cars.audi.weight) nested composite type attribute with dots (c.a.r.s.audi.weight) In order to identify those we process the provided attribute name as follows: if the name contains dots find out the attribute name by starting from left most and appending the right parts if nothing left it's primitive type attribute with dots else what's left is the key of composite data, process it recursively Let's see the example of processing the last option "c.a.r.s.audi.weight" where the attribute name is "c.a.r.s" which contains composite data "audi" with nested composite data "weight": key with dots "c", "c.a", "c.a.r" will throw the exception, handle it attribute = "c.a.r.s" (this will work) there is non-empty part left ("audi.weight"), get the value by recursively processing each part as composite data
        Hide
        Benjamin Goodrich added a comment -

        So recursively evaluate attribute from left to right, based on if it throws an exception...

        • Evaluate left field, if it does throw an exception, increase left (by one right field) and try again.
        • If it does not throw an exception left part of argument is stored as attribute, right part of argument is further recursively processed based on previous attributes.
        • If down to only "left" field (no more right fields) and there is an exception after processing, throw an error.

        In the end you would have a list of attributes and sub-attributes either with or without a dot or an error.

        Is that summary correct?

        Show
        Benjamin Goodrich added a comment - So recursively evaluate attribute from left to right, based on if it throws an exception... Evaluate left field, if it does throw an exception, increase left (by one right field) and try again. If it does not throw an exception left part of argument is stored as attribute , right part of argument is further recursively processed based on previous attributes. If down to only "left" field (no more right fields) and there is an exception after processing, throw an error. In the end you would have a list of attributes and sub-attributes either with or without a dot or an error. Is that summary correct?
        Hide
        dimir added a comment - - edited

        Yep, that is correct. And what's left on the right side (after we find the attribute part on the left) is the key(s) of the composite data.

        Show
        dimir added a comment - - edited Yep, that is correct. And what's left on the right side (after we find the attribute part on the left) is the key(s) of the composite data.
        Hide
        Benjamin Goodrich added a comment -

        I see this as a possibility, (attribute.1.attribute.2.key.1)

        attribute1=attribute.1
        attribute2=attribute.2
        key=key.1

        Would the algorithm be able to handle this? Could the key value also contain a dot? As long as it can find every possible attribute, then I assume whatever is "left over" on the right side would be the key value?

        Show
        Benjamin Goodrich added a comment - I see this as a possibility, (attribute.1.attribute.2.key.1) attribute1=attribute.1 attribute2=attribute.2 key=key.1 Would the algorithm be able to handle this? Could the key value also contain a dot? As long as it can find every possible attribute, then I assume whatever is "left over" on the right side would be the key value?
        Hide
        dimir added a comment -

        Please see the update description of the flow above. I hope it's more clear this way. And I have an example application with exactly the attributes like mentioned there.

        Show
        dimir added a comment - Please see the update description of the flow above. I hope it's more clear this way. And I have an example application with exactly the attributes like mentioned there.
        Hide
        Benjamin Goodrich added a comment -

        What if there was a value 'c.a.r.s.audi.r8.weight.kg' where "c.a.r.s" is the attribute name, with composite data "audi.r8", with nested data "weight.kg"? Is that a possible valid combination?

        Show
        Benjamin Goodrich added a comment - What if there was a value 'c.a.r.s.audi.r8.weight.kg' where "c.a.r.s" is the attribute name, with composite data "audi.r8", with nested data "weight.kg"? Is that a possible valid combination?
        Hide
        dimir added a comment - - edited

        Good point. Actually this is quite a valid combination. The field name of the composite data can be of any type, including String, which, of course, can contain dots. Such a situation was not handled before. But how to handle it?

        I can get all the keys of composite data and then start guessing which part of the name did the user want to use. This would not help for next case though (attribute = "cars"):

        • cars:
          • audi.r8 = "The ultimate car"
          • audi:
            • r8:
              • weight = 123

        User provided attribute name "cars.audi.r8" would bring in the confusion. So I guess we have to drop the support for field names with dots.

        Show
        dimir added a comment - - edited Good point. Actually this is quite a valid combination. The field name of the composite data can be of any type, including String, which, of course, can contain dots. Such a situation was not handled before. But how to handle it? I can get all the keys of composite data and then start guessing which part of the name did the user want to use. This would not help for next case though (attribute = "cars"): cars: audi.r8 = "The ultimate car" audi: r8: weight = 123 User provided attribute name "cars.audi.r8" would bring in the confusion. So I guess we have to drop the support for field names with dots.
        Hide
        dimir added a comment -

        We are currently discussing how to handle such a situation. One way is escape the dots that are not acting as delimiters. Introducing third parameter will not probably suffice, as in case of nested composite data we would need a separate parameter for each one. In the latter case it would be not possible to add a parameter at a later time without introducing a regression.

        Show
        dimir added a comment - We are currently discussing how to handle such a situation. One way is escape the dots that are not acting as delimiters. Introducing third parameter will not probably suffice, as in case of nested composite data we would need a separate parameter for each one. In the latter case it would be not possible to add a parameter at a later time without introducing a regression.
        Hide
        Benjamin Goodrich added a comment -

        Actually, Vladimir, that sounds like a much simpler and better solution. Nice.

        Show
        Benjamin Goodrich added a comment - Actually, Vladimir, that sounds like a much simpler and better solution. Nice.
        Hide
        dimir added a comment -

        You mean a separate parameter for each field of composite data? Like:

        jmx[objectName:type=Cars, audi, weight]
        jmx[objectName:type=Cars, the.trucks, volvo, model.x, weight]

        ? Yes, this is clean solution but the problem is that if we decide to add an additional parameter at a later time it would introduce a regression, as we would need to add it before the variable-length list.

        Show
        dimir added a comment - You mean a separate parameter for each field of composite data? Like: jmx [objectName:type=Cars, audi, weight] jmx [objectName:type=Cars, the.trucks, volvo, model.x, weight] ? Yes, this is clean solution but the problem is that if we decide to add an additional parameter at a later time it would introduce a regression, as we would need to add it before the variable-length list.
        Hide
        Benjamin Goodrich added a comment -

        Actually, I was commenting that escaping the dots was a simpler and better solution.

        jmx[ObjectName:type=Cars\.model,audi\.r8.weight\.kg]

        You are correct that three fields could not definitively work, for instance:(java.lang:type=GarbageCollector,name=ConcurrentMarkSweep,LastGcInfo,memoryUsageAfterGc,CMS Old Gen,committed)

        Show
        Benjamin Goodrich added a comment - Actually, I was commenting that escaping the dots was a simpler and better solution. jmx [ObjectName:type=Cars\.model,audi\.r8.weight\.kg] You are correct that three fields could not definitively work, for instance:(java.lang:type=GarbageCollector,name=ConcurrentMarkSweep,LastGcInfo,memoryUsageAfterGc,CMS Old Gen,committed)
        Hide
        dimir added a comment - - edited

        Right. So this is how it was done. We added a support for escaping the dot that is not a separator by using a backslash. In case of a backslash symbol in the attribute name user would need to escape it with another backslash. The development branch will be ready a bit later today. This will be fixed for both 2.0 and trunk. Deploying the fix will require recompiling and restarting Zabbix java gateway. No regression introduced.

        Show
        dimir added a comment - - edited Right. So this is how it was done. We added a support for escaping the dot that is not a separator by using a backslash. In case of a backslash symbol in the attribute name user would need to escape it with another backslash. The development branch will be ready a bit later today. This will be fixed for both 2.0 and trunk. Deploying the fix will require recompiling and restarting Zabbix java gateway. No regression introduced.
        Hide
        Benjamin Goodrich added a comment -

        That's great, thanks for finding a solution for this.

        Show
        Benjamin Goodrich added a comment - That's great, thanks for finding a solution for this.
        Hide
        dimir added a comment -

        Fixed in development branch: svn://svn.zabbix.com/branches/dev/ZBX-4663

        Show
        dimir added a comment - Fixed in development branch: svn://svn.zabbix.com/branches/dev/ZBX-4663
        Hide
        Alexander Vladishev added a comment - - edited

        (1) Please review my changes in r30236.

        <dimir> Thanks, looks great! CLOSED

        Show
        Alexander Vladishev added a comment - - edited (1) Please review my changes in r30236. <dimir> Thanks, looks great! CLOSED
        Hide
        Alexander Vladishev added a comment - - edited

        (2) attributes or keys with a composite data doesn't work with trailing backslashes.

        Key to reproduce it: jmx[com.example:type=Hello,backslashend\\.a]
        where: "backslashend\" - attribute name
               "a" - key
        

        <dimir> RESOLVED in r30243
        <Sasha> CLOSED

        Show
        Alexander Vladishev added a comment - - edited (2) attributes or keys with a composite data doesn't work with trailing backslashes. Key to reproduce it: jmx[com.example:type=Hello,backslashend\\.a] where: "backslashend\" - attribute name "a" - key <dimir> RESOLVED in r30243 <Sasha> CLOSED
        Hide
        dimir added a comment -

        Merging this to 2.0 and trunk will have to wait till 2.0 is unfrozen.

        Show
        dimir added a comment - Merging this to 2.0 and trunk will have to wait till 2.0 is unfrozen.
        Show
        dimir added a comment - - edited Documentation updated: http://www.zabbix.com/documentation/2.0/manual/config/items/itemtypes/jmx_monitoring http://www.zabbix.com/documentation/2.0/manual/introduction/whatsnew204
        Hide
        Alexander Vladishev added a comment -

        Fixed in pre-2.0.4 r30620 and pre-2.1.0 (trunk) r30621.

        Show
        Alexander Vladishev added a comment - Fixed in pre-2.0.4 r30620 and pre-2.1.0 (trunk) r30621.
        Hide
        David added a comment -

        Is this actually working?

        Redacted logs from our 2.0.6 installation:
        17801:20130509:213853.364 item [hostname:jmx["Metrics:context=apiCounters",ExceptionCounts.com\.DeviceDisconnectedException"]] became not supported: Argument key="com\" is not an existing item name for this CompositeData instance.

        Item key: jmx["Metrics:context=apiCounters",ExceptionCounts.com\.DeviceDisconnectedException"]

        It seems to cut off after any dot, no matter what escaping I try. ExceptionCounts is a composite item, the attribute name is "com.DeviceDisconnectedException".

        Show
        David added a comment - Is this actually working? Redacted logs from our 2.0.6 installation: 17801:20130509:213853.364 item [hostname:jmx ["Metrics:context=apiCounters",ExceptionCounts.com\.DeviceDisconnectedException"] ] became not supported: Argument key="com\" is not an existing item name for this CompositeData instance. Item key: jmx ["Metrics:context=apiCounters",ExceptionCounts.com\.DeviceDisconnectedException"] It seems to cut off after any dot, no matter what escaping I try. ExceptionCounts is a composite item, the attribute name is "com.DeviceDisconnectedException".
        Hide
        David added a comment -

        Apologies, our Java Gateway was still running 2.0.3. This is working for us now. Please disregard the above comment.

        Show
        David added a comment - Apologies, our Java Gateway was still running 2.0.3. This is working for us now. Please disregard the above comment.

          People

          • Assignee:
            Unassigned
            Reporter:
            Vladimir K
          • Votes:
            1 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved: