CentOS, jrockit

I would be really glad for any workaround suggestion.

We have some custom mbeans with dots in their names. Seeing the following error in the server log

23768:20120828:091903.317 Item [server-1_application:jmx["app.devops.buildinfo:name=secure","app.name"]] became not supported: No such property: app

(workaround is to escape () the dot "." character in the string)

Spoke too soon, looked like it worked when I made the change, but still reporting an error in the log and later turned back to "unsupported"

23767:20120828:102903.954 Item [server-1_application:jmx["app.devops.buildinfo:name=secure","app\.name"]] became not supported: No such property: app\

Escaping didn't work.
Went to code and disabled dot related logic inside.

I think "." dot is being used for attributes that return a composite data value. I wonder if an optional third field could be used instead in the JMX definition for composite data types.

In case of "jrockit.gc.pool.heap.used" the returned value isn't composite, isn't it just "java.lang.Long"?

Any simple example to reproduce it? I do not want to setup Oracle or whatever it takes to use that MBeans and I couldn't find an attribute with dots in MXBean:

http://docs.oracle.com/javase/6/docs/api/java/lang/management/MemoryMXBean.html

I'm using OpenJDK by the way.

That's true. Currently Zabbix Java gateway works so that if attribute name contains dot it treats it as a separator between an attribute name with composite data and the field name of that data. We would need to add a way to destinguish an attribute containing composite data from an attribute with simple data that just contains dots.

I believe that even with a dot "." in the string, it is still a valid JMX attribute?

Sure. We can fix the issue by handling an exceptions when handling attributes with dots. But I was thinking if it would be more obvious to introduce optional 3rd parameter to indicate attributes with composite data:

jmx[oracle.jrockit.management:type=PerfCounters,jrockit.gc.pool.heap,used] (3 parameters, composite data)
jmx[com.example:type=Hello,test.my.attribute] (2 parameters, primitive data)

This would introduce a regression though.

I think it is more clear for composite types than a dot, regardless. But any method that would resolve the issue (including exception handling) is preferable to not handling JMX attributes with dots in them.

We handle next situations:

• primitive type attribute without dots (simple)
• primitive type attribute with dots (s.i.m.p.l.e)
• composite type attribute without dots (fruits.apple)
• composite type attribute with dots (f.r.u.i.t.s.apple)
• nested composite type attribute without dots (cars.audi.weight)
• nested composite type attribute with dots (c.a.r.s.audi.weight)

In order to identify those we process the provided attribute name as follows:

• if the name contains dots
  • find out the attribute name by starting from left most and appending the right parts
  • if nothing left
    • it's primitive type attribute with dots
  • else
    • what's left is the key of composite data, process it recursively

Let's see the example of processing the last option "c.a.r.s.audi.weight" where the attribute name is "c.a.r.s" which contains composite data "audi" with nested composite data "weight":

• key with dots
• "c", "c.a", "c.a.r" will throw the exception, handle it
• attribute = "c.a.r.s" (this will work)
• there is non-empty part left ("audi.weight"), get the value by recursively processing each part as composite data

So recursively evaluate attribute from left to right, based on if it throws an exception...

• Evaluate left field, if it does throw an exception, increase left (by one right field) and try again.
• If it does not throw an exception left part of argument is stored as attribute, right part of argument is further recursively processed based on previous attributes.
• If down to only "left" field (no more right fields) and there is an exception after processing, throw an error.

In the end you would have a list of attributes and sub-attributes either with or without a dot or an error.

Is that summary correct?

Yep, that is correct. And what's left on the right side (after we find the attribute part on the left) is the key(s) of the composite data.

I see this as a possibility, (attribute.1.attribute.2.key.1)

attribute1=attribute.1
attribute2=attribute.2
key=key.1

Would the algorithm be able to handle this? Could the key value also contain a dot? As long as it can find every possible attribute, then I assume whatever is "left over" on the right side would be the key value?

Please see the update description of the flow above. I hope it's more clear this way. And I have an example application with exactly the attributes like mentioned there.

What if there was a value 'c.a.r.s.audi.r8.weight.kg' where "c.a.r.s" is the attribute name, with composite data "audi.r8", with nested data "weight.kg"? Is that a possible valid combination?

Good point. Actually this is quite a valid combination. The field name of the composite data can be of any type, including String, which, of course, can contain dots. Such a situation was not handled before. But how to handle it?

I can get all the keys of composite data and then start guessing which part of the name did the user want to use. This would not help for next case though (attribute = "cars"):

• cars:
  • audi.r8 = "The ultimate car"
  • audi:
    • r8:
      • weight = 123

User provided attribute name "cars.audi.r8" would bring in the confusion. So I guess we have to drop the support for field names with dots.

We are currently discussing how to handle such a situation. One way is escape the dots that are not acting as delimiters. Introducing third parameter will not probably suffice, as in case of nested composite data we would need a separate parameter for each one. In the latter case it would be not possible to add a parameter at a later time without introducing a regression.

Actually, Vladimir, that sounds like a much simpler and better solution. Nice.

You mean a separate parameter for each field of composite data? Like:

jmx[objectName:type=Cars, audi, weight]
jmx[objectName:type=Cars, the.trucks, volvo, model.x, weight]

? Yes, this is clean solution but the problem is that if we decide to add an additional parameter at a later time it would introduce a regression, as we would need to add it before the variable-length list.

Actually, I was commenting that escaping the dots was a simpler and better solution.

jmx[ObjectName:type=Cars\.model,audi\.r8.weight\.kg]

You are correct that three fields could not definitively work, for instance:(java.lang:type=GarbageCollector,name=ConcurrentMarkSweep,LastGcInfo,memoryUsageAfterGc,CMS Old Gen,committed)

Right. So this is how it was done. We added a support for escaping the dot that is not a separator by using a backslash. In case of a backslash symbol in the attribute name user would need to escape it with another backslash. The development branch will be ready a bit later today. This will be fixed for both 2.0 and trunk. Deploying the fix will require recompiling and restarting Zabbix java gateway. No regression introduced.

That's great, thanks for finding a solution for this.

Fixed in development branch: svn://svn.zabbix.com/branches/dev/ZBX-4663

(1) Please review my changes in r30236.

<dimir> Thanks, looks great! CLOSED

(2) attributes or keys with a composite data doesn't work with trailing backslashes.

Key to reproduce it: jmx[com.example:type=Hello,backslashend\\.a]
where: "backslashend\" - attribute name
       "a" - key

<dimir> RESOLVED in r30243
<Sasha> CLOSED

Merging this to 2.0 and trunk will have to wait till 2.0 is unfrozen.

Documentation updated:

• http://www.zabbix.com/documentation/2.0/manual/config/items/itemtypes/jmx_monitoring
• http://www.zabbix.com/documentation/2.0/manual/introduction/whatsnew204

Fixed in pre-2.0.4 r30620 and pre-2.1.0 (trunk) r30621.

Is this actually working?

Redacted logs from our 2.0.6 installation:
17801:20130509:213853.364 item [hostname:jmx["Metrics:context=apiCounters",ExceptionCounts.com\.DeviceDisconnectedException"]] became not supported: Argument key="com\" is not an existing item name for this CompositeData instance.

Item key: jmx["Metrics:context=apiCounters",ExceptionCounts.com\.DeviceDisconnectedException"]

It seems to cut off after any dot, no matter what escaping I try. ExceptionCounts is a composite item, the attribute name is "com.DeviceDisconnectedException".

Apologies, our Java Gateway was still running 2.0.3. This is working for us now. Please disregard the above comment.