[ZBX-10577] Custom LLD permanently disabled after single unexpected response Created: 2016 Mar 24  Updated: 2017 May 30  Resolved: 2016 Apr 01

Status: Closed
Project: ZABBIX BUGS AND ISSUES
Component/s: Server (S)
Affects Version/s: 3.0.0, 3.0.1
Fix Version/s: None

Type: Incident report Priority: Major
Reporter: dingham Assignee: Unassigned
Resolution: Won't fix Votes: 0
Labels: json, lld
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

Ubuntu 14.04.4 LTS
mysql Ver 14.14 Distrib 5.5.47


Attachments: PNG File LLD config.png     Text File Zbx-LLD.txt    
Issue Links:
Duplicate
duplicates ZBX-7854 Discovery item returns "Value should ... Closed

 Description   

I created a custom low-level discovery rule tied with UserParameter & a script which outputs JSON.

It failed with "Value should be a JSON object" the first time round due to a trailing comma, however despite the issue now being fixed the LLD rule is now hung in a disabled state.

The JSON returned by zabbix_sender & reported in the log ("get value from agent result:") validates on jsonlint.com, so the issue is with Zabbix.

Disabling & re-enabling the rule, & restarting zabbix server makes no difference.

LogLevel 5 output:

7849:20160324:143255.501 get value from agent result: '{"data":[{"{#DSET}":"server0","{#DSRV}":"mysql"},{"{#DSET}":"server0","{#DSRV}":"rsync"},{"{#DSET}":"server1","{#DSRV}":"mysql"},{"{#DSET}":"server2","{#DSRV}":"mysql"}]}'
7849:20160324:143255.501 End of get_value_agent():SUCCEED
7849:20160324:143255.501 End of get_value():SUCCEED
7849:20160324:143255.501 In activate_host() hostid:10159 itemid:29425 type:0
7849:20160324:143255.501 End of activate_host()
7849:20160324:143255.501 In lld_process_discovery_rule() itemid:29425
7849:20160324:143255.501 query [txnlev:0] [select hostid,key_,state,evaltype,formula,error,lifetime from items where itemid=29425]
7849:20160324:143255.501 In substitute_simple_macros() data:'30'
7849:20160324:143255.502 query [txnlev:0] [select item_conditionid,macro,value from item_condition where itemid=29425]
7849:20160324:143255.502 In lld_rows_get()
7849:20160324:143255.502 End of lld_rows_get():FAIL
7849:20160324:143255.502 End of lld_process_discovery_rule()
7849:20160324:143255.502 End of get_values():1
7849:20160324:143255.502 __zbx_zbx_setproctitle() title:'poller #2 [got 10 values in 0.099303 sec, idle 1 sec]'


 Comments   
Comment by richlv [ 2016 Mar 25 ]

what's shown in the lld rule list ?
how do you determine it's "hung in a disabled state" ?

Comment by dingham [ 2016 Mar 28 ]

Actually, disabled is possibly the wrong word. The Status is "Enabled", however there's a red cross in the Info column, and when you hover over it it gives the error "Value should be a JSON object."
I can see Zabbix executing the rule in the logs, as above, but they all end with that "End of lld_rows_get():FAIL" & no items or triggers are added from the prototypes (the configurations of which are in the screenshots above).

Comment by dingham [ 2016 Mar 28 ]

Log after LLD rule deleted & recreated. The invalid JSON warning no longer appears in the Info column, however the items still aren't being added & the log still ends with:

In lld_rows_get()
End of lld_rows_get():FAIL

Comment by richlv [ 2016 Mar 28 ]

it looks like a support case to me - no indication of a bug. i'd suggest discussing it in the available support channels (IRC etc) : http://zabbix.org/wiki/Getting_help

Comment by dingham [ 2016 Mar 28 ]

I spent a couple of hours with the guys on IRC, and it was finally decided a bug should be raised.

Comment by richlv [ 2016 Mar 29 ]

for the record, we continued this a bit more on IRC, some further testing could provide more info

Comment by Aleksandrs Saveljevs [ 2016 Mar 29 ]

The invalid JSON warning no longer appears in the Info column, however the items still aren't being added...

Is there any error message in the "Info" column now or is it green?

Comment by richlv [ 2016 Mar 29 ]

no matter how this particular case turns out, the following is a quite terse debug output :

7849:20160324:143255.502 In lld_rows_get()
7849:20160324:143255.502 End of lld_rows_get():FAIL

can we output something more useful here maybe ?

Comment by dingham [ 2016 Mar 29 ]

So, since I wrote that the warning disappeared after I deleted & re-created the rule, it's now come back.
Status: Enabled (green), Info: "Value should be a JSON object."

I created a separate test rule today, and the items were added as expected.

*** Discovery rule ***
Name: Test LLD
Type: Zabbix agent
Key: test.lld
zabbix_agentd.conf: UserParameter=test.lld[*],/mnt/pool0/zabbix/testlld.sh $1

------ testlld.sh -------
#!/bin/bash
if [[ -z $1 ]]; then
        echo '{"data":[{"{#MACTEST}":"item1","{#MACTEST2}":"abc"},{"{#MACTEST}":"item1","{#MACTEST2}":"def"},{"{#MACTEST}":"item2","{#MACTEST2}":"abc"},{"{#MACTEST}":"item3","{#MACTEST2}":"abc"},{"{#MACTEST}":"item4","{#MACTEST2}":"abc"},{"{#MACTEST}":"item4","{#MACTEST2}":"def"},{"{#MACTEST}":"item5","{#MACTEST2}":"abc"},{"{#MACTEST}":"item6","{#MACTEST2}":"abc"},{"{#MACTEST}":"item6","{#MACTEST2}":"def"},{"{#MACTEST}":"item7","{#MACTEST2}":"abc"},{"{#MACTEST}":"item8","{#MACTEST2}":"abc"},{"{#MACTEST}":"item9","{#MACTEST2}":"def"},{"{#MACTEST}":"item10","{#MACTEST2}":"abc"},{"{#MACTEST}":"item11","{#MACTEST2}":"def"},{"{#MACTEST}":"item12","{#MACTEST2}":"abc"},{"{#MACTEST}":"item13","{#MACTEST2}":"abc"},{"{#MACTEST}":"item14","{#MACTEST2}":"abc"},{"{#MACTEST}":"item15","{#MACTEST2}":"abc"}]}'
else
        echo "$1"
fi
----------------------

*** Item prototype ***
Name: Macro Test: {#MACTEST} {#MACTEST2}
Type: Zabbix agent
Key: test.lld[{#MACTEST},{#MACTEST2}]
Type of information: Character

Going back to my original rule & JSON, I changed the discovery rule to "Zabbix trapper" & used zabbix_sender to send in the JSON (as returned by zabbix_get) to the Zabbix server. The moment I did that, all of the items appeared & started generating data as expected (seen in "Latest data").

root@dingham:~# zabbix_sender -z monitor0 -s "backup0" -k backup.descobrir -o '{"data":[{"{#DSET}":"mdl_InglesInfoEdu","{#DSRV}":"mysql"},{"{#DSET}":"Evento2016","{#DSRV}":"mysql"},{"{#DSET}":"serve-NP","{#DSRV}":"mysql"},{"{#DSET}":"mdl_japao","{#DSRV}":"mysql"},{"{#DSET}":"GDE_PEDA2012","{#DSRV}":"mysql"},{"{#DSET}":"srv0","{#DSRV}":"mysql"},{"{#DSET}":"sites0","{#DSRV}":"mysql"},{"{#DSET}":"mdl_Ped0709","{#DSRV}":"mysql"},{"{#DSET}":"ESP_EduCamp","{#DSRV}":"mysql"},{"{#DSET}":"srv2","{#DSRV}":"mysql"},{"{#DSET}":"inter0","{#DSRV}":"mysql"},{"{#DSET}":"srv-finalizados","{#DSRV}":"mysql"},{"{#DSET}":"mdl_PMD","{#DSRV}":"mysql"},{"{#DSET}":"mdl_AdmPub","{#DSRV}":"mysql"},{"{#DSET}":"sites1","{#DSRV}":"mysql"},{"{#DSET}":"srv1","{#DSRV}":"mysql"},{"{#DSET}":"Srv-GCDH","{#DSRV}":"mysql"},{"{#DSET}":"mdl_Eja","{#DSRV}":"mysql"}]}'
info from server: "processed: 1; failed: 0; total: 1; seconds spent: 0.134209"
sent: 1; skipped: 0; total: 1

I changed the rule type back to "Zabbix agent" & the warning in the Info column has come back, "Value should be a JSON object".

Comment by richlv [ 2016 Mar 30 ]

that looks very much like something is wrong with the json indeed. try :

  • set it back to trapper, send in value that makes it work
  • bump loglevel for trappers to 4, tcpdump zabbix port on the agent (to have less extra traffic - i doubt network would mangle it)
  • change it back to agent

it is likely to fail again. carefully check debug output & tcpdump output. does the json still look ok ?

Comment by dingham [ 2016 Apr 01 ]

OK, all resolved, and it turns out to be a "PEBKAC" issue.

I came across this very similar issue raised in 2014 & found the key in the comments:
https://support.zabbix.com/browse/ZBX-7854?focusedCommentId=101692&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-101692

In short, instead of adding an extra condition to the loop in my script & dropping the trailing comma there, I cheated & closed the JSON prefixed with a backslash escaped backspace & sent the output through echo.
This all appeared fine when receiving the output to the shell (using zabbix_get & executing the script manually), and it also didn't show anything unusual when parsing the logs with grep & less. But opening the log in vi, it was clear that the unescaped ",\b" was being included in the stream.

/me goes back to scriptkiddie school.

Generated at Fri Apr 04 14:50:56 EEST 2025 using Jira 9.12.4#9120004-sha1:625303b708afdb767e17cb2838290c41888e9ff0.