[ZBX-10577] Custom LLD permanently disabled after single unexpected response Created: 2016 Mar 24 Updated: 2017 May 30 Resolved: 2016 Apr 01 |
|
Status: | Closed |
Project: | ZABBIX BUGS AND ISSUES |
Component/s: | Server (S) |
Affects Version/s: | 3.0.0, 3.0.1 |
Fix Version/s: | None |
Type: | Incident report | Priority: | Major |
Reporter: | dingham | Assignee: | Unassigned |
Resolution: | Won't fix | Votes: | 0 |
Labels: | json, lld | ||
Remaining Estimate: | Not Specified | ||
Time Spent: | Not Specified | ||
Original Estimate: | Not Specified | ||
Environment: |
Ubuntu 14.04.4 LTS |
Attachments: |
![]() ![]() |
||||||||
Issue Links: |
|
Description |
I created a custom low-level discovery rule tied with UserParameter & a script which outputs JSON. It failed with "Value should be a JSON object" the first time round due to a trailing comma, however despite the issue now being fixed the LLD rule is now hung in a disabled state. The JSON returned by zabbix_sender & reported in the log ("get value from agent result:") validates on jsonlint.com, so the issue is with Zabbix. Disabling & re-enabling the rule, & restarting zabbix server makes no difference. LogLevel 5 output: 7849:20160324:143255.501 get value from agent result: '{"data":[{"{#DSET}":"server0","{#DSRV}":"mysql"},{"{#DSET}":"server0","{#DSRV}":"rsync"},{"{#DSET}":"server1","{#DSRV}":"mysql"},{"{#DSET}":"server2","{#DSRV}":"mysql"}]}' 7849:20160324:143255.501 End of get_value_agent():SUCCEED 7849:20160324:143255.501 End of get_value():SUCCEED 7849:20160324:143255.501 In activate_host() hostid:10159 itemid:29425 type:0 7849:20160324:143255.501 End of activate_host() 7849:20160324:143255.501 In lld_process_discovery_rule() itemid:29425 7849:20160324:143255.501 query [txnlev:0] [select hostid,key_,state,evaltype,formula,error,lifetime from items where itemid=29425] 7849:20160324:143255.501 In substitute_simple_macros() data:'30' 7849:20160324:143255.502 query [txnlev:0] [select item_conditionid,macro,value from item_condition where itemid=29425] 7849:20160324:143255.502 In lld_rows_get() 7849:20160324:143255.502 End of lld_rows_get():FAIL 7849:20160324:143255.502 End of lld_process_discovery_rule() 7849:20160324:143255.502 End of get_values():1 7849:20160324:143255.502 __zbx_zbx_setproctitle() title:'poller #2 [got 10 values in 0.099303 sec, idle 1 sec]' |
Comments |
Comment by richlv [ 2016 Mar 25 ] |
what's shown in the lld rule list ? |
Comment by dingham [ 2016 Mar 28 ] |
Actually, disabled is possibly the wrong word. The Status is "Enabled", however there's a red cross in the Info column, and when you hover over it it gives the error "Value should be a JSON object." |
Comment by dingham [ 2016 Mar 28 ] |
Log after LLD rule deleted & recreated. The invalid JSON warning no longer appears in the Info column, however the items still aren't being added & the log still ends with: In lld_rows_get() |
Comment by richlv [ 2016 Mar 28 ] |
it looks like a support case to me - no indication of a bug. i'd suggest discussing it in the available support channels (IRC etc) : http://zabbix.org/wiki/Getting_help |
Comment by dingham [ 2016 Mar 28 ] |
I spent a couple of hours with the guys on IRC, and it was finally decided a bug should be raised. |
Comment by richlv [ 2016 Mar 29 ] |
for the record, we continued this a bit more on IRC, some further testing could provide more info |
Comment by Aleksandrs Saveljevs [ 2016 Mar 29 ] |
Is there any error message in the "Info" column now or is it green? |
Comment by richlv [ 2016 Mar 29 ] |
no matter how this particular case turns out, the following is a quite terse debug output : 7849:20160324:143255.502 In lld_rows_get() 7849:20160324:143255.502 End of lld_rows_get():FAIL can we output something more useful here maybe ? |
Comment by dingham [ 2016 Mar 29 ] |
So, since I wrote that the warning disappeared after I deleted & re-created the rule, it's now come back. I created a separate test rule today, and the items were added as expected. *** Discovery rule *** Name: Test LLD Type: Zabbix agent Key: test.lld zabbix_agentd.conf: UserParameter=test.lld[*],/mnt/pool0/zabbix/testlld.sh $1 ------ testlld.sh ------- #!/bin/bash if [[ -z $1 ]]; then echo '{"data":[{"{#MACTEST}":"item1","{#MACTEST2}":"abc"},{"{#MACTEST}":"item1","{#MACTEST2}":"def"},{"{#MACTEST}":"item2","{#MACTEST2}":"abc"},{"{#MACTEST}":"item3","{#MACTEST2}":"abc"},{"{#MACTEST}":"item4","{#MACTEST2}":"abc"},{"{#MACTEST}":"item4","{#MACTEST2}":"def"},{"{#MACTEST}":"item5","{#MACTEST2}":"abc"},{"{#MACTEST}":"item6","{#MACTEST2}":"abc"},{"{#MACTEST}":"item6","{#MACTEST2}":"def"},{"{#MACTEST}":"item7","{#MACTEST2}":"abc"},{"{#MACTEST}":"item8","{#MACTEST2}":"abc"},{"{#MACTEST}":"item9","{#MACTEST2}":"def"},{"{#MACTEST}":"item10","{#MACTEST2}":"abc"},{"{#MACTEST}":"item11","{#MACTEST2}":"def"},{"{#MACTEST}":"item12","{#MACTEST2}":"abc"},{"{#MACTEST}":"item13","{#MACTEST2}":"abc"},{"{#MACTEST}":"item14","{#MACTEST2}":"abc"},{"{#MACTEST}":"item15","{#MACTEST2}":"abc"}]}' else echo "$1" fi ---------------------- *** Item prototype *** Name: Macro Test: {#MACTEST} {#MACTEST2} Type: Zabbix agent Key: test.lld[{#MACTEST},{#MACTEST2}] Type of information: Character Going back to my original rule & JSON, I changed the discovery rule to "Zabbix trapper" & used zabbix_sender to send in the JSON (as returned by zabbix_get) to the Zabbix server. The moment I did that, all of the items appeared & started generating data as expected (seen in "Latest data"). root@dingham:~# zabbix_sender -z monitor0 -s "backup0" -k backup.descobrir -o '{"data":[{"{#DSET}":"mdl_InglesInfoEdu","{#DSRV}":"mysql"},{"{#DSET}":"Evento2016","{#DSRV}":"mysql"},{"{#DSET}":"serve-NP","{#DSRV}":"mysql"},{"{#DSET}":"mdl_japao","{#DSRV}":"mysql"},{"{#DSET}":"GDE_PEDA2012","{#DSRV}":"mysql"},{"{#DSET}":"srv0","{#DSRV}":"mysql"},{"{#DSET}":"sites0","{#DSRV}":"mysql"},{"{#DSET}":"mdl_Ped0709","{#DSRV}":"mysql"},{"{#DSET}":"ESP_EduCamp","{#DSRV}":"mysql"},{"{#DSET}":"srv2","{#DSRV}":"mysql"},{"{#DSET}":"inter0","{#DSRV}":"mysql"},{"{#DSET}":"srv-finalizados","{#DSRV}":"mysql"},{"{#DSET}":"mdl_PMD","{#DSRV}":"mysql"},{"{#DSET}":"mdl_AdmPub","{#DSRV}":"mysql"},{"{#DSET}":"sites1","{#DSRV}":"mysql"},{"{#DSET}":"srv1","{#DSRV}":"mysql"},{"{#DSET}":"Srv-GCDH","{#DSRV}":"mysql"},{"{#DSET}":"mdl_Eja","{#DSRV}":"mysql"}]}' info from server: "processed: 1; failed: 0; total: 1; seconds spent: 0.134209" sent: 1; skipped: 0; total: 1 I changed the rule type back to "Zabbix agent" & the warning in the Info column has come back, "Value should be a JSON object". |
Comment by richlv [ 2016 Mar 30 ] |
that looks very much like something is wrong with the json indeed. try :
it is likely to fail again. carefully check debug output & tcpdump output. does the json still look ok ? |
Comment by dingham [ 2016 Apr 01 ] |
OK, all resolved, and it turns out to be a "PEBKAC" issue. I came across this very similar issue raised in 2014 & found the key in the comments: In short, instead of adding an extra condition to the loop in my script & dropping the trailing comma there, I cheated & closed the JSON prefixed with a backslash escaped backspace & sent the output through echo. /me goes back to scriptkiddie school. |