[ZBXNEXT-4502] Support of 'advanced' JSON Path expressions Created: 2018 Apr 16  Updated: 2024 Apr 10  Resolved: 2019 Jul 22

Status: Closed
Project: ZABBIX FEATURE REQUESTS
Component/s: Proxy (P), Server (S)
Affects Version/s: None
Fix Version/s: 4.0.11rc1, 4.2.5rc1, 4.4.0alpha1, 4.4 (plan)

Type: New Feature Request Priority: Major
Reporter: Vitaly Zhuravlev Assignee: Andris Zeila
Resolution: Fixed Votes: 45
Labels: json, preprocessing
Σ Remaining Estimate: Not Specified Remaining Estimate: Not Specified
Σ Time Spent: Not Specified Time Spent: Not Specified
Σ Original Estimate: Not Specified Original Estimate: Not Specified

Attachments: PNG File image-2019-06-04-15-08-35-646.png     File jsonpath_sum.diff    
Issue Links:
Causes
Sub-task
part of ZBX-16052 Something is wrong with JSONpath prep... Closed
Sub-Tasks:
Key
Summary
Type
Status
Assignee
ZBXNEXT-5243 Front end validation for JSON Path Change Request (Sub-task) Closed Aleksandrs Larionovs  
Team: Team A
Sprint: Sprint 52 (May 2019), Sprint 53 (Jun 2019), Sprint 54 (Jul 2019)
Story Points: 5

 Description   

Together with ZBXNEXT-4200 and ZBXNEXT-4163, support of 'advanced' JSON Path expressions is required to select specific JSON object's attribute from the item.

For example (mentioned in ZBXNEXT-4163)
Is there any possibility of using LLD MACROS in item preprocessing.
For example process JSON based on LLD result:

like:
1.

[JSON Path]
 
$.data[?(@.CHECKNAME == '{#CHECKNAME}' )].value 

2. Another example from openstack cinder:
JSON response:

{
    "pools": [
        {
            "capabilities": {
                "QoS_support": false,
                "allocated_capacity_gb": 3,
                "driver_version": "3.0.0",
                "filter_function": null,
                "free_capacity_gb": 7.01,
                "goodness_function": null,
                "location_info": "LVMVolumeDriver:devstack-ubuntu16:stack-volumes-lvmdriver-1:default:0",
                "max_over_subscription_ratio": 1.0,
                "multiattach": true,
                "pool_name": "lvmdriver-1",
                "provisioned_capacity_gb": 3.0,
                "reserved_percentage": 0,
                "storage_protocol": "iSCSI",
                "thick_provisioning_support": true,
                "thin_provisioning_support": false,
                "timestamp": "2017-10-17T13:14:17.014588",
                "total_capacity_gb": 10.01,
                "total_volumes": 2,
                "vendor_name": "Open Source",
                "volume_backend_name": "lvmdriver-1"
            },
            "name": "devstack-ubuntu16@lvmdriver-1#lvmdriver-1"
        },
        {
            "capabilities": {
                "QoS_support": false,
                "allocated_capacity_gb": 0,
                "driver_version": "3.0.0",
                "filter_function": null,
                "free_capacity_gb": 8.01,
                "goodness_function": null,
                "location_info": "LVMVolumeDriver:devstack-ubuntu16:stack-volumes-lvmdriver-1:default:0",
                "max_over_subscription_ratio": 1.0,
                "multiattach": true,
                "pool_name": "lvmdriver-1",
                "provisioned_capacity_gb": 3.0,
                "reserved_percentage": 0,
                "storage_protocol": "iSCSI",
                "thick_provisioning_support": true,
                "thin_provisioning_support": false,
                "timestamp": "2017-10-17T13:14:21.222133",
                "total_capacity_gb": 10.01,
                "total_volumes": 2,
                "vendor_name": "Open Source",
                "volume_backend_name": "lvmdriver-1"
            },
            "name": "devstack-ubuntu16@custom-type#lvmdriver-1"
        }
    ]
}

In order to get free space from specific pool you would need JSON path like this to work in preprocessing:

$.pools[?(@.name == "devstack-ubuntu16@custom-type#lvmdriver-1")].capabilities.free_capacity_gb

And It looks like that this is not going to work at the moment.

More expression examples:
http://goessner.net/articles/JsonPath/



 Comments   
Comment by Vitaly Zhuravlev [ 2018 Aug 09 ]

Once we can turn CSV into JSON, we may also use this to select specific item values from CSV.

Comment by Leonardo Savoini [ 2018 Dec 14 ]

Please, I need the posibility to filter when I'm using db.odbc.discovery

A simple example:

{
    "data": [
        {
            "{#HOST}": "host1",
            "{#PORT}": "in",
            "{#COUNT}": "5"
        },
        {
            "{#HOST}": "host2",
            "{#PORT}": "in",
            "{#COUNT}": "12"
        },
        {
            "{#HOST}": "host3",
            "{#PORT}": "in",
            "{#COUNT}": "3"
        },
        {
            "{#HOST}": "host1",
            "{#PORT}": "out",
            "{#COUNT}": "7"
        },
        {
            "{#HOST}": "host2",
            "{#PORT}": "out",
            "{#COUNT}": "6"
        },
        {
            "{#HOST}": "host3",
            "{#PORT}": "out",
            "{#COUNT}": "5"
        }
    ]
}

And filter:

$.data[?(@.{#PORT} == 'out' )].{#COUNT}

Now I'm getting the error "json path not valid: unsupported character in json path starting with".

Comment by Craig Hopkins [ 2019 Jan 07 ]

Lack of string searching is frustrating:

I can get $.body.quota[0].quota_remaining, but I can't look for a specific ID and return it. This means if the order of the array ever changes, my JSON is getting the wrong output.

Specifically, trying to run

$.body.quota[?(@.ID=="43639"].quota_remaining

which is perfectly valid in JSONPath, but returns

Item preprocessing step #1 failed: cannot extract value from json by path "$.body.quota[?(@.ID=="43639"].quota_remaining": unsupported character in json path starting with: "[?(@.ID=="43639"].quota_remaining"

in Zabbix.

 

Sample JSON:

 

{
 "quota": [
    {
      "ID": "43639",
      "quota_remaining": "243620855268",
      "quota_timestamp": "2018-12-04 14:00:00"
    },
    {
      "ID": "12345",
      "quota_remaining": "243620123456",
      "quota_timestamp": "2018-12-04 14:00:00"
    }    
  ]
}

 
Comment by Vitaly Zhuravlev [ 2019 Jan 17 ]

We can check if we can extend our basic JSONPath to support ?() filter expression as described here:

https://goessner.net/articles/JsonPath/index.html

It seem that the most asked Zabbix use case is to filter JSON object by the value of it's attribute:

 

?(@.id) - Object that has attribute id with any value 

?(@.id == 'something') - object has attribute that equals to 'something'

?(@.id > 10) - object has attribute which is > than 10

?(@.id < 10) - object has attribute which is < than 10

also to support:
?(@.id == 10)
?(@.id != 10)
?(@.id >= 10)
?(@.id <= 10)

Dear voters, would that be enough?

Comment by Craig Hopkins [ 2019 Jan 17 ]

That's definitely the functionality I'm after

Comment by Seth Simmons [ 2019 Jan 17 ]

That would be an amazing addition, I desperately need this capability to take full use of dependent items.

Comment by Rodrigo Leme Nunes [ 2019 Feb 04 ]

I'm  using regex for while:

Expression:

"Other - ADR block file read".?"{#WAIT}":"([^"])

json:

{ "data":[ {"{#FILTRO}":"1","{#EVENTO}":"Other - ADR block file read","{#WAIT}":"12.57"}]}

 

Return: 12.57

 

 

Comment by darkblaze69 [ 2019 Feb 11 ]

Also waiting for this feature. Not only about LLD, but it would be great also in simple http agent use to filter out values by IDs (attribute names) not by index.

Comment by James Howe [ 2019 Feb 12 ]

Even support for a simple .. or [*] is missing. You need this in e.g. elasticsearch monitoring, where you can query a single item but don't know its id (P5NnsStFQgmhq-rDesa3PA).

{"nodes":{"P5NnsStFQgmhq-rDesa3PA":{"timestamp":1549990321690,"name":"node-1.example.com","transport_address":"127.0.0.1:9300","host":"127.0.0.1","ip":"127.0.0.1:9300","roles":["master","data","ingest"],"attributes":{"ml.machine_memory":"3934777344","xpack.installed":"true","ml.max_open_jobs":"20","ml.enabled":"true"},"indices":{"store":{"size_in_bytes":1809484588}}}}}

If the expression ends up returning a list, Zabbix should just use the first value.

Comment by Lukáš Malý [ 2019 Mar 11 ]

I have a similar problem

JSON Result:

{{{ "jobs": [\{ "name": "PRC_CLV", "status": "OK"},\{ "name": "Subscribers", "status": "OK"},\{ "name": "K_ElComm", "status": "OK"},\{ "name": "K_Trans", "status": "OK"},\{ "name": "AML_BI", "status": "OK"},\{ "name": "K_FavNum", "status": "OK"},\{ "name": "K_PersInfo", "status": "OK"},\{ "name": "PlayersBonuses_SAP", "status": "OK"},\{ "name": "PlayersBonuses_BI", "status": "OK"},\{ "name": "GamesTotalBetsWins_SAP", "status": "OK"},\{ "name": "GamesTotalBetsWins_BI", "status": "OK"},\{ "name": "Transactions_SAP", "status": "OK"},\{ "name": "BRANCHES_IMPORT", "status": "OK"},\{ "name": "Correspondence_BI", "status": "OK"},\{ "name": "ODSPlayerSync", "status": "OK"},\{ "name": "OperatorsDetails_BI", "status": "OK"},\{ "name": "PlayersCases_BI", "status": "OK"},\{ "name": "OperatorsAuditTransactionsChanges_BI", "status": "OK"},\{ "name": "PlayersDocuments_BI", "status": "OK"},\{ "name": "OperatorsAuditPlayersChanges_BI", "status": "OK"},\{ "name": "OperatorsAuditPPMsChanges_BI", "status": "OK"},\{ "name": "PlayersCaseActivities_BI", "status": "OK"},\{ "name": "Transactions_BI", "status": "OK"},\{ "name": "PlayersLogins_BI", "status": "OK"},\{ "name": "PlayersPaymentMethods_BI", "status": "OK"},\{ "name": "GetSportTicketComboSystemLines", "status": "OK"},\{ "name": "PlayersBets_SAP", "status": "OK"},\{ "name": "Players_BI", "status": "OK"},\{ "name": "Extranet_EVOS", "status": "OK"},\{ "name": "Extranet_AD", "status": "OK"},\{ "name": "UIStatistics", "status": "OK"},\{ "name": "Extranet_SAP", "status": "OK"},\{ "name": "PlayersBets_BI", "status": "OK"},\{ "name": "Billing_Fraud", "status": "OK"},\{ "name": "Funnel_Fraud", "status": "OK"},\{ "name": "GetDeclinedTickets", "status": "OK"},\{ "name": "GetCombinatorBets", "status": "OK"},\{ "name": "GetSportTickets", "status": "OK"},\{ "name": "GetCustomers", "status": "OK"},\{ "name": "GetSportsMarketList", "status": "OK"},\{ "name": "DWHICTTRANS_dbo_Agent", "status": "OK"},\{ "name": "DWHICTTRANS_sce_proactive", "status": "OK"},\{ "name": "DWHICTTRANS_vw_ContactsByContacts", "status": "OK"},\{ "name": "DWHICTTRANS_cls_Contacts", "status": "OK"},\{ "name": "DWHICTTRANS_cls_Actions", "status": "OK"},\{ "name": "DWHICTTRANS_dbo_eCSRStat", "status": "OK"},\{ "name": "DWHICTTRANS_dbo_iAgentPerformanceStat", "status": "OK"},\{ "name": "DWHICTTRANS_dbo_iSkillsetStat", "status": "OK"},\{ "name": "GDPR_SAZKAMOBILE", "status": "OK"},\{ "name": "PlayersBonusesBI_XXmin", "status": "OK"},\{ "name": "TransactionsBI_XXmin", "status": "OK"},\{ "name": "PlayersBetsBI_XXmin", "status": "OK"},\{ "name": "PlayersBI_XXmin", "status": "OK"}]}}}

JSON Path depend item

$.jobs[?(@.name=='FindCharacter')].status

 

At http://jsonpath.com/ it works like this:

$.jobs[?(@.name=='PRC_CLV')].status return ["OK"]

Comment by Craig Hopkins [ 2019 Apr 02 ]

Any news on this one? I was hoping it would have been in 4.2, even in just an initial form. 

Comment by Vitaly Zhuravlev [ 2019 Apr 02 ]

Sorry, not coming in 4.2. You may consider to use JavaScript preprocessing at this point.

Comment by Kirill Savin [ 2019 May 06 ]

I has tested on version 4.2.1, the problem remains,

The expression "$.directoryQuotas[?(@.type == 'dir')].name" in the LLD macros section gives an error:

Cannot process LLD macro "{#SNQUOTAID}": json path not valid: unsupported character in json path starting with: "[?(@.type=='dir')].name".

Comment by Roberto Cremasco [ 2019 Jun 04 ]

Supporting paths like "$.*~" would allow to create any number of dependent items.

Also, I'm currently using Javascript Preprocessing to work around the JSONPath limits, but I'm incurring in more JSONPath limits... seems like the current implementation doesn't support "-" inside paths.

Path:

// $.disks.VolGroup-lv_root.Size

Error:

// Cannot create item: invalid value for preprocessing step #1: json path not valid: unsupported character in json path starting with: "-lv_root.Size".

Expected:

// As it works in "JSONPath Online Evaluator"

Comment by Andris Zeila [ 2019 Jun 05 ]

With dot notation only alphanumeric, '_' and '.' characters can be used in field names. Bracket notation can be used for more complex names: $.disks['VolGroup-lv_root'].Size

Comment by Roberto Cremasco [ 2019 Jun 05 ]

Thanks. I'm now bracket-escaping everything that matches

key.search(/^[0-9]*$|^[A-z]*$/) < 0
Comment by Gutsycat [ 2019 Jun 21 ]

I'm completely confused with "Fix Version/s: 4.0.11rc1" and that Vitaly Zhuravlev said "Sorry, not coming in 4.2.".
So Will it be in 4.0.11? I'll keep my fingers crossed.

Comment by Andris Mednis [ 2019 Jun 21 ]

I'm reviewing branch "feature/ZBXNEXT-4502-4.0", planned for 4.0.

Comment by Andris Zeila [ 2019 Jul 16 ]

Released ZBXNEXT-4502 in:

  • pre-4.0.11rc1 fa62eac8d3
  • pre-4.2.5rc1 8ceddc49eb
  • pre-4.4.0alpha1 c9de6ac944

The detailed description of implemented features will be added to documentation later. It's close to Java implementation with some features added and some removed.Shortly:

  • only comparison and regular expression operators are supported
  • sum(), first() functions are added
  • stddev() function is not supported
Comment by Craig Hopkins [ 2019 Jul 16 ]

Just in case I've missed it, do we know when 4.0.11 and 4.2.5 are due for release?

Comment by Andris Mednis [ 2019 Jul 16 ]

~ 2 weeks

Comment by Alexander Vladishev [ 2019 Jul 22 ]

Updated documentation:

Comment by Vitaly Zhuravlev [ 2019 Sep 19 ]

It would be great to extend this functionality a little bit further now!  ZBXNEXT-5422

Generated at Fri Apr 26 22:33:31 EEST 2024 using Jira 9.12.4#9120004-sha1:625303b708afdb767e17cb2838290c41888e9ff0.