[ZBXNEXT-1527] cascaded/nested lld Created: 2012 Nov 28  Updated: 2024 Feb 27

Status: Open
Project: ZABBIX FEATURE REQUESTS
Component/s: API (A), Frontend (F), Server (S)
Affects Version/s: None
Fix Version/s: None

Type: New Feature Request Priority: Trivial
Reporter: richlv Assignee: Unassigned
Resolution: Unresolved Votes: 107
Labels: lld, nested
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Causes
causes ZBXNEXT-7527 Nested host prototypes Open
Duplicate
is duplicated by ZBX-12436 Graph prototype process create only t... Closed
is duplicated by ZBX-9762 Nested LLD Closed
is duplicated by ZBX-11111 Smarter nested LLD? Closed

 Description   

in some cases one-level lld is not enough. for example, one might want to discover all databases and then discover all tables for them, to monitor some statistic for all of them.

this might be doable by creating some sort of "lld rule prototype", which could then cascade almost indefinitely (although i can't think of more than 2 level practical application right now)



 Comments   
Comment by Volker Fröhlich [ 2014 Sep 26 ]

http://zabbix.org/wiki/Docs/howto/Nested_LLD

Comment by Stefan [ 2014 Sep 26 ]

i have an other example: to monitor Dell Equallogic Disks i need 2Querys, for example:
snmpwalk -v2c -c public 192.168.1.1 .1.3.6.1.4.1.12740.2.1.1.1.9.1
iso.3.6.1.4.1.12740.2.1.1.1.9.1.563984362 = STRING: "foo"
so this number is need to get the disks:
snmpwalk -v2c -c public 192.168.1.1 .1.3.6.1.4.1.12740.3.1.1.1.8.1
iso.3.6.1.4.1.12740.3.1.1.1.8.1.563984362.1 = INTEGER: 1
iso.3.6.1.4.1.12740.3.1.1.1.8.1.563984362.2 = INTEGER: 1
iso.3.6.1.4.1.12740.3.1.1.1.8.1.563984362.3 = INTEGER: 1
iso.3.6.1.4.1.12740.3.1.1.1.8.1.563984362.4 = INTEGER: 1
iso.3.6.1.4.1.12740.3.1.1.1.8.1.563984362.5 = INTEGER: 1
iso.3.6.1.4.1.12740.3.1.1.1.8.1.563984362.6 = INTEGER: 1
iso.3.6.1.4.1.12740.3.1.1.1.8.1.563984362.7 = INTEGER: 1
iso.3.6.1.4.1.12740.3.1.1.1.8.1.563984362.8 = INTEGER: 1
iso.3.6.1.4.1.12740.3.1.1.1.8.1.563984362.9 = INTEGER: 1
iso.3.6.1.4.1.12740.3.1.1.1.8.1.563984362.10 = INTEGER: 1
iso.3.6.1.4.1.12740.3.1.1.1.8.1.563984362.11 = INTEGER: 1
iso.3.6.1.4.1.12740.3.1.1.1.8.1.563984362.12 = INTEGER: 2

actually i can only use SNMPINDEX and i get 563984362, but i need the last number too. so i must use .1.3.6.1.4.1.12740.3.1.1.1.8.1.{#SNMPINDEX}.1, a better will be .1.3.6.1.4.1.12740.3.1.1.1.8.1.{#SNMPINDEX1}.{#SNMPINDEX2} to get the second one, or other idea {#SNMPINDEX(2)} to get the last 2 numbers (563984362.1), or {#SNMPINDEX(#7)} to get a specific number (for #7 i will get 12740)

Comment by Naruto [ 2016 Aug 18 ]

Should this be closed? It's supported now.

Comment by Raimo [ 2016 Aug 18 ]

It is supported now? How? Can't find it in the documentation.

Comment by Raimo [ 2016 Aug 19 ]

Stefan Krügers approach above (.1.3.6.1.4.1.12740.3.1.1.1.8.1.

{#SNMPINDEX1}

.

{#SNMPINDEX2}

) is the way to go in my opinion. It is the most intuitive solution from a users perspective....

Comment by Naruto [ 2016 Aug 19 ]

We are touching two different topics here, one is the built-in "discovery" keys, and another is the support for custom LLD rules.

As Volker pointed out, custom nested LLD's are somehow already supported: http://zabbix.org/wiki/Docs/howto/Nested_LLD

Comment by richlv [ 2016 Aug 19 ]

well, technically, the method you are using is nice & it works, but it is still a hack and not officially supported, so it would still be more like "adding support for nested lld"

Comment by Naruto [ 2016 Aug 19 ]

The complete structure of the discovered nested values is defined by the user on the template.
Asking the user to define that structure again on the JSON code is redundant and an open door to many problems; it's complicated and overloaded.

Think in folders and labels. They are two different approaches. Gmail uses labels, no hierarchy. Filesystems use directories, that's a hierarchy.

When you have already defined the hierarchy on the template side (for example, I'll use the "#DATABASE" macro for zabbix Applications, and "#TBL" for item names), there is no need to define the same hierarchy again on the JSON making side. Therefore the "labels" approach is the right approach.

Each line of the JSON code must represent a combination of labels.

But there is a problem with the current implementation of this feature.
For example, we'll suppose that I am Zabbix Server and I receive this JSON:

{
 "data":[
   { "{#INSTANCE}" : "MSSQL$TEST_INST", "{#DATABASE}" : "DBA" },
   { "{#INSTANCE}" : "MSSQL$TEST_INST", "{#DATABASE}" : "DBA2" },
   { "{#INSTANCE}" : "SQLServer", "{#DATABASE}" : "SQL_SERVER_MONITORING" },
   { "{#INSTANCE}" : "SQLServer", "{#DATABASE}" : "ORACLE_MONITORING" },
   { "{#INSTANCE}" : "SQLServer", "{#DATABASE}" : "DBA" },
 ]
}
  • Ok, now I know that I have to create items for a prototype that is referring to "#DATABASE" macro/label only. I'll create 4 items, one for each "#DATABASE" value: "DBA", "DBA2", "SQL_SERVER_MONITORING" and "ORACLE_MONITORING"; and I'll rise an error, because the fifth value is repeated ("DBA"). OK, it is not possible, and the request is not normal, I mean, why should I want a list of all databases, when I know that I could have the same name in multiple instances (by coincidence)?, it's weird. So, ok, next prototype.
  • It turns out that I've been asked to create items for another prototype, referring to both "#INSTANCE" and "#DATABASE". I will create 5 items:
    MSSQL$TEST_INST - DBA
    MSSQL$TEST_INST - DBA2
    SQLServer - SQL_SERVER_MONITORING
    SQLServer - ORACLE_MONITORING
    SQLServer - DBA
    And no error. OK. This request is normal.
  • Great, now I'm being asked to create items for another prototype, this time referring to "#INSTANCE" only. It's normal! I want to refer to instances, completely normal request! However I, Zabbix, throw errors for lines 2, 4 and 5, because I'm blindly trying to create an item for each JSON line.

How I see the solution to this problem? Simple, we don't need errors here; the user knows what he's doing; just don't try to create an item that is already created, in the first place. I can admit that that first request was weird, I know it, but it could be useful in some scenarios, having a list of unique values across many "containers".
Now, the third request, it's 100% normal, and wanted, and useful.

The question is, what is the purpose of attempting to create something that was already created? There was already an item created for "SQLServer", then why is Zabbix trying to perform the same action again? It makes no sense.

Comment by richlv [ 2016 Aug 23 ]

i wouldn't say this is supported.
host prototypes sort of provide one extra level, but only if you want separate hosts.
the hack with the somewhat duplicate entries is mostly working, and it might even end up being the supported method, but until it does become one, this issue still tracking that

Comment by Naruto [ 2016 Aug 23 ]

Hi, richlv, I know, right, that's why I'm opening this discussion about this feature, it's in an inconsistent state right now. It doesn't work well in one way or another. We need to define this, right now. What is the way to go, and how.

To me, it's pretty clear and easy. Just read my comment above, as I said there, labels is the way to go, and no hierarchy. Let the hierarchy in the user's mind, he knows what he's doing.

Otherwise you'll end up having to give support to things like this:

{
	"data": [
		{ "{#INS}": "MSSQL$TESTE_INST", "{#INL}": "TESTE_INST", "data": [
			{ "{#DBS}": "DBA" }, 
			{ "{#DBS}": "DBA2" }
	    ]},
	  	{ "{#INS}": "SQLServer", "{#INL}": "SQL Server", "data": [
			{ "{#DBS}": "SQL_SERVER_MONITORING" }, 
			{ "{#DBS}": "ORACLE_MONITORING" }, 
			{ "{#DBS}": "DBA" }
	   ]}
	]
}

Which is a pain in the butt, for both, the user and the developer.

Comment by richlv [ 2018 Apr 18 ]

Here's a usecase where the current hacks fall short.

A script discovers JMX ports on a box and creates JMX items for each found JVM. Now, on each JVM one might want to have another level of LLD, but that is not possible - the JMX discovery would have to stem from the discovered items.

The problem is that not all information can be gathered in one script and then sent to an LLD rule, different discovery methods/types have to be used.

Comment by Stéphane Olivier [ 2018 Dec 18 ]

Hi.

I'm trying to understand the following sentence from https://www.zabbix.org/wiki/Docs/howto/Nested_LLD:

"In a single go: You can implement the secondary rule with a trapper item of your choice, if possible connectivity-wise. Just invoke the trapping from your script"

For me, trapper items have to be created so you can use zabbix_sender to send values.

I don't see how to implement the second discovery rule with a trapper.

Thx

Comment by richlv [ 2018 Dec 18 ]

My interpretation:
The first method uses two LLD rules that separately gather information about the two levels (using the example from the wiki page, one to collect list of databases, and one to collect list of db:tables). The second method collects information on databases and tables in one go, and sends data to both LLD rules.

Comment by Stéphane Olivier [ 2018 Dec 18 ]

There is one problem with a big json file which includes dbs and tables. ie:

{
  "data": [
    { "#DBNAME": "db01", "#TBLNAME": "tbl01" },
    { "#DBNAME": "db01", "#TBLNAME": "tbl02" },
    { "#DBNAME": "db02", "#TBLNAME": "tbl01" },
    { "#DBNAME": "db02", "#TBLNAME": "tbl02" }
  ]
}

if you have an item property named db.size[#DBNAME] in your db LLD, you'll get duplicate keys.

That's why i was not sure to have properly understood the sentence

Comment by richlv [ 2018 Dec 18 ]

The script would send two different JSONs to two different rules, it's just that the data collection would happen once.

Comment by Stéphane Olivier [ 2018 Dec 18 ]

Data collection for both discovery rules would happen only once if you don't use "Zabbix agent" or "Zabbix agent (active)" as type for them else the script will be called twice.

Comment by Jack Valko [ 2021 Apr 21 ]

Here's another use case that really would make a nested LLD w/ host prototypes powerful.

I'm managing a large Meraki installation on 3 continents.  The only way to properly get data from devices is to use Meraki's API (snmpwalk wont scale and cant see devices behind a meraki appliance).  Meraki devices are configured hierarchically --> Org/Network/Device.

My Org discovery returns this:

{['name':'Org Name','id':'1234567','url':'https://a-meraki-url.com/foo'}

]

I then want to discover networks so I create a host from the 'name' key and use it to discover networks.  Once those networks are discovered I'd like to create hosts for those and start to discover devices.

I can write an external script to do this but I'd like to do this natively within Zabbix and not have to shell out every time I discover.  Nested LLD is about the only practical way to discover large groups of devices.  

Comment by dimir [ 2021 Aug 20 ]

jackvalko, you might be interested in voting and following ZBXNEXT-6844 then.

Comment by Stefan [ 2021 Aug 20 ]

Sorry, but no we are not interested in other features we want this

Comment by James Kirsop [ 2022 May 04 ]

I outlined another use case for nested LLD here: https://www.zabbix.com/forum/zabbix-help/433212-cascading-lld-building-juniper-cos-items-on-interface-and-queue-indexes

@palivoda can we get this request assigned to someone for review?

Comment by Gustavo Guido [ 2022 May 26 ]

The link https://www.zabbix.org/wiki/Docs/howto/Nested_LLD in pointing to https://www.zabbix.com/community, not to a particular doc, so I did not read it (sorry if Im saying something that is explained in that article)
There are many situations where the LLD should iterate, when adding tags to items or tirgger prototypes, when an entity has sub-entitys and you need to relate triggers (no trigger in sub-entity if the master entity is offline)

Comment by Gergely Czuczy [ 2022 Jun 27 ]

This would also make it possible to more easily monitor nginx JSON status.

The open module's format is here, it's an upstream/server nesting: https://github.com/nginx-modules/ngx_http_json_status_module

And the commercial official API module also has some nesting: https://nginx.org/en/docs/http/ngx_http_api_module.html

 

Comment by Gergely Czuczy [ 2022 Jun 28 ]

The Solr cluster status response is also a nested json: https://solr.apache.org/guide/6_6/collections-api.html#CollectionsAPI-clusterstatus

$.collections.{collection}.{shard}.replicas.{replica}

That's 3 levels of nesting. Without this feature, enterprises using solr cannot really utilize zabbix. It's pretty much like ZBXNEXT-1, when people need a template generator specific for their setup, due to a lack of a well needed feature in zabbix.

 

Comment by Stefan [ 2022 Jun 28 ]

maybe this feature can be funded by the community via https://bountysource.com/ ?

Comment by Oleg Ivanivskyi [ 2022 Aug 08 ]

Improved filtering is another possible use case for this feature.

For example, you would like to monitor SFP transceiver power status for a Cisco device. You can get all SFP modules via SNMP, e.g. port, serial, model, etc.:

.1.3.6.1.2.1.47.1.1.1.1.2.311 = STRING: "Transceiver (slot:11-port:11)"
.1.3.6.1.2.1.47.1.1.1.1.12.311 = STRING: "CISCO-AVAGO"
.1.3.6.1.2.1.47.1.1.1.1.13.311 = STRING: "SFP-10G"

Each module has different sensors, e.g. receive power, transmit power, etc. SFP module with 311 index has 5 different sensors:

.1.3.6.1.2.1.47.1.1.1.1.4.10001 = INTEGER: 311
.1.3.6.1.2.1.47.1.1.1.1.4.10002 = INTEGER: 311
.1.3.6.1.2.1.47.1.1.1.1.4.10003 = INTEGER: 311
.1.3.6.1.2.1.47.1.1.1.1.4.10004 = INTEGER: 311
.1.3.6.1.2.1.47.1.1.1.1.4.10005 = INTEGER: 311

Sensor data (e.g. name, status, etc.)

.1.3.6.1.2.1.47.1.1.1.1.2.10001 = STRING: "Ethernet11/11 Lane 1 Transceiver Receive Power Sensor"
.1.3.6.1.4.1.9.9.91.1.2.1.1.3.10001.1 = INTEGER: 3

At the moment, you can discover sensors but can't use module related information as a filter. I.e. you can't monitor power sensors for "SFP-10G" modules only.

Comment by Ben Hanson [ 2022 Sep 09 ]

"Improved filtering is another possible use case for this feature"

This is my motivator as well, though with a configuration/design use case rather than hardware discovery.

We utilize Cisco gear and do port discovery of all of our switches.  I currently filter based on port type and up/down status.  There is an SNMP OID that uses the interface index that will report CDP neighbor name(s) (.1.3.6.1.4.1.9.9.23.1.2.1.1.6.{IFINDEX}.#) .  Unfortunately, it has a trailing number after the index, and it also is only populated if a value is present.  If the LLD could include this OID in discovery, then I could create one discovery rule for ports with a neighbor based on device name prefix, and a second for everything else.  This would allow me to automatically assign triggers based on expected port type (uplink vs device, etc).

So, the ability to include {IFINDEX} in a discovery label would be helpful.

Comment by John Ivan [ 2022 Nov 11 ]

Might be related.  Using SNMP discovery to generate monitoring items based on a storage type.  Presently I can have several discoveries run the same SNMP walk, each with a different discovery filter by storage type (scheduled to avoid concurrency that causes the discovery to time out).  Seems to me this would be better with a master discovery (with no item prototypes) that is run once to return all storage types, then run dependent discoveries with preprocessing to match the storage type and generate the items.  The dependent discoveries do not have to repeat the SNMP walk.

This is similar to the existing feature where a Master Item run a command that returns multiple values and dependent items can preprocess those results without having to run the command again.  

Comment by Oleksii Zagorskyi [ 2023 Feb 03 ]

ZBXNEXT-7527 asks to host prototypes nesting, which is very related to current case, but still a little bit specific.

Comment by Craig Hopkins [ 2023 Feb 23 ]

Nested LLD would definitely be useful for multi-lane optics. The OID looks like

SNMPv2-SMI::enterprises.2636.3.60.1.2.1.1.6.{PORTINDEX}.{LANE}

but there's no way currently (that I can find) within Zabbix to support this.

Comment by LivreAcesso.Pro [ 2023 Jun 27 ]

Where the info at http://zabbix.org/wiki/Docs/howto/Nested_LLD go?

Comment by Volker Fröhlich [ 2023 Jun 27 ]

https://web.archive.org/web/20171115143613/http://zabbix.org/wiki/Docs/howto/Nested_LLD

The wiki is no more.

Comment by Arthur Ivanov [ 2024 Feb 27 ]

Another case for cascaded LLD is redfish API:

/redfish/v1/Chassis/<ID>/Power/PowerSupplies/<ID>

Currently, there is no way to enumerate Chassis and then Power Supplies within one template and http-agent.

Supermicro:

{
    "@odata.context": "/redfish/v1/$metadata#ChassisCollection.ChassisCollection",
    "@odata.type": "#ChassisCollection.ChassisCollection",
    "@odata.id": "/redfish/v1/Chassis",
    "Name": "Chassis Collection",
    "Members": [
        {
            "@odata.id": "/redfish/v1/Chassis/1"
        },
        {
            "@odata.id": "/redfish/v1/Chassis/HA-RAID.0.StorageEnclosure.0"
        }
    ],
    "[email protected]": 2
}

Lenovo:

{
    "@odata.type": "#ChassisCollection.ChassisCollection",
    "@odata.id": "/redfish/v1/Chassis",
    "Members": [
        {
            "@odata.id": "/redfish/v1/Chassis/1"
        },
        {
            "@odata.id": "/redfish/v1/Chassis/3"
        },
        {
            "@odata.id": "/redfish/v1/Chassis/4"
        }
    ],
    "@odata.etag": "\"2b1611aac39424e542c\"",
    "@odata.context": "/redfish/v1/$metadata#ChassisCollection.ChassisCollection",
    "Description": "A collection of Chassis resource instances.",
    "[email protected]": 3,
    "Name": "ChassisCollection"
}

Dell:

{
    "@odata.context": "/redfish/v1/$metadata#ChassisCollection.ChassisCollection",
    "@odata.id": "/redfish/v1/Chassis",
    "@odata.type": "#ChassisCollection.ChassisCollection",
    "Description": "Collection of Chassis",
    "Members": [
        {
            "@odata.id": "/redfish/v1/Chassis/System.Embedded.1"
        },
        {
            "@odata.id": "/redfish/v1/Chassis/Enclosure.Internal.0-1:RAID.SL.3-1"
        }
    ],
    "[email protected]": 2,
    "Name": "Chassis Collection"
}

ZBXNEXT-3643

Generated at Fri Apr 26 13:18:54 EEST 2024 using Jira 9.12.4#9120004-sha1:625303b708afdb767e17cb2838290c41888e9ff0.