I think an underutilized function of zabbix is is service monitoring https://www.zabbix.com/documentation/4.0/manual/it_services. Service monitoring allows for big picture and aggregate level awareness of where all the problems are in a large enterprise scale. It also allows you to provide zabbix a hierarchical and structured awareness of all the dependencies of a complex infrastructure and container environment.

These days in cloud, yml, json, infrastructure, container, etc, alot of the hierarchical dependency linkage in complex systems is already defined by service teams that run their service. Parsing that data in some way and from it creating zabbix service definitions linked to triggers for the iterative components of that service would let zabbix dynamically map and understand the big picture linkage. e.g. kubectl json api output mapping to the N layers of zabbix servers could be created with code/llds for the zabbix server definitions.

This could also allow for a phenomenal de-duplicaton of alerts by showing the highest services and hosts in an infrastructure hierarchical structure that have a Trigger in a True state. Sort of an autonomous root cause analysis that would only show you, or could only alert you for the services and hosts highest in your hierarchical structure of nodes and containers that have a firing trigger.

However the only way i see to create zabbix service definitions is by clicking and through the zabbix rest api. both paths are time consuming and not ideal and right now we use written runbooks to teach people how to root cause things. 

 

The only other alternative i've seen to achieve something like this large scale infrastructure root cause analysis and alert deduplication is per trigger level AND/OR/NOR/XOR logic. and to do so you have to link a trigger from one template, to a trigger from another template, e.g. trigger for linux os issue with trigger for a application specific ngins/apache state. That doesn't scale well and I think the Service monitoring linked with LLDs and macros could be extremely advantageous.

For large complex infrastructures and cloud with 10,000 to 100,000 nodes or more like mine and lots of networ, storage, containers, compute, hots and more metrics, this would be hugely beneficial.