Uploaded image for project: 'ZABBIX FEATURE REQUESTS'
  1. ZABBIX FEATURE REQUESTS
  2. ZBXNEXT-5398

Prometheus LLD Filtering Improvement

XMLWordPrintable

    • Icon: Change Request Change Request
    • Resolution: Unresolved
    • Icon: Trivial Trivial
    • None
    • 4.2.6
    • None

      I will try to explain a problem, before writing some suggestion.
      In my example I have huge Prometheus output ~2000 metrics from kubernetes cluster.
      Since we are talking about kubernetes, basically creating simple items is absolutely not an option. Instead Low Level Discovery should be used.

      Currently I want to discover and create prototypes on this part:

      kube_node_status_allocatable{node="node1",resource="memory",unit="byte"} 1.504894976e+09
      kube_node_status_allocatable{node="node1",resource="pods",unit="integer"} 110
      kube_node_status_allocatable{node="node1",resource="cpu",unit="core"} 1
      kube_node_status_allocatable{node="node1",resource="ephemeral_storage",unit="byte"} 9.975103472e+09
      kube_node_status_allocatable{node="node1",resource="hugepages_2Mi",unit="byte"} 0
      kube_node_status_allocatable{node="node2",resource="memory",unit="byte"} 1.504894976e+09
      kube_node_status_allocatable{node="node2",resource="pods",unit="integer"} 110
      kube_node_status_allocatable{node="node2",resource="cpu",unit="core"} 1
      kube_node_status_allocatable{node="node2",resource="ephemeral_storage",unit="byte"} 9.975103472e+09
      kube_node_status_allocatable{node="node2",resource="hugepages_2Mi",unit="byte"} 0
      kube_node_status_allocatable{node="master",resource="pods",unit="integer"} 110
      kube_node_status_allocatable{node="master",resource="cpu",unit="core"} 2
      kube_node_status_allocatable{node="master",resource="ephemeral_storage",unit="byte"} 1.9317915617e+10
      kube_node_status_allocatable{node="master",resource="hugepages_2Mi",unit="byte"} 0
      kube_node_status_allocatable{node="master",resource="memory",unit="byte"} 3.868737536e+09
      

      I am interested only in memory, cpu and storage resources.
      Since metric name is the same, I would want to manage all this within one single LLD.

      I create a LLD rule with Prometheus to JSON

      kube_node_status_allocatable{node=~".*",resource=~".*",unit=~".*"}
      

      And filter

      {#RESOURCE} matches memory|cpu|ephemeral_storage
      

      Note, that I cannot use one single Item prototype because of multiple reasons

      • Units of item itself are different in all 3 available resources
      • Type of information is different (As example memory is float and cpu is integer)
      • Trigger thresholds should be different for each resource
      • etc

      What I tried to do is:
      create a prototype for memory and add prometheus pattern preprocessing, where i statically define that resource should be memory and units=byes for all discovered nodes.

      {#METRIC}{node="{#NODE}",resource="memory",unit="byte"}
      

      Obviously it doesn't work as expected. I still get items created for all discovered resources. Simply all of them have static resource and unit label in preprocessing steps.

      From this result, I don't see any other way to achieve my result as only creating separate LLD rules for each resource.
      In this kube_node_status_allocatable example I would have 3 LLD rules. But I have more then 1000 metrics available. Creating and maintaining all those LLDs seems like a nightmare.

            Unassigned Unassigned
            dlamberts Dmitrijs Lamberts
            Votes:
            6 Vote for this issue
            Watchers:
            10 Start watching this issue

              Created:
              Updated: