Uploaded image for project: 'ZABBIX BUGS AND ISSUES'
  1. ZABBIX BUGS AND ISSUES
  2. ZBX-21570

Kubernetes: Get state metrics cannot fetch public kube-state-metrics

    XMLWordPrintable

Details

    • Incident report
    • Status: Confirmed
    • Trivial
    • Resolution: Unresolved
    • 6.2.2
    • None
    • Templates (T)
    • None

    Description

      Hello,

      We used the Zabbix Helm Chart to deploy zabbix_agent and kube-state-metrics in our cluster. It works well.

      However, since we already have 2 Zabbix Proxy servers in the subnet, we don't want to deploy ZabbixProxy inside the cluster and want to re-use our already working Zabbix Proxy server.

      How the Kubernetes: Get state metrics template is currently working (3 steps)

      Template : https://git.zabbix.com/projects/ZBX/repos/zabbix/browse/templates/app/kubernetes_http/kubernetes_state_http/template_kubernetes_state.yaml

      • apiRequest() check if api-server API endpoint is working with the provided Auth Token, Urls,....
      • getMetricsEndpoint() do a GET request to /api/v1/endpoints and fetch the IP / Port of the kube-state-metrics Pod. (ex: 172.16.0.57:8080)
      • getStateMetrics() do a GET request to [http://\|http:]{{ IP }}:{{ PORT}}/metrics from previous values returned by getMetricsEndpoint()

      It only works when Zabbix-Proxy is deployed inside the cluster, because /api/v1/endpoints only return private IPs (containers subnet).

      Because we can disable the deployments of ZabbixProxy in the HelmChart, we should be allowed to use an external kube-state-metrics endpoint through a LoadBalancer or NodePort service and then configure a Template variable to fetch metrics from that custom endpoint.

      As a workaround, i've removed the ApiRequest() and getMetricsEndpoint() functions and hardcoded the URL variable in getMetricsEndpoint() to match the a new zabbix template variable : {$KUBE.STATE.ENDPOINT.URL}

      var response,
        request = new HttpRequest(),
        url = Kube.params.state_endpoint_url;
      

      and provide a new Template variable :

      state_endpoint_url = "https://kube-state-metrics.private.k8s.local:8080/metrics"
      

      I'm not comfortable with Zabbix, so not sure if this is the best solution to make the template compatible with internal or external exposition of kube-state-metrics.

      Another solution could be to request /api/v1/services instead of /api/v1/endpoints to fetch the external IP?

      Kindly,
      Benjamin

      Attachments

        Activity

          People

            zit Zabbix Integration Team
            bpinchon Benjamin Pinchon
            Votes:
            1 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated: