Uploaded image for project: 'ZABBIX BUGS AND ISSUES'
  1. ZABBIX BUGS AND ISSUES
  2. ZBX-17694

High Memory utilization by preprocessing manager

    XMLWordPrintable

    Details

    • Team:
      Team A
    • Sprint:
      Sprint 64 (May 2020), Sprint 65 (Jun 2020)
    • Story Points:
      1

      Description

      Problem related with a gathering data from kafka_exporter.
      HTTP agent item (basic GET request) is using as Master item for 11 LLD Rules with 1 item prototype (the same Master Item) on each one. Discovered >= 1500 items for each LLD.
      Size of web page - 1,4 Mb
      Each LLD using Preprocessing (Prometheus to json), LLD macro are defined by JsonPath
      Each Item Prototype using Preprocessing steps: 1. Prometheus pattern 2. Custom multiplier (1)
      Master Item update interval - 5 minutes
      In this configuration subprocess preprocessing manager using over 19Gb of memory in customer environment, and RAM is not freed after time
      Situation is reproduced in test environment (Zabbix-Proxy 4.4.7 4 CPU, 8Gb RAM, 10 Pollers, 10 Preprocessing Workers), template crushed the system (not enough memory)

      Then, I've tryed to modify the rules - clone Master Item to 5 ones (same logic with uniqe key) and distributed LLDs among them (2-3 LLD per one Master Item)
      Result - only 4,5 Gb of memory is using by preprocessing manager, but it's not freed after time

      I've inreased update interval for all master items to 10m, updated config cache on server and proxy - no result. After restarting the proxy - bursts of disposal are observing every 10 minutes, but memory is freed finally.

       

      Steps to reproduce:

      1. Import the template  kafka to zabbix 
      2. Create simple web document with kafka exporter data
      3. Create a new host, link it to the some proxy and template
      4. Wait for creating all items from item prototypes

      Result:

      High RAM Utilization by preprocessing manager process

      Expected:
      RAM is freed by preprocessing manager fast without need of zabbix-proxy restarting and separating LLDs to several master items

        Attachments

        1. aps.json
          212 kB
        2. callgrind.png
          callgrind.png
          385 kB
        3. image-2020-05-08-14-10-04-050.png
          image-2020-05-08-14-10-04-050.png
          115 kB
        4. image-2020-05-29-14-18-18-631.png
          image-2020-05-29-14-18-18-631.png
          149 kB
        5. image-2020-05-29-14-29-00-413.png
          image-2020-05-29-14-29-00-413.png
          33 kB
        6. Proxy with json.jpg
          Proxy with json.jpg
          49 kB
        7. Proxy without json.jpg
          Proxy without json.jpg
          14 kB
        8. ZBX-17694-5.0-TEST.diff
          11 kB

          Activity

            People

            Assignee:
            vso Vladislavs Sokurenko
            Reporter:
            igorbach Igor Gorbach
            Votes:
            5 Vote for this issue
            Watchers:
            10 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved: