Uploaded image for project: 'ZABBIX BUGS AND ISSUES'
  2. ZBX-17694

High Memory utilization by preprocessing manager


    • Sprint 64 (May 2020), Sprint 65 (Jun 2020)
    • 1

      Problem related with a gathering data from kafka_exporter.
      HTTP agent item (basic GET request) is using as Master item for 11 LLD Rules with 1 item prototype (the same Master Item) on each one. Discovered >= 1500 items for each LLD.
      Size of web page - 1,4 Mb
      Each LLD using Preprocessing (Prometheus to json), LLD macro are defined by JsonPath
      Each Item Prototype using Preprocessing steps: 1. Prometheus pattern 2. Custom multiplier (1)
      Master Item update interval - 5 minutes
      In this configuration subprocess preprocessing manager using over 19Gb of memory in customer environment, and RAM is not freed after time
      Situation is reproduced in test environment (Zabbix-Proxy 4.4.7 4 CPU, 8Gb RAM, 10 Pollers, 10 Preprocessing Workers), template crushed the system (not enough memory)

      Then, I've tryed to modify the rules - clone Master Item to 5 ones (same logic with uniqe key) and distributed LLDs among them (2-3 LLD per one Master Item)
      Result - only 4,5 Gb of memory is using by preprocessing manager, but it's not freed after time

      I've inreased update interval for all master items to 10m, updated config cache on server and proxy - no result. After restarting the proxy - bursts of disposal are observing every 10 minutes, but memory is freed finally.


      Steps to reproduce:

      1. Import the template  kafka to zabbix 
      2. Create simple web document with kafka exporter data
      3. Create a new host, link it to the some proxy and template
      4. Wait for creating all items from item prototypes


      High RAM Utilization by preprocessing manager process

      RAM is freed by preprocessing manager fast without need of zabbix-proxy restarting and separating LLDs to several master items

        1. aps.json
          212 kB
        2. callgrind.png
          385 kB
        3. image-2020-05-08-14-10-04-050.png
          115 kB
        4. image-2020-05-29-14-18-18-631.png
          149 kB
        5. image-2020-05-29-14-29-00-413.png
          33 kB
        6. Proxy with json.jpg
          Proxy with json.jpg
          49 kB
        7. Proxy without json.jpg
          Proxy without json.jpg
          14 kB
        8. ZBX-17694-5.0-TEST.diff
          11 kB

            vso Vladislavs Sokurenko
            igorbach Igor Gorbach
            Team A
            5 Vote for this issue
            10 Start watching this issue