Problem related with a gathering data from kafka_exporter.
HTTP agent item (basic GET request) is using as Master item for 11 LLD Rules with 1 item prototype (the same Master Item) on each one. Discovered >= 1500 items for each LLD.
Size of web page - 1,4 Mb
Each LLD using Preprocessing (Prometheus to json), LLD macro are defined by JsonPath
Each Item Prototype using Preprocessing steps: 1. Prometheus pattern 2. Custom multiplier (1)
Master Item update interval - 5 minutes
In this configuration subprocess preprocessing manager using over 19Gb of memory in customer environment, and RAM is not freed after time
Situation is reproduced in test environment (Zabbix-Proxy 4.4.7 4 CPU, 8Gb RAM, 10 Pollers, 10 Preprocessing Workers), template crushed the system (not enough memory)
Then, I've tryed to modify the rules - clone Master Item to 5 ones (same logic with uniqe key) and distributed LLDs among them (2-3 LLD per one Master Item)
Result - only 4,5 Gb of memory is using by preprocessing manager, but it's not freed after time
I've inreased update interval for all master items to 10m, updated config cache on server and proxy - no result. After restarting the proxy - bursts of disposal are observing every 10 minutes, but memory is freed finally.
Steps to reproduce:
- Import the template kafka to zabbix
- Create simple web document with kafka exporter data
- Create a new host, link it to the some proxy and template
- Wait for creating all items from item prototypes
High RAM Utilization by preprocessing manager process
RAM is freed by preprocessing manager fast without need of zabbix-proxy restarting and separating LLDs to several master items