Hi guys, checking on the script responsible for gathering nodes and pods information [1], we think we might be hitting a memory limit in [2][3]. Pod information is gathered in batches of 1000 from Kubernetes API and stored in the array 'result' [2]. In our case, the information gathered from pods exceeds the maximum allowed [3],  and there is no way to override the memory limit or implement any annotation and label filters, as filters are processed only later in the script.

Changing the annotation and label filters to iterate over the "1000" limit batches [5] before storing pods in the "result" array first may help mitigate the issue, as we could apply them to get only the pods we really want to track and reduce the need for memory. 

Perhaps another alternative would be creating an environment variable to override the memory limit instead of having it hard-coded, but maybe there is some good reason to keep it that way and I'm just not seeing it yet.

[1] https://git.zabbix.com/projects/ZBX/repos/zabbix/browse/templates/app/kubernetes_http/kubernetes_nodes_http/template_kubernetes_nodes.yaml?at=refs%2Fheads%2Frelease%2F6.4#37

[2] https://git.zabbix.com/projects/ZBX/repos/zabbix/browse/templates/app/kubernetes_http/kubernetes_nodes_http/template_kubernetes_nodes.yaml?at=refs%2Fheads%2Frelease%2F6.4#173

[3] https://git.zabbix.com/projects/ZBX/repos/zabbix/browse/src/libs/zbxembed/embed.c?at=refs%2Ftags%2F6.4.10#31

[4] https://git.zabbix.com/projects/ZBX/repos/zabbix/browse/src/libs/zbxembed/embed.c?at=refs%2Ftags%2F6.4.10#659

[5] https://git.zabbix.com/projects/ZBX/repos/zabbix/browse/templates/app/kubernetes_http/kubernetes_nodes_http/template_kubernetes_nodes.yaml?at=refs%2Fheads%2Frelease%2F6.4#39

Additional info:

The large server of my initial post has around 3500 pods total.
Even on a smaller server with ~300-400 pods the pod discovery itsself works (no memory issue) but results in:

Cannot create item "kube.pod.phase.failed[somepod]": maximum dependent item count reached.

Hello!

Thank you for reporting the issue and for the detailed investigation!

As you're able to see - the problem is related to hardcode limits by defining the environment variable or changing the source values directly in

https://git.zabbix.com/projects/ZBX/repos/zabbix/browse/src/libs/zbxembed/embed.c?at=refs%2Ftags%2F6.4.10#31

and build the custom zabbix-proxy image for the helm chart, but be careful with implementation to prod.

Regards, Igor

The same is true for the maximum dependent item count (29999 limit by default).

Igor, do you see any potential implications with raising those limits? (memory and dependent item one?)

Hi igorbach, wouldn't it make sense to have ZBX_ES_MEMORY_LIMIT as an environment variable? I see the memory limit increased over time [1][2]. 

That would help us to tweak the memory limit to fit our requirements.

[1] https://support.zabbix.com/browse/ZBXNEXT-6386

[2] https://support.zabbix.com/browse/ZBX-22805

Hi sfernand, these size ZBX_ES_MEMORY_LIMIT are made by security design.

Create ZBXNEXT task with full reasoning that the ZBX_ES_MEMORY_LIMIT as an environment variable needs to be added to the configuration.

hi,

Changing ZBX_ES_MEMORY_LIMIT wasn't our first option but considering it has been increased several times in the past I believe it is worth considering as a feature request. We would like to know from you if any other alternatives or workarounds would be possible to prevent the get Nodes/Pods from reaching these limits. Maybe changing the script itself, splitting it in two so we would have a script only for gathering node data and a separate one for pods).

I've created a feature request to make ZBX_ES_MEMORY_LIMIT configurable through an environment variable https://support.zabbix.com/browse/ZBXNEXT-9685