|
ZBX_ES_MEMORY_LIMIT controls the amount of memory that can be allocated by scripts on each poller. It looks like it used to be 10MB, but was increased to 64MB in ZBXNEXT-6386 and to 512MB in ZBX-22805. Currently Zabbix 6.2 has 64MB limit, 6.0, 6.4 and master - 512MB.
To test how this affects memory consumption, only one poller process was started, 110 memory-intensive script items were created, then "Execute now" was used for all items. Here's the script that was used:
function randInt(min, max)
{
return min + Math.floor(Math.random() * (max - min + 1));
}
var len = randInt(1000000, 2000000);
var sum = 0;
var arr = new Array(len);
for (var i = 0; i < len; i++)
{
arr[i] = i;
}
for (var i = 0; i < len; i++)
{
sum += arr[i] & 0xFFFFFFFF;
}
return sum & 0x0FFFFFFF;
Lowering memory limit to 64MB and adding some logging in es_malloc() reveals that it often returns NULL (i.e., script engine has already used all "available" memory) - almost every call to get_value_script() results in "out of memory". Though script items are executed successfully. Memory consumption of dev virtual machine varies between 2.21GB and 2.28GB during the test.
Rising memory limit back to 512MB gives considerably larger memory consumption and much smaller number of "failed" calls to es_malloc() which seem to be "periodic" - once per ~20 calls to get_value_script(). Memory consumption of dev vm is between 2.31GB and 2.72GB during the test.
When memory limit is increased to 2048MB, calls to es_malloc() don't result in "out of memory" anymore with the mentioned test items, though the memory consumption gets very high until all "Execute now" tasks are finished. When they're done, most of the memory gets released, though memory consumption during idle periods still remains higher than with lower ZBX_ES_MEMORY_LIMIT values.
From the tests it looks like script engine is optimizing memory management either by reusing allocated memory or by delaying releasing the memory. When ZBX_ES_MEMORY_LIMIT is high, this can result in high memory usage.
To fix the issue, either ZBX_ES_MEMORY_LIMIT has to be lowered (we might want to make it configurable), or script engine has to release memory more agressively.
This might apply to JavaScript preprocessing and other places where script engine is used.
| 
image-2023-11-17-17-59-38-905.png
Team A
