Hi,
Please share zabbix agent log, dmesg and memory usage statistic (you can provide memory graph (Latest data) from official linux template metrics and create one item directly on the host with proc.mem[,zabbix] key)

Regards, Elina

Also, do you monitor any logs?

This one may be related: ZBX-22943

Hi,

@Alex: No we do not monitor any logfiles

dmesg

log

We have the same problem with 6.0.19. Memory leak caused prod cluster to crash today. All cluster nodes were affected.

All servers with 6.0.19 are affected - bare metal and VPS. With plugins and without plugins.

I'm using Ubuntu 22.04.2LTS arm64 Version: 1:6.0.19-1+ubuntu22.04 from Version: 1:6.0.18-1+ubuntu22.04 and a steady memory leak occurs. Nothing in journal nor zabbix logs indicating any errors, warns or otherwise. After about a week it'll exhaust all memory. Once restarting zabbix-agent2 it's back to normal and again a memory leak (re)starts (Note: this server is a client running only zabbix-agent2). FYI. I do not use/have log monitoring configured.

root@stargate:~# apt list --installed zabbix*
Listing... Done
zabbix-agent2-plugin-mongodb/jammy,now 0:6.0.19-1+ubuntu22.04 arm64 [installed,automatic]
zabbix-agent2-plugin-postgresql/jammy,now 0:6.0.19-1+ubuntu22.04 arm64 [installed,automatic]
zabbix-agent2/jammy,now 1:6.0.19-1+ubuntu22.04 arm64 [installed]
zabbix-release/jammy,now 1:6.0-5+ubuntu22.04 all [installed]

(btw. love the product, it's fantastic! Keep up the great work).

Update 10/3:

Recent upgrade of zabbix-agent2 version (1:6.0.22-1+debian11) installed over (6.0.18-1+debian11) shows no signs of the previous memory leak and appears stable.

We downgraded all servers to version 6.0.18. Version 6.0.19 is unusable now.

I found a similar issue.

https://support.zabbix.com/si/jira.issueviews:issue-html/ZBX-23053/ZBX-23053.html

I don't know much about it, but it looks like you forgot X509_free in this function.

https://github.com/zabbix/zabbix/blob/master/src/go/pkg/tls/tls.go#L706

we face the same problem in 6.0.19 and .20 on linux windows works just fine

It has been tested on CentOS 7, Rocky 8, Rocky 9, openSUSE 15.5, and occurs on all clients running zabbix-agent2 using Encryption(cert).
Agent Version: 6.0.19, 6.0.20

 If you're using certificate-based encryption, it may be fixed in the next release.

https://support.zabbix.com/browse/ZBX-23221

Hello

trying the 6.0.21 right now and I will monitor it, thanks for the great job !

Gaël

As soon as 6.0.21 was released, it was installed on two servers and compared, but it seems that the memory of the server using the cert is subtly increasing little by little. Both servers are very lightweight servers that do the same job. We still have to wait and see, but below is the monitoring result after 12 hours.

using cert server (6.0.21)

using psk server (6.0.21)

using cert server (6.0.18)

On RHEL 8, I updated to 6.0.21 yesterday at lunch (freed almost 2GB) and since then, available memory looks pretty stable, comparing to previous version where we see it decreasing every hour

I will continue monitoring today, but it is looking good to me

In my environment it seems to be very stable after updating to 6.0.21.

If it is unclear whether the certificate-based encryption is the cause, I think it is better to create a new issue.

Distribution: Debian 11.7
Package: zabbix-agent2:amd64 (1:6.0.21-1+debian11)
Templates: Linux by Zabbix agent, Systemd by Zabbix agent 2

28 hours have elapsed. It certainly doesn't seem to have been fixed. Compare the pid with what I posted above. Servers that use encryption using certificates have steadily increased memory.

using cert server (6.0.21)

using psk server (6.0.21)

using cert server (6.0.18)

The problem was fixed with 6.0.21

Available memory graph:

Thank you for reporting the improvement.

SEUNGHYUN YI's report is also occurring in 6.0.18, so it would be better to create a new issue.

Initially tested on openSUSE, Rocky 8, Rocky 9, CentOS7. This has been happening since 6.0.19 and later.

Starting with the 6.0.21 release, we are only testing on Rocky 8. The three servers are shown below.

6.0.21 agent with certificate encryption, 6.0.21 agent with PSK encryption, 6.0.18 with certificate encryption.

What's important here is

1. Servers using the 6.0.18 agent also seem to have a memory leak, but the level is not high.
2. Servers using PSK encryption do not happen.
3. Servers using certificate encryption will happen unconditionally.

This is the result after 68 hours. Compare with the screenshot above.

using cert server (6.0.21) - I see a noticeable increase in memory.

using psk server (6.0.21) - no memory increase at all.

using cert server (6.0.18) - slightly increased memory.