[ZBX-20586] Memory leak in zabbix agent2 Created: 2022 Feb 16  Updated: 2024 Apr 10  Resolved: 2022 Jul 13

Status: Closed
Project: ZABBIX BUGS AND ISSUES
Component/s: Agent (G)
Affects Version/s: 5.4.10, 6.0.0
Fix Version/s: 5.0.26rc1, 6.0.7rc1, 6.2.1rc1, 6.4.0alpha1, 6.4 (plan)

Type: Problem report Priority: Major
Reporter: Kalchenko Oleksandr Assignee: Andris Zeila
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: PNG File image-2022-02-16-10-01-58-184.png     PNG File image-2022-02-16-10-05-43-622.png    
Team: Team A
Sprint: Sprint 89 (Jun 2022), Sprint 90 (Jul 2022)
Story Points: 0.25

 Description   

Env:

Zabbix agent2 5.4.10/6.0.0

Windows 2016\2019, Gentoo Linux, Debian 10

Passive agent check mainly used

Steps to reproduce:

  1. Upgraded zabbix agent (5.4.10) to zabbix agent2 (5.4.10) on ~600 servers.
  2. We've started to see memory leak in zabbix agent2 on some (<10) hosts. Hosts with problem are on different OS (Windows 2016\Windows 2019\Gentoo Linux\Debian).
  3. Reinstalled some windows hosts from scratch - memory leak still exists. 
  4. With help of the libleak we've identified, that leak probably exists in libcrypto.so (linux version).
  5. Disabled agent encryption on problem hosts (from server to agent) - memory leak stops to appear.

Some screens with memory leak:

Portion of libleak output:

 Comments   
Comment by Andris Mednis [ 2022 Feb 22 ]

Observed with Agent2 6.0.0rc1:

  • 1 passive item
  • polled every second
  • PSK:

tls_new_server() called
tls_new() called
tls_psk_server_cb() called
but somehow neither tls_free_context() nor tls_free() are called.

wiper: Contexts are cached during package initialization, so tls_free_context is not supposed to be called (unless the package is re-initialized).
tls_free is called when garbage collector destroys client/server objects, which apparently doesn't happen often enough as GO has no idea about C memory allocation. When manually triggered with runtime.GC() the objects are destroyed and tls resources are freed properly.
We should try manually free tls resources when connection is closed. If that is not possible, then we could try triggering garbage collection, but could have performance impact.

Comment by Kalchenko Oleksandr [ 2022 Jun 21 ]

Somebody help?

Issue still exists in latest 6.0.5

Comment by Andris Zeila [ 2022 Jul 13 ]

Released ZBX-20586 in:

  • pre-5.0.26rc1 8311ef85496
  • pre-6.0.7rc1 c9f65d8517a
  • pre-6.2.1rc1 f7d332853d7
  • pre-6.4.0alpha1 20515991147
Generated at Fri Jun 13 04:57:26 EEST 2025 using Jira 9.12.4#9120004-sha1:625303b708afdb767e17cb2838290c41888e9ff0.