[ZBX-8506] wmi.get on Windows agent causes memory leak Created: 2014 Jul 21  Updated: 2017 May 30  Resolved: 2014 Dec 04

Status: Closed
Project: ZABBIX BUGS AND ISSUES
Component/s: Agent (G)
Affects Version/s: 2.2.5
Fix Version/s: 2.2.8rc1, 2.4.3rc1, 2.5.0

Type: Incident report Priority: Critical
Reporter: Glenn Matthys Assignee: Unassigned
Resolution: Fixed Votes: 1
Labels: memoryleak, patch, wmi
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

Windows Server 2012


Attachments: Text File wmi.cpp     File zabbix-memory.ps1     File zabbix_agentd.exe_140724_202202.dmp    
Issue Links:
Duplicate

 Description   

Each time calling wmi.get, the zabbix agentd process leaks memory. After a while this memory leak grows into hundreds of megabytes. Seeing zabbix agentd processes hanging around using up to 800 MB of RAM is not uncommon.

To verify this, I've written a powershell script that repeatedly calls wmi.get 4 times (see attachment zabbix-memory.ps1). After 100 iterations of this, memory usage of the agentd process has increased by almost 1 MB (Start: 7392, end: 8288 kb, delta: 896 kb).



 Comments   
Comment by Glenn Matthys [ 2014 Jul 24 ]

Memory dump after a clean zabbix_agentd start and running the powershell script (100 iterations of 4 times wmi.get) (Powershell script result: Start: 4400, end: 5824 kb, delta: 1424 kb)

Comment by Glenn Matthys [ 2014 Oct 26 ]

Tested on 2.4.1, issue still exists. (Start: 4296, end: 5644 kb, delta: 1348 kb)

Comment by Glenn Matthys [ 2014 Nov 18 ]

Can someone take a look at this? We can't deploy the agent until this is fixed...

Comment by Pedro Nunes [ 2014 Nov 20 ]

Hi,

I looked at the code and there are at least 1 pointer that is never released (pclsObj).
Think thats the source of the memory leak.

I spend some time and rebuild completely the source code of wmi.cpp.
Also it now supports WMI arrays. Very usefull to get HP operational status info:
wmi.get[root\hpq,SELECT OperationalStatus FROM HP_ComputerSystem]

Attached the 2 files. My wmi.cpp and the 32 bits agent already compiled.

Can someone test if the memory leak is still there?

Comment by Marc [ 2014 Nov 20 ]

pintonunes, can you add your modifications in form of a patch/unidiff please?

Comment by Pedro Nunes [ 2014 Nov 20 ]

Marc, just replace the current wmi.cpp with the one in attach and compile.
If you really need the patch file let me know.

Comment by Marc [ 2014 Nov 20 ]

pintonunes, it's just good practice and is easier to merge for devs - especially when merging with a different/newer code base.

Comment by Pedro Nunes [ 2014 Nov 21 ]

Marc, i will.. just finishing the all code. Add log lines, etc etc..
I am running this agent for 2 days in about 10 servers and the agent stabilized at 11Mb of RAM usage.

Comment by Pedro Nunes [ 2014 Nov 25 ]

Marc, attached the last wmi.cpp. The other one had some bugs.
Unfortunately i dont know how to build a patch file with visual studio 2010 express.
But like i said, its just a matter of replacing the current wmi.cpp file.

Comment by Aleksandrs Saveljevs [ 2014 Nov 26 ]

Pedro, your fix in wmi.cpp does two things: (a) it fixes a memory leak and (b) introduces support for arrays. Could you please add your array patch to ZBXNEXT-2459, but reattach wmi.cpp here that only fixes the memory leak in the original code? The memory leak fix can be merged rather quickly, while the array support has to be extensively tested and is a slower process.

Comment by Pedro Nunes [ 2014 Nov 26 ]

Attached the wmi.cpp without array support.
The code is actually pretty diferent from the original one.
2 things about it:

  • extern "C" void zbx_co_uninitialize(){} is not needed but you need to remove it from zabbix_agentd.c
  • static CoCreateInstanceGuard ccig; is static so WMI is initialized when agent starts which makes it dependent on the WMI windows service. If WMI service restarts you need to restart zabbix agent also. It can be changed to non static easly.

Regards

Comment by Aleksandrs Saveljevs [ 2014 Nov 26 ]

Pedro, I have narrowed the fix regarding the memory leak in your patch to just the following:

$ svn di -c 50877
Index: src/libs/zbxsysinfo/win32/wmi.cpp
===================================================================
--- src/libs/zbxsysinfo/win32/wmi.cpp   (revision 50876)
+++ src/libs/zbxsysinfo/win32/wmi.cpp   (revision 50877)
@@ -216,6 +216,9 @@
 out:
        VariantClear(&vtProp);
 
+       if (0 != pclsObj)
+               pclsObj->Release();
+
        if (0 != pEnumerator)
                pEnumerator->Release();

This seems to fix the leak, which is the main topic in this issue. If you feel there is something else to fix like not properly handling WMI service restarts, please register a new ZBX.

Comment by Aleksandrs Saveljevs [ 2014 Nov 26 ]

Memory leak fix is available in development branch svn://svn.zabbix.com/branches/dev/ZBX-8506 .

Comment by Pedro Nunes [ 2014 Nov 26 ]

Yes, that the obvious fix using the existing code.
COM objects should use RAII to create and release resources, etc etc...
Anyway, thanks for the fix

Comment by Andris Zeila [ 2014 Dec 04 ]

Unfortunately ATL is not available in Visual Studio Express editions or Windows SDK. So we had to do it the hard way.

Comment by Andris Zeila [ 2014 Dec 04 ]

Successfully tested

Comment by Aleksandrs Saveljevs [ 2014 Dec 04 ]

Fixed in pre-2.2.8 r51003, pre-2.4.3 r51004, and pre-2.5.0 (trunk) r51005.

Comment by Aleksandrs Saveljevs [ 2014 Dec 04 ]

Issue regarding WMI service restarts has been reported separately in ZBX-9112.

Comment by Aleksandrs Saveljevs [ 2014 Dec 05 ]

Pedro, as stated in ZBX-9112, problem with WMI service restarts was not observed. If you have any information on how to reproduce the issue, please add a comment to ZBX-9112.

Comment by Marc [ 2014 Dec 23 ]

pintonunes, regarding:
Unfortunately i dont know how to build a patch file with visual studio 2010 express.

I don't know whether there is a build-in functionality to create proper patches in Visual Studio 2010 Express, but you could give it a try to do it the UNIX way.
See: http://sourceforge.net/projects/unxutils/

C:\TEMP\zabbix-2.2.7>diff.exe -u src\libs\zbxsysinfo\win32\wmi.cpp.orig src\libs\zbxsysinfo\win32\wmi.cpp > zbx8506.patch
Comment by Stefan Bauer [ 2015 Feb 12 ]

When will be zabbix agent 2.5 available for windows?

Generated at Tue Apr 23 19:55:05 EEST 2024 using Jira 9.12.4#9120004-sha1:625303b708afdb767e17cb2838290c41888e9ff0.