[ZBX-4848] Zabbix Agent Memory Leak Created: 2012 Apr 09  Updated: 2017 May 30  Resolved: 2013 Apr 10

Status: Closed
Project: ZABBIX BUGS AND ISSUES
Component/s: Agent (G)
Affects Version/s: 1.8.11
Fix Version/s: None

Type: Incident report Priority: Blocker
Reporter: Gioacchino Mazzurco Assignee: Unassigned
Resolution: Cannot Reproduce Votes: 0
Labels: memoryleak
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

Linux OpenWrt 2.6.32.27 #1 Thu Apr 5 15:58:38 CEST 2012 mips GNU/Linux



 Description   

I am trying zabbix as monitoring system on our community network in Italy ( ninux.org )

I am experiencing a big problem with the agent, our devices are embedded system with openwrt and ~16MB of ram so we have noticed rapidly of a memory leak on latest version of zabbix_agentd

As you can see there [0] the free memory on the device is slowly decreasing after starting zabbix agentd end grow up instantly when /etc/init.d/zabbix_agentd restart is runned

To be able to run Zabbix on an embedded system like openwrt we had to apply some little patches [1] but they doesn't seems the cause of the memory leak

[0] http://eigenlab.org/~gioacchino/Immagini/zabbix_agentd_memory_leak.png
[1] http://eigenlab.org/~gioacchino/zabbix/patches/

More info here:

root@OpenWrt:~# zabbix_agentd -V
Zabbix Agent (daemon) v1.8.11 (revision 26205) (20 March 2012)
Compilation time: Apr 5 2012 15:56:04

root@OpenWrt:~# ps ax
PID USER VSZ STAT COMMAND
1 root 1384 S init
2 root 0 SW [kthreadd]
3 root 0 SW [ksoftirqd/0]
4 root 0 SW [events/0]
5 root 0 SW [khelper]
8 root 0 SW [async/mgr]
39 root 0 SW [sync_supers]
41 root 0 SW [bdi-default]
43 root 0 SW [kblockd/0]
71 root 0 SW [kswapd0]
72 root 0 SW [aio/0]
73 root 0 SW [crypto/0]
85 root 0 SW [mtdblockd]
91 root 0 SW [ar71xx-spi]
225 root 0 SW [ipolldevd]
278 root 0 SWN [jffs2_gcd_mtd4]
293 root 1376 S /bin/sh /etc/init.d/rcS S boot
294 root 1384 S init
296 root 1376 S logger -s -p 6 -t sysinit
308 root 1388 S syslogd -C16
310 root 1368 S klogd
324 root 780 S /sbin/hotplug2 --override --persistent --set-worker /
432 root 0 SW [events_nrt]
434 root 0 SW [cfg80211]
441 root 0 SW [khubd]
511 root 0 SW [phy0]
561 root 0 SW [bat_events]
670 root 1456 S hostapd -P /var/run/wifi-phy0.pid -B /var/run/hostapd
848 root 1132 S /usr/sbin/dropbear -P /var/run/dropbear.1.pid -s -p 2
858 root 800 S /usr/bin/mini_snmpd -c public -i br-clients,wlan0-1 -
900 root 1380 S watchdog -t 5 /dev/watchdog
1424 root 1380 S /usr/sbin/ntpd -n -p 0.openwrt.pool.ntp.org -p 1.open
5043 zabbix 1584 S /usr/sbin/zabbix_agentd
5044 zabbix 1584 S /usr/sbin/zabbix_agentd
5045 zabbix 1844 S /usr/sbin/zabbix_agentd <--- JUST AFTER SOME MINUTER FROM RESTART
5046 zabbix 1592 S /usr/sbin/zabbix_agentd
5084 root 1204 S /usr/sbin/dropbear -P /var/run/dropbear.1.pid -s -p 2
5085 root 1388 S -ash
5140 root 1376 R ps ax

root@OpenWrt:~# cat /etc/zabbix/zabbix_agentd.conf

  1. This is config file for zabbix_agentd
  2. To get more information about ZABBIX, go http://www.zabbix.com
                        1. GENERAL PARAMETERS #################
  1. List of comma delimited IP addresses (or hostnames) of ZABBIX servers.
  2. No spaces allowed. First entry is used for sending active checks.
  3. Note that hostnames must resolve hostname->IP address and
  4. IP address->hostname.

Server=zabbix.eigenlab.org

  1. Server port for sending active checks

#ServerPort=10051

  1. Unique hostname. Required for active checks.

Hostname=localhost

  1. Listen port. Default is 10050

#ListenPort=10050

  1. IP address to bind agent
  2. If missing, bind to all available IPs

#ListenIP=127.0.0.1

  1. Number of pre-forked instances of zabbix_agentd.
  2. Default value is 5
  3. This parameter must be between 1 and 16

StartAgents=1

  1. How often refresh list of active checks. 2 minutes by default.

#RefreshActiveChecks=120

  1. Disable active checks. The agent will work in passive mode listening server.

#DisableActive=1

  1. Enable remote commands for ZABBIX agent. By default remote commands disabled.

#EnableRemoteCommands=1

  1. Specifies debug level
  2. 0 - debug is not created
  3. 1 - critical information
  4. 2 - error information
  5. 3 - warnings
  6. 4 - information (default)
  7. 5 - for debugging (produces lots of information)

DebugLevel=3

  1. Name of PID file

PidFile=/var/run/zabbix/zabbix_agentd.pid

  1. Name of log file.
  2. If not set, syslog will be used

LogFile=/var/log/zabbix/zabbix_agentd.log

  1. Spend no more than Timeout seconds on processing
  2. Must be between 1 and 30

Timeout=3

              1. USER-DEFINED MONITORED PARAMETERS #######
  1. Format: UserParameter=<key>,<shell command>
  2. Note that shell command must not return empty string or EOL only
    #UserParameter=system.test,who|wc -l
      1. Set of parameter for monitoring MySQL server (v3.23.42 and later)
      2. Change -u<username> and add -p<password> if required
        #UserParameter=mysql.ping,mysqladmin -uroot ping|grep alive|wc -l
        #UserParameter=mysql.uptime,mysqladmin -uroot status|cut -f2 -d":"|cut -f1 -d"T"
        #UserParameter=mysql.threads,mysqladmin -uroot status|cut -f3 -d":"|cut -f1 -d"Q"
        #UserParameter=mysql.questions,mysqladmin -uroot status|cut -f4 -d":"|cut -f1 -d"S"
        #UserParameter=mysql.slowqueries,mysqladmin -uroot status|cut -f5 -d":"|cut -f1 -d"O"
        #UserParameter=mysql.qps,mysqladmin -uroot status|cut -f9 -d":"
        #UserParameter=mysql.version,mysql -V


 Comments   
Comment by Gioacchino Mazzurco [ 2012 Apr 09 ]

here you can see the real time free memory graph http://zabbix.eigenlab.org/history.php?action=showgraph&itemid=22522&sid=2d7d93a558b68c6b

as you can see free memory is decreasing from latest zabbix_agentd restart

moreover i have a question why with ps ax i can see 4 agentd process if in config it is set to 1 ?

end here more updated info
root@OpenWrt:~# ps ax
PID USER VSZ STAT COMMAND
1 root 1384 S init
2 root 0 SW [kthreadd]
3 root 0 SW [ksoftirqd/0]
4 root 0 SW [events/0]
5 root 0 SW [khelper]
8 root 0 SW [async/mgr]
39 root 0 SW [sync_supers]
41 root 0 SW [bdi-default]
43 root 0 SW [kblockd/0]
71 root 0 SW [kswapd0]
72 root 0 SW [aio/0]
73 root 0 SW [crypto/0]
85 root 0 SW [mtdblockd]
91 root 0 SW [ar71xx-spi]
225 root 0 SW [ipolldevd]
278 root 0 SWN [jffs2_gcd_mtd4]
293 root 1376 S /bin/sh /etc/init.d/rcS S boot
294 root 1384 S init
296 root 1376 S logger -s -p 6 -t sysinit
308 root 1388 S syslogd -C16
310 root 1368 S klogd
324 root 780 S /sbin/hotplug2 --override --persistent --set-worker /
432 root 0 SW [events_nrt]
434 root 0 SW [cfg80211]
441 root 0 SW [khubd]
511 root 0 SW [phy0]
561 root 0 SW [bat_events]
670 root 1456 S hostapd -P /var/run/wifi-phy0.pid -B /var/run/hostapd
848 root 1132 S /usr/sbin/dropbear -P /var/run/dropbear.1.pid -s -p 2
858 root 800 S /usr/bin/mini_snmpd -c public -i br-clients,wlan0-1 -
900 root 1380 S watchdog -t 5 /dev/watchdog
1424 root 1380 S /usr/sbin/ntpd -n -p 0.openwrt.pool.ntp.org -p 1.open
5043 zabbix 1584 S /usr/sbin/zabbix_agentd
5044 zabbix 1584 S /usr/sbin/zabbix_agentd
5045 zabbix 4040 S /usr/sbin/zabbix_agentd <-------------------------- HERE IT COMES!!
5046 zabbix 1600 S /usr/sbin/zabbix_agentd <-------------------------- maybe this too but very slowly
5867 root 1204 S /usr/sbin/dropbear -P /var/run/dropbear.1.pid -s -p 2
5868 root 1384 S -ash
5875 root 1376 R ps ax
root@OpenWrt:~# date
Mon Apr 9 10:29:49 UTC 2012

Comment by Gioacchino Mazzurco [ 2012 Apr 09 ]

As someone suggested me on irc i have changed the config file in this way and restarted zabbix

root@OpenWrt:~# /etc/init.d/zabbix_agentd restart
root@OpenWrt:~# date
Mon Apr 9 11:18:44 UTC 2012
root@OpenWrt:~# cat /etc/zabbix/zabbix_agentd.conf

  1. This is config file for zabbix_agentd
  2. To get more information about ZABBIX, go http://www.zabbix.com
                        1. GENERAL PARAMETERS #################
  1. List of comma delimited IP addresses (or hostnames) of ZABBIX servers.
  2. No spaces allowed. First entry is used for sending active checks.
  3. Note that hostnames must resolve hostname->IP address and
  4. IP address->hostname.

Server=2001:1418:1a9:eeab::100

  1. Server port for sending active checks

#ServerPort=10051

  1. Unique hostname. Required for active checks.

Hostname=localhost

  1. Listen port. Default is 10050

#ListenPort=10050

  1. IP address to bind agent
  2. If missing, bind to all available IPs

#ListenIP=127.0.0.1

  1. Number of pre-forked instances of zabbix_agentd.
  2. Default value is 5
  3. This parameter must be between 1 and 16

StartAgents=1

  1. How often refresh list of active checks. 2 minutes by default.

#RefreshActiveChecks=120

  1. Disable active checks. The agent will work in passive mode listening server.

DisableActive=1

  1. Enable remote commands for ZABBIX agent. By default remote commands disabled.

#EnableRemoteCommands=1

  1. Specifies debug level
  2. 0 - debug is not created
  3. 1 - critical information
  4. 2 - error information
  5. 3 - warnings
  6. 4 - information (default)
  7. 5 - for debugging (produces lots of information)

DebugLevel=3

  1. Name of PID file

PidFile=/var/run/zabbix/zabbix_agentd.pid

  1. Name of log file.
  2. If not set, syslog will be used

LogFile=/var/log/zabbix/zabbix_agentd.log

  1. Spend no more than Timeout seconds on processing
  2. Must be between 1 and 30

Timeout=3

              1. USER-DEFINED MONITORED PARAMETERS #######
  1. Format: UserParameter=<key>,<shell command>
  2. Note that shell command must not return empty string or EOL only
    #UserParameter=system.test,who|wc -l
      1. Set of parameter for monitoring MySQL server (v3.23.42 and later)
      2. Change -u<username> and add -p<password> if required
        #UserParameter=mysql.ping,mysqladmin -uroot ping|grep alive|wc -l
        #UserParameter=mysql.uptime,mysqladmin -uroot status|cut -f2 -d":"|cut -f1 -d"T"
        #UserParameter=mysql.threads,mysqladmin -uroot status|cut -f3 -d":"|cut -f1 -d"Q"
        #UserParameter=mysql.questions,mysqladmin -uroot status|cut -f4 -d":"|cut -f1 -d"S"
        #UserParameter=mysql.slowqueries,mysqladmin -uroot status|cut -f5 -d":"|cut -f1 -d"O"
        #UserParameter=mysql.qps,mysqladmin -uroot status|cut -f9 -d":"
        #UserParameter=mysql.version,mysql -V
Comment by Gioacchino Mazzurco [ 2012 Apr 09 ]

added zabbix_agentd used memory graph

http://zabbix.eigenlab.org/history.php?action=showgraph&itemid=22525&sid=cb8ad6e906416fd5

Comment by Gioacchino Mazzurco [ 2012 Apr 09 ]

It seems that removing system.hostname ( that replies ZBX_NOTSUPPORTED ) and system.uname ( that replies the correct info ) from zabbix server item list, memory used by zabbix_agentd stop growing on the embedded system

Comment by Gioacchino Mazzurco [ 2012 Apr 09 ]

my bad it doesn't stopped but slowed

Comment by Alexei Vladishev [ 2012 Jun 13 ]

Please include at least one evidence that the agent is leaking memory. Free memory usage is irrelevant information. It would be nice if you could upgrade to the latest 1.8.13.

Comment by richlv [ 2013 Feb 07 ]

please reopen if you have additional information

Comment by Oleksii Zagorskyi [ 2013 Apr 10 ]

Reopen to unify labels

Generated at Tue May 20 07:26:42 EEST 2025 using Jira 9.12.4#9120004-sha1:625303b708afdb767e17cb2838290c41888e9ff0.