Uploaded image for project: 'ZABBIX BUGS AND ISSUES'
  1. ZABBIX BUGS AND ISSUES
  2. ZBX-4602

Memory leak in the poller processes

XMLWordPrintable

    • Icon: Incident report Incident report
    • Resolution: Cannot Reproduce
    • Icon: Blocker Blocker
    • None
    • 1.8.10
    • Server (S)
    • OS: CentOS release 5.5 (Final)
       Zabbix server 1.8.10 (revision 24303)
       net-snmp version 5.7.1

      We have issue with memory allocation of zabbix_server poller processes.
      Server is configured to start 5 pollers. Memory utilization is constantly growing and for 10 days
      it reach over 1GB. History graph is attached. Once it goes over 1GB, we are restarting zabbix_server (drop visible in graph).
      Without restart, usage grows until OS runs out of memory and start swapping heavily.

      Version info:
      Zabbix server 1.8.10 (revision 24303), OS: CentOS release 5.5 (Final), net-snmp version 5.7.1
      History graph, ps -aux output, and server config attached. Detailed information in the file "zabbix_server_mem.txt"

      Attached output of these commands:

      1. for i in 7320 7321 7322 7323 7324 ; do pmap -d $i; echo; done > pollers_pmap_d.txt
      2. for i in 7320 7321 7322 7323 7324 ; do pmap -x $i; echo; done > pollers_pmap_x.txt
      3. for i in 7320 7321 7322 7323 7324 ; do pmap $i; echo; done > pollers_pmap.txt

      Memory leak is monitored since begining of January, but I believe it was there for couple of months before causing performance issues.
      I cannot relate it to zabbix upgrades.
      Two weeks ago, zabbix server config is changed, cache size is reduced from 768M to 256M, but that didn't change memory utilization.
      Received sum RSS size is multiplied by 1024 in item configuration (item definition attached)
      Maximum value for this item in hystory is 1GB, because we use monit to restart zabbix server once it reach 1GB.
      Before January, server had 2GB RAM so 1GB for zabbix_server was a lot (this box is running mysql and apache frontend also).
      Now we have 4GB RAM (3546 MB reported by OS, 32 bit), I will disable automatic restart and monitor it manually.

        1. memHistory.png
          memHistory.png
          35 kB
        2. oleksiy's_v_pre1.8.11_leak.png
          oleksiy's_v_pre1.8.11_leak.png
          44 kB
        3. oleksiy's_v1.9.9_leak.png
          oleksiy's_v1.9.9_leak.png
          44 kB
        4. oleksiy's_v1.9.9_with_pause_6hours.png
          oleksiy's_v1.9.9_with_pause_6hours.png
          26 kB
        5. pollers_pmap_d.txt
          30 kB
        6. pollers_pmap_x.txt
          0.9 kB
        7. pollers_pmap.txt
          21 kB
        8. UserP.procmem[zabbix_server].gif
          UserP.procmem[zabbix_server].gif
          44 kB
        9. v1.9.9_single_item.png
          v1.9.9_single_item.png
          29 kB
        10. zabbix_rss_after_update_and_upgrade.png
          zabbix_rss_after_update_and_upgrade.png
          49 kB
        11. zabbix_server_mem.txt
          7 kB
        12. zabbix_server.conf
          4 kB
        13. zabbix_virtmem_graph_after_update_and_upgrade.png
          zabbix_virtmem_graph_after_update_and_upgrade.png
          39 kB
        14. zbx_reg_proc_mem4.sh
          0.7 kB

            Unassigned Unassigned
            zalex_ua Oleksii Zagorskyi
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

              Created:
              Updated:
              Resolved: