Uploaded image for project: 'ZABBIX BUGS AND ISSUES'
  1. ZABBIX BUGS AND ISSUES
  2. ZBX-19088

VMWare cache fragmentation.

    XMLWordPrintable

    Details

    • Team:
      Team C
    • Sprint:
      Sprint 74 (Mar 2021), Sprint 75 (Apr 2021)
    • Story Points:
      2

      Description

      Steps to reproduce:
      Upgraded from 4.0 to 5.2. Proxy VMware Collector starts craches every N days with out of memory error:

       64203:20210303:130308.959 __mem_malloc: skipped 21896 asked 2050344 skip_min 256 skip_max 2049752
       64203:20210303:130308.959 [file:vmware.c,line:92] __zbx_mem_realloc(): out of memory (requested 2050344 bytes)
       64203:20210303:130308.959 [file:vmware.c,line:92] __zbx_mem_realloc(): please increase VMwareCacheSize configuration parameter
       64203:20210303:130308.963 === memory statistics for vmware cache size ===
       64203:20210303:130308.963 free chunks of size     24 bytes:       34
       64203:20210303:130308.963 free chunks of size     32 bytes:        3
       64203:20210303:130308.963 free chunks of size     40 bytes:       62
       64203:20210303:130308.963 free chunks of size     48 bytes:        7
       64203:20210303:130308.963 free chunks of size     56 bytes:        1
       64203:20210303:130308.963 free chunks of size     80 bytes:        1
       64203:20210303:130308.963 free chunks of size     96 bytes:        1
       64203:20210303:130308.963 free chunks of size    136 bytes:        1
       64203:20210303:130308.963 free chunks of size    176 bytes:        1
       64203:20210303:130308.963 free chunks of size >= 256 bytes:    21896
       64203:20210303:130308.963 min chunk size:         24 bytes
       64203:20210303:130308.963 max chunk size:    2049752 bytes
       64203:20210303:130308.963 memory of total size 1063324976 bytes fragmented into 651004 chunks
       64203:20210303:130308.963 of those,  996280440 bytes are in    22007 free chunks
       64203:20210303:130308.963 of those,   67044536 bytes are in   628997 used chunks
       64203:20210303:130308.963 of those,   10416048 bytes are used by allocation overhead
       64203:20210303:130308.963 ================================
       64203:20210303:130309.023 === Backtrace: ===
       64203:20210303:130309.024 12: /usr/sbin/zabbix_proxy: vmware collector #3 [updated 0, removed 0 VMware services in 0.000030 sec, querying VMware services](zbx_backtrace+0x42) [0x56316da01654]
       64203:20210303:130309.024 11: /usr/sbin/zabbix_proxy: vmware collector #3 [updated 0, removed 0 VMware services in 0.000030 sec, querying VMware services](__zbx_mem_realloc+0x169) [0x56316d9fc519]
       64203:20210303:130309.024 10: /usr/sbin/zabbix_proxy: vmware collector #3 [updated 0, removed 0 VMware services in 0.000030 sec, querying VMware services](+0x84008) [0x56316d96f008]
       64203:20210303:130309.024 9: /usr/sbin/zabbix_proxy: vmware collector #3 [updated 0, removed 0 VMware services in 0.000030 sec, querying VMware services](zbx_vector_ptr_reserve+0x57) [0x56316dae85d9]
       64203:20210303:130309.024 8: /usr/sbin/zabbix_proxy: vmware collector #3 [updated 0, removed 0 VMware services in 0.000030 sec, querying VMware services](+0x8836c) [0x56316d97336c]
       64203:20210303:130309.024 7: /usr/sbin/zabbix_proxy: vmware collector #3 [updated 0, removed 0 VMware services in 0.000030 sec, querying VMware services](+0x8fd9f) [0x56316d97ad9f]
       64203:20210303:130309.024 6: /usr/sbin/zabbix_proxy: vmware collector #3 [updated 0, removed 0 VMware services in 0.000030 sec, querying VMware services](vmware_thread+0x34f) [0x56316d97cfdc]
       64203:20210303:130309.024 5: /usr/sbin/zabbix_proxy: vmware collector #3 [updated 0, removed 0 VMware services in 0.000030 sec, querying VMware services](zbx_thread_start+0x37) [0x56316da06644]
       64203:20210303:130309.024 4: /usr/sbin/zabbix_proxy: vmware collector #3 [updated 0, removed 0 VMware services in 0.000030 sec, querying VMware services](MAIN_ZABBIX_ENTRY+0xd25) [0x56316d934712]
       64203:20210303:130309.024 3: /usr/sbin/zabbix_proxy: vmware collector #3 [updated 0, removed 0 VMware services in 0.000030 sec, querying VMware services](daemon_start+0x305) [0x56316da012a0]
       64203:20210303:130309.024 2: /usr/sbin/zabbix_proxy: vmware collector #3 [updated 0, removed 0 VMware services in 0.000030 sec, querying VMware services](main+0x30d) [0x56316d9339a3]
       64203:20210303:130309.024 1: /lib64/libc.so.6(__libc_start_main+0xf5) [0x7f9ba73c0545]
       64203:20210303:130309.024 0: /usr/sbin/zabbix_proxy: vmware collector #3 [updated 0, removed 0 VMware services in 0.000030 sec, querying VMware services](+0x47849) [0x56316d932849]
       64051:20210303:130309.243 One child process died (PID:64203,exitcode/signal:1). Exiting ...
       64051:20210303:130309.816 syncing history data...
       64051:20210303:130309.834 syncing history data... 100.000000%
       64051:20210303:130309.834 syncing history data done
       64051:20210303:130309.893 Zabbix Proxy stopped. Zabbix 5.2.4 (revision e9f58de09c).
      

      Configuration won't change. It looks like some fragmentation of cache. According the log there is plenty of space in cache free, but zabbix complains about run out of memory.

      Expected:
      No crashed as it was in 4.0.X

        Attachments

        1. screenshot-1.png
          screenshot-1.png
          188 kB
        2. screenshot-2.png
          screenshot-2.png
          279 kB
        3. ZBX-19088-5.2.patch
          9 kB
        4. ZBX-19088-524-dbg2.patch
          2 kB

          Activity

            People

            Assignee:
            MVekslers Michael Veksler
            Reporter:
            neogan Andrei Gushchin
            Votes:
            4 Vote for this issue
            Watchers:
            7 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved: