[ZBX-14842] Zabbix server stoped ====== Fatal information: ====== 2129:20180908:065054.307 Program counter: 0x55624c90d5d1 Created: 2018 Sep 12  Updated: 2019 Jan 11  Resolved: 2019 Jan 11

Status: Closed
Project: ZABBIX BUGS AND ISSUES
Component/s: Server (S)
Affects Version/s: 3.4.13
Fix Version/s: None

Type: Incident report Priority: Critical
Reporter: Camilo Camacho Assignee: Aigars Kadikis
Resolution: Fixed Votes: 2
Labels: performance, usability
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

Server information Infraestruccture

Number of hosts (enabled/disabled/templates) 302 187 / 14 / 101
Number of items (enabled/disabled/not supported) 44135 29733 / 3503 / 10899
Number of triggers (enabled/disabled [problem/ok]) 14691 13298 / 1393 [7 / 13291]
Number of users (online) 10 5
Required server performance, new values per second 336.98

1 Server frontend
zabbix server
4 VCPU
12 GB

1 Server Database
Mariadb
size DB 271 GB

Number vcenter monitor=4
Host esxi hosts menber adminitrator vcenters=40

Config parameters zabbix_server.config

LogFile=/var/log/zabbix/zabbix_server.log
LogFileSize=0
DebugLevel=3
PidFile=/var/run/zabbix/zabbix_server.pid
SocketDir=/var/run/zabbix
StartPollers=20
StartPollersUnreachable=10
StartTrappers=1
StartPingers=8

StartVMwareCollectors=40
VMwareFrequency=120
VMwarePerfFrequency=120
VMwareCacheSize=2G
VMwareTimeout=180
StartSNMPTrapper=1

ListenIP=0.0.0.0

HousekeepingFrequency=0

CacheSize=96M
StartDBSyncers=5
HistoryCacheSize=32M
HistoryIndexCacheSize=8M
TrendCacheSize=16M
ValueCacheSize=32M
Timeout=10
AlertScriptsPath=/usr/lib/zabbix/alertscripts
ExternalScripts=/usr/lib/zabbix/externalscripts

Attachments: PNG File Ccahe Effectiveness.PNG     PNG File Gathering Process.PNG     PNG File Queue.PNG     PNG File Zabbix Internal process.PNG     PNG File screenshot-1.png    

 Description   

This is menssage zabbix server log,

 

  2076:20180908:065054.228 __mem_malloc: skipped 0 asked 168 skip_min 4294967295 skip_max 0
  2076:20180908:065054.228 [file:vmware.c,line:87] zbx_mem_malloc(): out of memory (requested 165 bytes)
  2076:20180908:065054.228 [file:vmware.c,line:87] zbx_mem_malloc(): please increase VMwareCacheSize configuration parameter
  2129:20180908:065054.306 Got signal [signal:11(SIGSEGV),reason:128,refaddr:(nil)]. Crashing ...
  2129:20180908:065054.306 ====== Fatal information: ======
  2129:20180908:065054.307 Program counter: 0x55624c90d5d1
  2129:20180908:065054.307 === Registers: ===
  2129:20180908:065054.307 r8      =             ffff =                65535 =                65535
  2129:20180908:065054.307 r9      =                1 =                    1 =                    1
  2129:20180908:065054.307 r10     =     7fafe6c66a80 =      140393467767424 =      140393467767424
  2129:20180908:065054.307 r11     =              213 =                  531 =                  531
  2129:20180908:065054.307 r12     =     55624c9c2770 =       93880680458096 =       93880680458096
  2129:20180908:065054.307 r13     =                d =                   13 =                   13
  2129:20180908:065054.307 r14     =                0 =                    0 =                    0
  2129:20180908:065054.307 r15     =                0 =                    0 =                    0
  2129:20180908:065054.307 rdi     =     7ffe43b29120 =      140730034196768 =      140730034196768
  2129:20180908:065054.307 rsi     =     7ffe43b29120 =      140730034196768 =      140730034196768
  2129:20180908:065054.307 rbp     =     7ffe43b29100 =      140730034196736 =      140730034196736
  2129:20180908:065054.307 rbx     =                d =                   13 =                   13
  2129:20180908:065054.307 rdx     =     7ffe43b29120 =      140730034196768 =      140730034196768
  2129:20180908:065054.307 rax     = 8000000000000018 =  9223372036854775832 = -9223372036854775784
  2129:20180908:065054.307 rcx     =     55624dc3cca0 =       93880699833504 =       93880699833504
  2129:20180908:065054.307 rsp     =     7ffe43b290e0 =      140730034196704 =      140730034196704
  2057:20180908:065054.307 One child process died (PID:2076,exitcode/signal:1). Exiting ...
  2129:20180908:065054.307 rip     =     55624c90d5d1 =       93880679716305 =       93880679716305
  2129:20180908:065054.307 efl     =            10206 =                66054 =                66054
  2129:20180908:065054.307 csgsfs  =               33 =                   51 =                   51
  2129:20180908:065054.307 err     =                0 =                    0 =                    0
  2129:20180908:065054.307 trapno  =                d =                   13 =                   13
  2129:20180908:065054.307 oldmask =                0 =                    0 =                    0
  2129:20180908:065054.307 cr2     =                0 =                    0 =                    0
  2129:20180908:065054.307 === Backtrace: ===
  2057:20180908:065056.313 syncing history data...
  2057:20180908:065056.320 syncing history data done
  2057:20180908:065056.320 syncing trend data...
  2057:20180908:065059.371 syncing trend data done
  2057:20180908:065102.049 Zabbix Server stopped. Zabbix 3.4.13 (revision 84139).


 Comments   
Comment by Camilo Camacho [ 2018 Sep 12 ]

Frequently the service of Zabbix server stops, indicating a problem with the VMwareCacheSize, despite having the maximum value allowed, and that is not evident in the graphics of performance that is used the totality of assigned cache (use cache size) however Still presented the crash, this occasioned unavailability of monitoring in the company

Comment by Vladislavs Sokurenko [ 2018 Sep 13 ]

Did it start after some version ?

Comment by Camilo Camacho [ 2018 Sep 13 ]

Not, this problem is present version 3.4.10 too, for this problem update version to 3.4.13, and problem is present.

Comment by Aigars Kadikis [ 2018 Sep 20 ]

Hello, Camilo!

Please look closely inside log file about "increase" keyword there may be other pointers too:

grep increase /var/log/zabbix/zabbix_server.log

 if the daemon is stopped make sure all subprocesses are gone:

systemctl stop zabbix-server
ps aux|grep [z]abbix_server

Only then start a fresh instance. Let us know how it goes.

Regards,

 

Comment by Camilo Camacho [ 2018 Sep 20 ]

Hello, Aigars

1. This is result execute command
grep increase /var/log/zabbix/zabbix_server.log
 

 13825:20180916:121215.075 __mem_malloc: skipped 0 asked 160 skip_min 4294967295 skip_max 0
 13825:20180916:121215.075 file:vmware.c,line:87 zbx_mem_malloc(): out of memory (requested 157 bytes)
 13825:20180916:121215.075 file:vmware.c,line:87 zbx_mem_malloc(): please increase VMwareCacheSize configuration parameter
 13796:20180916:121215.157 value cache is fully used: please increase ValueCacheSize configuration parameter
 13783:20180916:121215.169 One child process died (PID:13825,exitcode/signal:1). Exiting ...
 13783:20180916:121217.181 syncing history data...
 13783:20180916:121217.190 syncing history data done
 13783:20180916:121217.190 syncing trend data...
 13783:20180916:121219.523 syncing trend data done
 13783:20180916:121222.171 Zabbix Server stopped. Zabbix 3.4.13 (revision 84139).
 78366:20180916:121232.629 Starting Zabbix Server. Zabbix 3.4.13 (revision 84139).
-------------------------------------------------------------------------------------------------------------------------
 78460:20180917:024931.166 cannot send list of active checks to "10.3.2.95": host [TM520002095] not found
 78395:20180917:024933.468 __mem_malloc: skipped 673 asked 71289128 skip_min 264 skip_max 18283272
 78395:20180917:024933.468 file:vmware.c,line:87 zbx_mem_realloc(): out of memory (requested 71289128 bytes)
 78395:20180917:024933.468 file:vmware.c,line:87 zbx_mem_realloc(): please increase VMwareCacheSize configuration parameter
 78366:20180917:024933.555 One child process died (PID:78395,exitcode/signal:1). Exiting ...
 78366:20180917:024935.563 syncing history data...
 78366:20180917:024935.582 item "00000000-0000-0000-0000-00000000003d:vmware.hv.datastore.size[\{$URL},\{HOST.HOST},COBGVM_BOGTM01_C1_DATASTORE05_RDP_FC_15K,free]" became not supported: Unknown hypervisor uuid.
 78366:20180917:024935.596 syncing history data done
 78366:20180917:024935.596 syncing trend data...
 78366:20180917:024938.111 syncing trend data done
 78366:20180917:024940.623 Zabbix Server stopped. Zabbix 3.4.13 (revision 84139).
---------------------------------------------------------------------------------------------------------------------------
  2404:20180917:212817.951 cannot send list of active checks to "10.3.2.95": host [TM520002095] not found
  2341:20180917:212854.240 __mem_malloc: skipped 0 asked 24 skip_min 4294967295 skip_max 0
  2341:20180917:212854.240 file:vmware.c,line:87 zbx_mem_malloc(): out of memory (requested 24 bytes)
  2341:20180917:212854.240 file:vmware.c,line:87 zbx_mem_malloc(): please increase VMwareCacheSize configuration parameter
  2311:20180917:212854.334 One child process died (PID:2341,exitcode/signal:1). Exiting ...
  2311:20180917:212856.345 syncing history data...
  2311:20180917:212856.373 syncing history data done
  2311:20180917:212856.373 syncing trend data...
  2311:20180917:212858.124 syncing trend data done
  2311:20180917:212900.797 Zabbix Server stopped. Zabbix 3.4.13 (revision 84139).
 48754:20180917:212911.379 Starting Zabbix Server. Zabbix 3.4.13 (revision 84139).
-----------------------------------------------------------------------------------------------------------------------------
111455:20180919:103807.012 cannot send list of active checks to "10.3.2.95": host [TM520002095] not found
111391:20180919:103836.503 __mem_malloc: skipped 0 asked 24 skip_min 4294967295 skip_max 0
111391:20180919:103836.503 file:vmware.c,line:87 zbx_mem_malloc(): out of memory (requested 24 bytes)
111391:20180919:103836.503 file:vmware.c,line:87 zbx_mem_malloc(): please increase VMwareCacheSize configuration parameter
111365:20180919:103836.590 One child process died (PID:111391,exitcode/signal:1). Exiting ...
111365:20180919:103838.599 syncing history data...
111365:20180919:103838.599 syncing history data done
111365:20180919:103838.599 syncing trend data...
111365:20180919:103845.009 syncing trend data done
111365:20180919:103847.795 Zabbix Server stopped. Zabbix 3.4.13 (revision 84139).
  1135:20180919:114300.515 Starting Zabbix Server. Zabbix 3.4.13 (revision 84139).
-----------------------------------------------------------------------------------------------------------------------------
]
  1397:20180920:030749.965 __mem_malloc: skipped 586 asked 248031720 skip_min 256 skip_max 222534240
  1397:20180920:030749.966 file:vmware.c,line:87 zbx_mem_realloc(): out of memory (requested 248031720 bytes)
  1397:20180920:030749.966 file:vmware.c,line:87 zbx_mem_realloc(): please increase VMwareCacheSize configuration parameter
  1434:20180920:030750.041 Got signal [signal:11(SIGSEGV),reason:128,refaddr:(nil)]. Crashing ...
  1135:20180920:030750.041 One child process died (PID:1397,exitcode/signal:1). Exiting ...
  1434:20180920:030750.089 ====== Fatal information: ======
  1135:20180920:030752.051 syncing history data...
  1135:20180920:030752.057 syncing history data done
  1135:20180920:030752.057 syncing trend data...
  1135:20180920:030752.954 syncing trend data done
---------------------------------------------------------------------------------------------------------------------------
 51046:20180920:180425.819 __mem_malloc: skipped 479 asked 71289128 skip_min 256 skip_max 14647488
 51046:20180920:180425.819 file:vmware.c,line:87 zbx_mem_realloc(): out of memory (requested 71289128 bytes)
 51046:20180920:180425.819 file:vmware.c,line:87 zbx_mem_realloc(): please increase VMwareCacheSize configuration parameter
 51022:20180920:180425.917 One child process died (PID:51046,exitcode/signal:1). Exiting ...
 51022:20180920:180427.925 syncing history data...
 51022:20180920:180427.929 item "00000000-0000-0000-0000-00000000001e:vmware.hv.datastore.size[\{$URL},\{HOST.HOST},COBGVM_BOGTM01_C1_DATASTORE18_RDP_FC_15K,free]" became not supported: Unknown hypervisor uuid.
 51022:20180920:180427.933 syncing history data done
 51022:20180920:180427.933 syncing trend data...
 51022:20180920:180428.583 syncing trend data done
 51022:20180920:180431.157 Zabbix Server stopped. Zabbix 3.4.13 (revision 84139).

  51046:20180920:180425.819 __mem_malloc: skipped 479 asked 71289128 skip_min 256 skip_max 14647488
 51046:20180920:180425.819 file:vmware.c,line:87 zbx_mem_realloc(): out of memory (requested 71289128 bytes)
 51046:20180920:180425.819 file:vmware.c,line:87 zbx_mem_realloc(): please increase VMwareCacheSize configuration parameter
 51022:20180920:180425.917 One child process died (PID:51046,exitcode/signal:1). Exiting ...
 51022:20180920:180427.925 syncing history data...
 51022:20180920:180427.929 item "00000000-0000-0000-0000-00000000001e:vmware.hv.datastore.size[\{$URL},\{HOST.HOST},COBGVM_BOGTM01_C1_DATASTORE18_RDP_FC_15K,free]" became not supported: Unknown hypervisor uuid.
 51022:20180920:180427.933 syncing history data done
 51022:20180920:180427.933 syncing trend data...
 51022:20180920:180428.583 syncing trend data done
 51022:20180920:180431.157 Zabbix Server stopped. Zabbix 3.4.13 (revision 84139).

2. I verify all subprocesses are gone but subprocesses vmware collector is running, I have kill process id for start zabbix server.

 
ps aux|grep [z]abbix_server
ps aux|grep [z]abbix_server
zabbix    51074 45.6 19.7 5187180 2399268 ?     S    03:08 415:13 /usr/sbin/zabbix_server: vmware collector #34 [updated 0, removed 0 VMware services in 0.000003 sec, q

Comment by Vladislavs Sokurenko [ 2018 Sep 21 ]

Are you using 32 bit or 64 bit Zabbix server ?
Does downgrading help ?
Please also monitor vmware cache, is there something interesting ?
zabbix[vmware,buffer,<mode>]
https://www.zabbix.com/documentation/4.0/manual/config/items/itemtypes/internal

Comment by Donovan De Beuckelaer [ 2018 Sep 21 ]

I'm having the same Out of memory problem on 64bit 3.4.13 on a RH7.

I think it started @3.4.12

vmware cache was 99% free.

With a total crash on every entry, i set StartVMwareCollectors=0, it was set to 4, to keep server running.

There was one VMwareCollector process not killed, kill -9 was needed to restart zabbix server.

$grep increase /var/log/zabbix/zabbix_server.log*

 

/var/log/zabbix/zabbix_server.log: 45797:20180917:151509.774 file:vmware.c,line:87 zbx_mem_realloc(): please increase VMwareCacheSize configuration parameter
 /var/log/zabbix/zabbix_server.log-20180916: 7917:20180913:125546.640 file:vmware.c,line:87 zbx_mem_malloc(): please increase VMwareCacheSize configuration parameter
 /var/log/zabbix/zabbix_server.log-20180916: 39584:20180913:184708.366 file:vmware.c,line:87 zbx_mem_malloc(): please increase VMwareCacheSize configuration parameter
 /var/log/zabbix/zabbix_server.log-20180916: 56367:20180914:003853.054 file:vmware.c,line:87 zbx_mem_malloc(): please increase VMwareCacheSize configuration parameter
 /var/log/zabbix/zabbix_server.log-20180916: 30513:20180914:063058.060 file:vmware.c,line:87 zbx_mem_malloc(): please increase VMwareCacheSize configuration parameter
 /var/log/zabbix/zabbix_server.log-20180916: 65106:20180914:152111.902 file:vmware.c,line:87 zbx_mem_malloc(): please increase VMwareCacheSize configuration parameter
 /var/log/zabbix/zabbix_server.log-20180916: 38305:20180914:213839.433 file:vmware.c,line:87 zbx_mem_realloc(): please increase VMwareCacheSize configuration parameter
 /var/log/zabbix/zabbix_server.log-20180916: 65115:20180915:033315.145 file:vmware.c,line:87 zbx_mem_realloc(): please increase VMwareCacheSize configuration parameter
 /var/log/zabbix/zabbix_server.log-20180916: 29750:20180915:092243.517 file:vmware.c,line:87 zbx_mem_realloc(): please increase VMwareCacheSize configuration parameter
 /var/log/zabbix/zabbix_server.log.old: 60682:20180911:101824.784 file:vmware.c,line:87 zbx_mem_malloc(): please increase VMwareCacheSize configuration parameter
 /var/log/zabbix/zabbix_server.log.old: 36146:20180911:200319.997 file:vmware.c,line:87 zbx_mem_malloc(): please increase VMwareCacheSize configuration parameter
 /var/log/zabbix/zabbix_server.log.old: 10451:20180912:015528.626 file:vmware.c,line:87 zbx_mem_malloc(): please increase VMwareCacheSize configuration parameter
 /var/log/zabbix/zabbix_server.log.old: 58503:20180912:074712.217 file:vmware.c,line:87 zbx_mem_malloc(): please increase VMwareCacheSize configuration parameter
 /var/log/zabbix/zabbix_server.log.old: 3454:20180912:192406.000 file:vmware.c,line:87 zbx_mem_malloc(): please increase VMwareCacheSize configuration parameter
 /var/log/zabbix/zabbix_server.log.old: 8989:20180913:011317.799 file:vmware.c,line:87 zbx_mem_malloc(): please increase VMwareCacheSize configuration parameter
 /var/log/zabbix/zabbix_server.log.old: 14315:20180913:065727.087 file:vmware.c,line:87 zbx_mem_malloc(): please increase VMwareCacheSize configuration parameter

 

 

Comment by Camilo Camacho [ 2018 Sep 21 ]

Are you using 32 bit or 64 bit Zabbix server ?
CentOS Linux release 7.5.1804 (Core)
Linux zbx-bog-inf-wi.mon.o4it.loc 3.10.0-862.11.6.el7.x86_64 #1 SMP Tue Aug 14 21:49:04 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

Does downgrading help ?

The issue is present also version 3.4.12, present issue vmware.hv.datastore.size returns "became not supported: Unsupported item key" when getting data directly from ESXi node

Please also monitor vmware cache, is there something interesting ?

The graphic´s behavior free cache is 80 %

zabbix[vmware,buffer,<mode>]
https://www.zabbix.com/documentation/4.0/manual/config/items/itemtypes/internal

Comment by Vladislavs Sokurenko [ 2018 Sep 21 ]

Currently it’s not possible to reproduce, but if you could increase log level for vmware process and attach log when it crashes, it could help.

zabbix_server -R log_level_increase="vmware collector"

Comment by Vladislavs Sokurenko [ 2018 Sep 26 ]

Could also try adding such patch to monitor cache:

Index: src/zabbix_server/vmware/vmware.c
===================================================================
--- src/zabbix_server/vmware/vmware.c    (revision 85206)
+++ src/zabbix_server/vmware/vmware.c    (working copy)
@@ -4084,7 +4084,9 @@
     service->lastcheck = time(NULL);
 
     vmware_service_update_perf_entities(service);
-
+    zabbix_increase_log_level();
+    zbx_mem_dump_stats(vmware_mem);
+    zabbix_decrease_log_level();
     zbx_vmware_unlock();
 
     vmware_data_free(data);

 

Comment by Vladislavs Sokurenko [ 2018 Oct 01 ]

In one of the logs I see asked 248031720 bytes, that's 248 megabytes

1397:20180920:030749.965 __mem_malloc: skipped 586 asked 248031720 skip_min 256 skip_max 222534240
  1397:20180920:030749.966 file:vmware.c,line:87 zbx_mem_realloc(): out of memory (requested 248031720 bytes)
  1397:20180920:030749.966 file:vmware.c,line:87 zbx_mem_realloc(): please increase VMwareCacheSize configuration parameter
Comment by Camilo Camacho [ 2018 Oct 04 ]

The problem present again

 

 

 

/var/log/zabbix/zabbix_server.log-20180930- 16633:20180925:044320.369 End of vmware_hv_get_datacenter_name():SUCCEED
/var/log/zabbix/zabbix_server.log-20180930- 16633:20180925:044320.375 In vmware_service_create_datastore() datastore:'datastore-2560'
/var/log/zabbix/zabbix_server.log-20180930- 16633:20180925:044320.379 End of vmware_service_create_datastore()
/var/log/zabbix/zabbix_server.log-20180930- 16633:20180925:044320.379 In vmware_service_create_datastore() datastore:'datastore-2561'
/var/log/zabbix/zabbix_server.log-20180930- 16633:20180925:044320.383 End of vmware_service_create_datastore()
/var/log/zabbix/zabbix_server.log-20180930- 16633:20180925:044320.383 In vmware_service_create_datastore() datastore:'datastore-2685'
/var/log/zabbix/zabbix_server.log-20180930- 16633:20180925:044320.388 End of vmware_service_create_datastore()
/var/log/zabbix/zabbix_server.log-20180930- 16633:20180925:044320.388 In vmware_service_create_datastore() datastore:'datastore-2686'
/var/log/zabbix/zabbix_server.log-20180930- 16633:20180925:044320.392 End of vmware_service_create_datastore()
/var/log/zabbix/zabbix_server.log-20180930- 16633:20180925:044320.392 In vmware_service_create_datastore() datastore:'datastore-2687'
/var/log/zabbix/zabbix_server.log-20180930- 16633:20180925:044320.396 End of vmware_service_create_datastore()
/var/log/zabbix/zabbix_server.log-20180930- 16633:20180925:044320.396 In vmware_service_create_datastore() datastore:'datastore-2700'
/var/log/zabbix/zabbix_server.log-20180930- 16633:20180925:044320.402 End of vmware_service_create_datastore()
/var/log/zabbix/zabbix_server.log-20180930- 16633:20180925:044320.402 In vmware_service_create_datastore() datastore:'datastore-2723'
/var/log/zabbix/zabbix_server.log-20180930- 16633:20180925:044320.406 End of vmware_service_create_datastore()
–
/var/log/zabbix/zabbix_server.log-20180930:111695:20180927:030831.385 file:vmware.c,line:87 zbx_mem_realloc(): please increase VMwareCacheSize configuration parameter
/var/log/zabbix/zabbix_server.log-20180930-111668:20180927:030831.452 One child process died (PID:111695,exitcode/signal:1). Exiting ...
/var/log/zabbix/zabbix_server.log-20180930-111668:20180927:030833.459 syncing history data...
/var/log/zabbix/zabbix_server.log-20180930-111668:20180927:030833.502 item "00000000-0000-0000-0000-00000000000d:vmware.hv.network.in[\{$URL},\{HOST.HOST},bps]" became not supported: Couldn't connect to server
/var/log/zabbix/zabbix_server.log-20180930-111668:20180927:030833.502 item "00000000-0000-0000-0000-00000000000d:vmware.hv.network.out[\{$URL},\{HOST.HOST},bps]" became not supported: Couldn't connect to server
/var/log/zabbix/zabbix_server.log-20180930-111668:20180927:030833.502 item "00000000-0000-0000-0000-00000000003b:vmware.hv.datastore.size[\{$URL},\{HOST.HOST},COBGVM_BOGTM01_C2_DATASTORE03_RDP_FC_72K,free]" became not supported: Couldn't connect to server
/var/log/zabbix/zabbix_server.log-20180930-111668:20180927:030833.502 item "00000000-0000-0000-0000-00000000003b:vmware.hv.datastore.size[\{$URL},\{HOST.HOST},COBGVM_BOGTM01_C2_DATASTORE05_RDP_FC_72K,free]" became not supported: Couldn't connect to server
/var/log/zabbix/zabbix_server.log-20180930-111668:20180927:030833.502 item "00000000-0000-0000-0000-00000000003b:vmware.hv.datastore.size[\{$URL},\{HOST.HOST},COBGVM_BOGTM01_C1_DATASTORE07_RDP_FC_15K,pfree]" became not supported: Couldn't connect to server
/var/log/zabbix/zabbix_server.log-20180930-111668:20180927:030833.502 item "00000000-0000-0000-0000-00000000003b:vmware.hv.datastore.size[\{$URL},\{HOST.HOST},COBGVM_BOGTM01_C1_DATASTORE08_RDP_FC_15K,pfree]" became not supported: Couldn't connect to server
/var/log/zabbix/zabbix_server.log-20180930-111668:20180927:030833.502 item "00000000-0000-0000-0000-00000000003b:vmware.hv.datastore.size[\{$URL},\{HOST.HOST},COBGVM_BOGTM01_C1_DATASTORE03_RDP_FC_15K]" became not supported: Couldn't connect to server
/var/log/zabbix/zabbix_server.log-20180930-111668:20180927:030833.502 item "00000000-0000-0000-0000-00000000003b:vmware.hv.datastore.size[\{$URL},\{HOST.HOST},COBGVM_BOGTM01_C1_DATASTORE04_RDP_FC_15K]" became not supported: Couldn't connect to server
/var/log/zabbix/zabbix_server.log-20180930-111668:20180927:030833.502 item "00000000-0000-0000-0000-00000000005b:vmware.hv.datastore.size[\{$URL},\{HOST.HOST},COBGVM_BOGTM01_C2_DATASTORE07_RDP_FC_72K,free]" became not supported: Couldn't connect to server
/var/log/zabbix/zabbix_server.log-20180930-111668:20180927:030833.502 item "00000000-0000-0000-0000-00000000005b:vmware.hv.datastore.size[\{$URL},\{HOST.HOST},COBGVM_BOGTM01_C2_DATASTORE08_RDP_FC_72K,free]" became not supported: Couldn't connect to server
/var/log/zabbix/zabbix_server.log-20180930-111668:20180927:030833.502 item "00000000-0000-0000-0000-00000000005b:vmware.hv.datastore.size[\{$URL},\{HOST.HOST},COBGVM_BOGTM01_C2_DATASTORE05_RDP_FC_72K,pfree]" became not supported: Couldn't connect to server
/var/log/zabbix/zabbix_server.log-20180930-111668:20180927:030833.502 item "00000000-0000-0000-0000-00000000005b:vmware.hv.datastore.size[\{$URL},\{HOST.HOST},COBGVM_BOGTM01_C2_DATASTORE06_RDP_FC_72K,pfree]" became not supported: Couldn't connect to server
/var/log/zabbix/zabbix_server.log-20180930-111668:20180927:030833.502 item "00000000-0000-0000-0000-00000000005b:vmware.hv.datastore.size[\{$URL},\{HOST.HOST},COBGVM_BOGTM01_C1_DATASTORE08_RDP_FC_15K]" became not supported: Couldn't connect to server
–
/var/log/zabbix/zabbix_server.log-20180930: 34023:20180930:021002.349 file:vmware.c,line:87 zbx_mem_realloc(): please increase VMwareCacheSize configuration parameter
/var/log/zabbix/zabbix_server.log-20180930- 34000:20180930:021002.544 One child process died (PID:34023,exitcode/signal:1). Exiting ...
/var/log/zabbix/zabbix_server.log-20180930- 34000:20180930:021004.551 syncing history data...
/var/log/zabbix/zabbix_server.log-20180930- 34000:20180930:021004.551 syncing history data done
/var/log/zabbix/zabbix_server.log-20180930- 34000:20180930:021004.551 syncing trend data...
/var/log/zabbix/zabbix_server.log-20180930- 34000:20180930:021007.049 syncing trend data done
/var/log/zabbix/zabbix_server.log-20180930- 34000:20180930:021009.398 Zabbix Server stopped. Zabbix 3.4.13 (revision 84139).

Comment by Aigars Kadikis [ 2018 Oct 12 ]

Please set VMwareCacheSize=512M or VMwareCacheSize=768M. Stop Zabbix server. make sure there are no processes running:

ps aux|grep "[z]abbix_server"

Start Zabbix server again. Observe everything is started inside log file.

Comment by Camilo Camacho [ 2018 Oct 16 ]

Hello, I change value indicate, but The issue was presented again, so increase the value to VMwareCacheSize=900M.

 

48162:20181015:192627.644 cannot send list of active checks to "10.3.2.95": host [TM520002095] not found
 48129:20181015:192659.681 __mem_malloc: skipped 634 asked 110236320 skip_min 256 skip_max 6079032
 48129:20181015:192659.681 file:vmware.c,line:87 zbx_mem_realloc(): out of memory (requested 110236320 bytes)
 48129:20181015:192659.681 file:vmware.c,line:87 zbx_mem_realloc(): please increase VMwareCacheSize configuration parameter
 48096:20181015:192700.077 One child process died (PID:48129,exitcode/signal:1). Exiting ...
 48096:20181015:192702.084 syncing history data...
 48096:20181015:192702.085 syncing history data done
 48096:20181015:192702.085 syncing trend data...
 48096:20181015:192749.968 syncing trend data done

Comment by Camilo Camacho [ 2018 Nov 15 ]

Hi,

The configuration recomendate fail, presen issues again

 

51205:20181114:020822.019 file:vmware.c,line:87 zbx_mem_malloc(): out of memory (requested 24 bytes)
 51205:20181114:020822.019 file:vmware.c,line:87 zbx_mem_malloc(): please increase VMwareCacheSize configuration parameter
 51174:20181114:020822.673 One child process died (PID:51205,exitcode/signal:1). Exiting ...
 51174:20181114:020824.681 syncing history data...
 51174:20181114:020824.681 syncing history data done
 51174:20181114:020824.682 syncing trend data...
 51174:20181114:020848.720 syncing trend data done
 51174:20181114:020850.050 Zabbix Server stopped. Zabbix 3.4.13 (revision 84139).
–
  1419:20181115:163608.340 r15     =                0 =                    0 =                    0
  1419:20181115:163608.340 rdi     =     7ffe9a688bb0 =      140731488963504 =      140731488963504
  1419:20181115:163608.340 rsi     =     7ffe9a688bb0 =      140731488963504 =      140731488963504
  1419:20181115:163608.340 rbp     =     7ffe9a688b90 =      140731488963472 =      140731488963472
  1419:20181115:163608.340 rbx     =               12 =                   18 =                   18
  1419:20181115:163608.340 rdx     =     7ffe9a688bb0 =      140731488963504 =      140731488963504
  1419:20181115:163608.340 rax     = 8000000000000018 =  9223372036854775832 = -9223372036854775784
  1419:20181115:163608.341 rcx     =     5561bda53d40 =       93878281911616 =       93878281911616
  1419:20181115:163608.341 rsp     =     7ffe9a688b70 =      140731488963440 =      140731488963440
  1419:20181115:163608.341 rip     =     5561bbfe35d1 =       93878254187985 =       93878254187985
  1419:20181115:163608.341 efl     =            10206 =                66054 =                66054
  1419:20181115:163608.341 csgsfs  =               33 =                   51 =                   51
  1419:20181115:163608.341 err     =                0 =                    0 =                    0
  1419:20181115:163608.341 trapno  =                d =                   13 =                   13
  1419:20181115:163608.341 oldmask =                0 =                    0 =                    0
  1419:20181115:163608.341 cr2     =                0 =                    0 =                    0
  1419:20181115:163608.341 === Backtrace: ===
  1371:20181115:163608.364 One child process died (PID:1396,exitcode/signal:1). Exiting ...
  1371:20181115:163610.412 syncing history data...
  1371:20181115:163610.451 syncing history data done
  1371:20181115:163610.451 syncing trend data...
  1371:20181115:163718.204 syncing trend data done
  1371:20181115:163719.458 Zabbix Server stopped. Zabbix 3.4.13 (revision 84139).

_______________________________________________________________________

[root@zbx-bog-inf-wi ~]# tail -f /var/log/zabbix/zabbix_server.log
 51238:20181114:020736.494 cannot send list of active checks to "10.3.2.95": hos                                                                             t [TM520002095] not found
 51205:20181114:020822.019 __mem_malloc: skipped 0 asked 24 skip_min 4294967295                                                                              skip_max 0
 51205:20181114:020822.019 file:vmware.c,line:87 zbx_mem_malloc(): out of memo                                                                             ry (requested 24 bytes)
 51205:20181114:020822.019 file:vmware.c,line:87 zbx_mem_malloc(): please incr                                                                             ease VMwareCacheSize configuration parameter
 51174:20181114:020822.673 One child process died (PID:51205,exitcode/signal:1).                                                                              Exiting ...
 51174:20181114:020824.681 syncing history data...
 51174:20181114:020824.681 syncing history data done
 51174:20181114:020824.682 syncing trend data...
 51174:20181114:020848.720 syncing trend data done
 51174:20181114:020850.050 Zabbix Server stopped. Zabbix 3.4.13 (revision 84139

 

Comment by Nikolay [ 2018 Dec 10 ]

I have same problem on zabbix 4.0.2

Comment by Vladislavs Sokurenko [ 2018 Dec 10 ]

It could be solved by ZBX-9038

Comment by Edgars Melveris [ 2018 Dec 14 ]

Can you try version ** 4.0.3rc1 and see if the included fix helps in your situation?

Comment by Aigars Kadikis [ 2018 Dec 28 ]

Hello Yusev ,

Since the stable 4.0.3 has released, could you please update to this version and give us feedback.

The issue should be fixed because of ZBX-9038.

Comment by Aigars Kadikis [ 2019 Jan 11 ]

Should be fixed in version 4.0.3

Generated at Mon Aug 11 06:42:28 EEST 2025 using Jira 9.12.4#9120004-sha1:625303b708afdb767e17cb2838290c41888e9ff0.