[ZBX-20032] Server crash during when monitored host is down (VMware vCenter) Created: 2021 Sep 30  Updated: 2024 Apr 10  Resolved: 2021 Nov 08

Status: Closed
Project: ZABBIX BUGS AND ISSUES
Component/s: Proxy (P), Server (S)
Affects Version/s: 5.0.15, 5.0.16
Fix Version/s: 4.0.36rc1, 5.0.18rc1, 5.4.8rc1, 6.0.0alpha6, 6.0 (plan)

Type: Problem report Priority: Trivial
Reporter: Peter Roka Assignee: Michael Veksler
Resolution: Fixed Votes: 1
Labels: crash
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

CentOS Linux 7.9 x64, Percona 5.7.35, running as a virtual server on Microsoft Hyper-V 2019.

Attachments: Text File ZBX-20032-5.0.patch     File zabbix-5.0.18rc1-ZBX20032.tar.gz     File zabbix-proxy-mysql-5.0.17-0.Z20032.el15.x86_64.rpm     File zabbix-proxy-mysql-5.0.17-0.Z20032.el7.x86_64.rpm     File zabbix-proxy-mysql-5.0.17-0.Z20032.el8.x86_64.rpm     File zabbix-proxy-pgsql-5.0.17-0.Z20032.el15.x86_64.rpm     File zabbix-proxy-pgsql-5.0.17-0.Z20032.el7.x86_64.rpm     File zabbix-proxy-pgsql-5.0.17-0.Z20032.el8.x86_64.rpm     File zabbix-proxy-sqlite3-5.0.17-0.Z20032.el15.x86_64.rpm     File zabbix-proxy-sqlite3-5.0.17-0.Z20032.el7.x86_64.rpm     File zabbix-proxy-sqlite3-5.0.17-0.Z20032.el8.x86_64.rpm     File zabbix-server-mysql-5.0.17-0.Z20032.el15.x86_64.rpm     File zabbix-server-mysql-5.0.17-0.Z20032.el7.x86_64.rpm     File zabbix-server-mysql-5.0.17-0.Z20032.el8.x86_64.rpm     File zabbix-server-pgsql-5.0.17-0.Z20032.el15.x86_64.rpm     File zabbix-server-pgsql-5.0.17-0.Z20032.el7.x86_64.rpm     File zabbix-server-pgsql-5.0.17-0.Z20032.el8.x86_64.rpm     Text File zabbix_server.log    
Issue Links:
Causes
Duplicate
is duplicated by ZBX-20022 Zabbix server crashes in "vmware coll... Closed
Team: Team B
Sprint: Sprint 81 (Oct 2021), Sprint 82 (Nov 2021)
Story Points: 0.25

 Description   

Steps to reproduce:

  1. VMware cluster (through vCenter) is monitored by Zabbix Server. Hosts, clusters, VMs are discovered by the Zabbix template discoveries.
  2. VMware vCenter appliance is restarted.

Result:

  1. After some time, before VMware vCenter appliance returns to online state, Zabbix server crashes and restarts.
  2. See uploaded log file.

Expected:
Zabbix Server does not crash when a monitored vCenter Server goes offline.

Note: Zabbix server was recently upgraded from 4.0.33 to 5.0.15, then to 5.0.16. This issue was NOT present on Zabbix Server 4.0!

 Comments   
Comment by Michael Veksler [ 2021 Oct 20 ]

rokapi Be so kind as to reproduce the crash with DebugLevel=4 and provide the Log file.

Comment by Peter Roka [ 2021 Oct 21 ]

Dear Michael!

 

Log file is too big to upload, even compressed. You can download it from the below link. Zabbix Server has chrashed and restarted 2x during the vCenter downtime. The times Zabbix Server crashed are around 20211021:122523 and 20211021:122650, respectively.

 

Download

 

Best regards,

Peter Roka

Comment by Michael Veksler [ 2021 Oct 21 ]

Dear rokapi  - Thanks !

Comment by Michael Veksler [ 2021 Oct 21 ]

Dear rokapi,
May I ask you to test the patch or you have no possibility ?

diff --git a/src/zabbix_server/vmware/vmware.c b/src/zabbix_server/vmware/vmware.c
index 1ba3c97fdb..8f2115bbb0 100644
--- a/src/zabbix_server/vmware/vmware.c
+++ b/src/zabbix_server/vmware/vmware.c
@@ -2049,6 +2049,12 @@ static   int     vmware_service_get_contents(CURL *easyhandle, char **version, char **
        *fullname = zbx_xml_read_doc_value(doc, ZBX_XPATH_VMWARE_ABOUT("fullName"));
        zbx_xml_free_doc(doc);
 
+       if (NULL == *version)
+       {
+               *error = zbx_strdup(*error, "VMware Virtual Center is not ready.");
+               return FAIL;
+       }
+
        return SUCCEED;
 
 #      undef ZBX_POST_VMWARE_CONTENTS
Comment by Peter Roka [ 2021 Oct 21 ]

Our server was installed using packages from the RPM repository, not by compiling from source, so I am unable to test your patch, unfortunately. Is there any other way that I can test it?

Comment by Michael Veksler [ 2021 Oct 21 ]

Can  you try to testing zabbix-5.0.18rc1-ZBX20032.tar.gz ?

Comment by Peter Roka [ 2021 Oct 21 ]

Apparently, based on a quick test, 5.0.18rc1 fixes the isue. I can produce a new debug log tomorrow, if requested.

 

Thank you!

Peter

Comment by Michael Veksler [ 2021 Oct 22 ]

New log does not required. Please do couple of additional tests and write a result.

Comment by Peter Roka [ 2021 Oct 22 ]

Subsequent test confirms that the issue is fixed.

When vCenter server goes offline, there are "item ... became not supported: Couldn't connect to server" lines in the log. After some time, lines like  "error reason for ... changed: VMware Virtual Center server is not ready." take over, and finally when vCenter server is back online, log lines like "item ... became supported" appear.

 

Thank you again!

Peter

Comment by Michael Veksler [ 2021 Nov 01 ]

Available in:

Comment by Jared Wight [ 2021 Nov 02 ]

Does this issue exist in 4.0.35?

Comment by Michael Veksler [ 2021 Nov 02 ]

Dear wightjw  - v4.0 has been fixed also.

Comment by Oleksii Zagorskyi [ 2021 Nov 09 ]

Backtrace from log in current issue, for easier searching:

228360:20210930:150757.653 server #25 started [vmware collector #1]
...
228360:20210930:150858.113 Got signal [signal:11(SIGSEGV),reason:1,refaddr:(nil)]. Crashing ...
...
228360:20210930:150858.115 === Backtrace: ===
228360:20210930:150858.116 13: /usr/sbin/zabbix_server: vmware collector #1 [updated 0, removed 0 VMware services in 0.000038 sec, querying VMware services](zbx_backtrace+0x42) [0x55b01d5ff1c0]
228360:20210930:150858.116 12: /usr/sbin/zabbix_server: vmware collector #1 [updated 0, removed 0 VMware services in 0.000038 sec, querying VMware services](zbx_log_fatal_info+0x151) [0x55b01d5ff43e]
228360:20210930:150858.116 11: /usr/sbin/zabbix_server: vmware collector #1 [updated 0, removed 0 VMware services in 0.000038 sec, querying VMware services](+0x1cdbb3) [0x55b01d5ffbb3]
228360:20210930:150858.116 10: /lib64/libpthread.so.0(+0xf630) [0x7f6c2b8ab630]
228360:20210930:150858.116 9: /lib64/libc.so.6(+0x13eec6) [0x7f6c28a70ec6]
228360:20210930:150858.116 8: /usr/sbin/zabbix_server: vmware collector #1 [updated 0, removed 0 VMware services in 0.000038 sec, querying VMware services](+0xa5c8c) [0x55b01d4d7c8c]
228360:20210930:150858.116 7: /usr/sbin/zabbix_server: vmware collector #1 [updated 0, removed 0 VMware services in 0.000038 sec, querying VMware services](+0xa67e7) [0x55b01d4d87e7]
228360:20210930:150858.116 6: /usr/sbin/zabbix_server: vmware collector #1 [updated 0, removed 0 VMware services in 0.000038 sec, querying VMware services](vmware_thread+0x34f) [0x55b01d4db6c0]
228360:20210930:150858.116 5: /usr/sbin/zabbix_server: vmware collector #1 [updated 0, removed 0 VMware services in 0.000038 sec, querying VMware services](zbx_thread_start+0x37) [0x55b01d60e36f]
228360:20210930:150858.117 4: /usr/sbin/zabbix_server: vmware collector #1 [updated 0, removed 0 VMware services in 0.000038 sec, querying VMware services](MAIN_ZABBIX_ENTRY+0xe6b) [0x55b01d47c279]
228360:20210930:150858.117 3: /usr/sbin/zabbix_server: vmware collector #1 [updated 0, removed 0 VMware services in 0.000038 sec, querying VMware services](daemon_start+0x305) [0x55b01d5fee0c]
228360:20210930:150858.117 2: /usr/sbin/zabbix_server: vmware collector #1 [updated 0, removed 0 VMware services in 0.000038 sec, querying VMware services](main+0x312) [0x55b01d47b3c4]
228360:20210930:150858.117 1: /lib64/libc.so.6(__libc_start_main+0xf5) [0x7f6c28954555]
228360:20210930:150858.117 0: /usr/sbin/zabbix_server: vmware collector #1 [updated 0, removed 0 VMware services in 0.000038 sec, querying VMware services](+0x483f9) [0x55b01d47a3f9]
...
228332:20210930:150858.135 One child process died (PID:228360,exitcode/signal:1). Exiting ...
zabbix_server [228332]: Error waiting for process with PID 228360: [10] No child processes
228332:20210930:150858.220 syncing history data...
228332:20210930:150858.245 syncing history data... 100.000000%
228332:20210930:150858.245 syncing history data done
228332:20210930:150858.245 syncing trend data...
228332:20210930:150908.720 syncing trend data done
228332:20210930:150908.723 Zabbix Server stopped. Zabbix 5.0.16 (revision 422e00c66a).
Comment by Edgar Akhmetshin [ 2021 Nov 09 ]

Final patch for LTS 5.0: ZBX-20032-5.0.patch.

RHEL8:
zabbix-server-pgsql-5.0.17-0.Z20032.el8.x86_64.rpm
zabbix-server-mysql-5.0.17-0.Z20032.el8.x86_64.rpm

zabbix-proxy-mysql-5.0.17-0.Z20032.el8.x86_64.rpm
zabbix-proxy-pgsql-5.0.17-0.Z20032.el8.x86_64.rpm
zabbix-proxy-sqlite3-5.0.17-0.Z20032.el8.x86_64.rpm

RHEL7:
zabbix-server-pgsql-5.0.17-0.Z20032.el7.x86_64.rpm
zabbix-server-mysql-5.0.17-0.Z20032.el7.x86_64.rpm

zabbix-proxy-mysql-5.0.17-0.Z20032.el7.x86_64.rpm
zabbix-proxy-pgsql-5.0.17-0.Z20032.el7.x86_64.rpm
zabbix-proxy-sqlite3-5.0.17-0.Z20032.el7.x86_64.rpm

SLES15:
zabbix-server-mysql-5.0.17-0.Z20032.el15.x86_64.rpm
zabbix-server-pgsql-5.0.17-0.Z20032.el15.x86_64.rpm

zabbix-proxy-mysql-5.0.17-0.Z20032.el15.x86_64.rpm
zabbix-proxy-pgsql-5.0.17-0.Z20032.el15.x86_64.rpm
zabbix-proxy-sqlite3-5.0.17-0.Z20032.el15.x86_64.rpm

Generated at Wed Apr 02 23:30:20 EEST 2025 using Jira 9.12.4#9120004-sha1:625303b708afdb767e17cb2838290c41888e9ff0.