[ZBX-14882] Zabbix proxy crash vmware monitoring Created: 2018 Sep 19  Updated: 2024 Apr 10  Resolved: 2018 Oct 08

Status: Closed
Project: ZABBIX BUGS AND ISSUES
Component/s: Proxy (P)
Affects Version/s: 3.0.22, 3.4.14, 4.0.0beta2
Fix Version/s: 3.0.23rc1, 3.4.15rc1, 4.0.1rc1, 4.2.0alpha1, 4.2 (plan)

Type: Problem report Priority: Major
Reporter: Chris Edgington Assignee: Vladislavs Sokurenko
Resolution: Fixed Votes: 0
Labels: vmware
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

RHEL 6.7

Attachments: File objdump.bz2     Text File zabbix_proxy.log    
Issue Links:
Causes
caused by ZBXNEXT-3106 Slow processing of VMware checks Closed
Duplicate
Team: Team A
Sprint: Sprint 43, Sprint 44
Story Points: 0.125

 Description   

**We're seeing our proxy server crashing daily at 15:20 and 18:10. 

zabbix_proxy.log - 

noname.xml:5: parser error : Opening and ending tag mismatch: link line 4 and head ^ noname.xml:10: parser error : Opening and ending tag mismatch: head line 2 and html ^ noname.xml:5: parser error : Opening and ending tag mismatch: link line 4 and head ^ noname.xml:10: parser error : Opening and ending tag mismatch: head line 2 and html ^ noname.xml:5: parser error : Opening and ending tag mismatch: link line 4 and head ^ noname.xml:10: parser error : Opening and ending tag mismatch: head line 2 and html

 

Full log file attached with vmware collector running in trace

 Comments   
Comment by Vladislavs Sokurenko [ 2018 Sep 19 ]

Thank you for your report, please attach objdump
objdump -DSswx zabbix_proxy

backtrace:

 29169:20180919:152114.509 Got signal [signal:11(SIGSEGV),reason:1,refaddr:0x40]. Crashing ...
 29169:20180919:152114.509 ====== Fatal information: ======
 29169:20180919:152114.509 Program counter: 0x7fd207c8a74b
 29169:20180919:152114.509 === Registers: ===
 29169:20180919:152114.509 r8      =         ffffffff =           4294967295 =           4294967295
 29169:20180919:152114.509 r9      =     7fd204653ed0 =      140539993603792 =      140539993603792
 29169:20180919:152114.509 r10     =     7fd204653ed0 =      140539993603792 =      140539993603792
 29169:20180919:152114.509 r11     =                0 =                    0 =                    0
 29169:20180919:152114.509 r12     =     7fd207d6659c =      140540051350940 =      140540051350940
 29169:20180919:152114.509 r13     =                a =                   10 =                   10
 29169:20180919:152114.509 r14     =                0 =                    0 =                    0
 29169:20180919:152114.509 r15     =                0 =                    0 =                    0
 29169:20180919:152114.509 rdi     =     7fd204653e80 =      140539993603712 =      140539993603712
 29169:20180919:152114.509 rsi     =                0 =                    0 =                    0
 29169:20180919:152114.509 rbp     =     7ffdcb4b1280 =      140728014148224 =      140728014148224
 29169:20180919:152114.509 rbx     =     7fd208877510 =      140540062954768 =      140540062954768
 29169:20180919:152114.509 rdx     =     7ffdcb4b12a0 =      140728014148256 =      140728014148256
 29169:20180919:152114.509 rax     =               40 =                   64 =                   64
 29169:20180919:152114.509 rcx     =                0 =                    0 =                    0
 29169:20180919:152114.509 rsp     =     7ffdcb4b11b0 =      140728014148016 =      140728014148016
 29169:20180919:152114.509 rip     =     7fd207c8a74b =      140540050450251 =      140540050450251
 29169:20180919:152114.509 efl     =            10202 =                66050 =                66050
 29169:20180919:152114.509 csgsfs  =               33 =                   51 =                   51
 29169:20180919:152114.509 err     =                4 =                    4 =                    4
 29169:20180919:152114.509 trapno  =                e =                   14 =                   14
 29169:20180919:152114.509 oldmask =                0 =                    0 =                    0
 29169:20180919:152114.509 cr2     =               40 =                   64 =                   64
 29169:20180919:152114.509 === Backtrace: ===
 29169:20180919:152114.516 11: /usr/sbin/zabbix_proxy: vmware collector #10 [updated 3, removed 0 VMware services in 5.226326 sec, querying VMware services](zbx_log_fatal_info+0x13c) [0x7fd207ce816a]
 29169:20180919:152114.516 10: /usr/sbin/zabbix_proxy: vmware collector #10 [updated 3, removed 0 VMware services in 5.226326 sec, querying VMware services](+0xc6581) [0x7fd207ce8581]
 29169:20180919:152114.516 9: /lib64/libc.so.6(+0x3f112326a0) [0x7fd2042f66a0]
 29169:20180919:152114.516 8: /usr/sbin/zabbix_proxy: vmware collector #10 [updated 3, removed 0 VMware services in 5.226326 sec, querying VMware services](+0x6874b) [0x7fd207c8a74b]
 29169:20180919:152114.516 7: /usr/sbin/zabbix_proxy: vmware collector #10 [updated 3, removed 0 VMware services in 5.226326 sec, querying VMware services](+0x6ac4f) [0x7fd207c8cc4f]
 29169:20180919:152114.516 6: /usr/sbin/zabbix_proxy: vmware collector #10 [updated 3, removed 0 VMware services in 5.226326 sec, querying VMware services](vmware_thread+0x32a) [0x7fd207c8eeec]
 29169:20180919:152114.516 5: /usr/sbin/zabbix_proxy: vmware collector #10 [updated 3, removed 0 VMware services in 5.226326 sec, querying VMware services](zbx_thread_start+0x37) [0x7fd207ce9333]
 29169:20180919:152114.516 4: /usr/sbin/zabbix_proxy: vmware collector #10 [updated 3, removed 0 VMware services in 5.226326 sec, querying VMware services](MAIN_ZABBIX_ENTRY+0xb32) [0x7fd207c573bb]
 29169:20180919:152114.516 3: /usr/sbin/zabbix_proxy: vmware collector #10 [updated 3, removed 0 VMware services in 5.226326 sec, querying VMware services](daemon_start+0x325) [0x7fd207ce7888]
 29169:20180919:152114.516 2: /usr/sbin/zabbix_proxy: vmware collector #10 [updated 3, removed 0 VMware services in 5.226326 sec, querying VMware services](main+0x30e) [0x7fd207c56882]
 29169:20180919:152114.516 1: /lib64/libc.so.6(__libc_start_main+0xfd) [0x7fd2042e2d5d]
 29169:20180919:152114.516 0: /usr/sbin/zabbix_proxy: vmware collector #10 [updated 3, removed 0 VMware services in 5.226326 sec, querying VMware services](+0x33069) [0x7fd207c55069]
Comment by Chris Edgington [ 2018 Sep 19 ]

objdump attached

Comment by Vladislavs Sokurenko [ 2018 Sep 19 ]

Probably this is the cause

	hv->props = xml_read_props(details, (char **)hv_propmap, ZBX_VMWARE_HVPROPS_NUM);

	if (NULL == hv->props[ZBX_VMWARE_HVPROP_HW_UUID])
		goto out;

In case hv->props is null

static char	**xml_read_props(const char *xml, char **propmap, int props_num)
{
	xmlDoc		*doc;
	xmlXPathContext	*xpathCtx;
	xmlXPathObject	*xpathObj;
	xmlNodeSetPtr	nodeset;
	xmlChar		*val;
	char		**props;
	int		i;

	if (NULL == (doc = xmlReadMemory(xml, strlen(xml), ZBX_VM_NONAME_XML, NULL, ZBX_XML_PARSE_OPTS)))
		return NULL;

It looks like very old issue, this code was not changed in a while.

Index: src/zabbix_server/vmware/vmware.c
===================================================================
--- src/zabbix_server/vmware/vmware.c	(revision 84586)
+++ src/zabbix_server/vmware/vmware.c	(working copy)
@@ -2710,7 +2710,8 @@
 	if (SUCCEED != vmware_service_get_hv_data(service, easyhandle, id, &details, error))
 		goto out;
 
-	hv->props = xml_read_props(details, hv_propmap, ZBX_VMWARE_HVPROPS_NUM);
+	if (NULL == (hv->props = xml_read_props(details, hv_propmap, ZBX_VMWARE_HVPROPS_NUM)))
+		goto out;
 
 	if (NULL == hv->props[ZBX_VMWARE_HVPROP_HW_UUID])
 		goto out;
Comment by Vladislavs Sokurenko [ 2018 Sep 20 ]

Fixed in development branch:
svn://svn.zabbix.com/branches/dev/ZBX-14882

Comment by Vladislavs Sokurenko [ 2018 Oct 03 ]

Fixed in:

  • pre-3.0.23rc1 r85376
  • pre-3.4.15rc1 r85379
  • pre-4.0.1rc1 r85380
  • pre-4.2.0alpha1 (trunk) r85381
Generated at Thu Apr 25 07:53:09 EEST 2024 using Jira 9.12.4#9120004-sha1:625303b708afdb767e17cb2838290c41888e9ff0.