[ZBX-15663] zabbix did not support vCenter6.7 low level discovery Created: 2019 Feb 18  Updated: 2020 Mar 25  Resolved: 2020 Mar 25

Status: Closed
Project: ZABBIX BUGS AND ISSUES
Component/s: Templates (T)
Affects Version/s: 3.0.25, 4.0.4, 4.4.4
Fix Version/s: None

Type: Incident report Priority: Trivial
Reporter: jack Assignee: Aigars Kadikis
Resolution: Workaround proposed Votes: 5
Labels: curl, vcenter
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

Zabbix 4.0 / Zabbix 3.4
vCenter6.7


Attachments: File [email protected]     File [email protected]     File [email protected]     XML File 6-5-soapui-project.xml     XML File 6-7-soapui-project.xml     Zip Archive 6.5.zip     PNG File 6.5_latest_data.PNG     Zip Archive 6.7.zip     PNG File 6.7_latest_data.PNG     PNG File Capture1.PNG     PNG File host_macro.PNG     PNG File host_template.PNG     PNG File host_template.PNG     PNG File host_user.PNG     PNG File image-2020-01-08-11-24-55-963.png     PNG File template_error.PNG     Text File zabbix_server.log    

 Description   

Steps to reproduce:

  1. In the Zabbix 4.0 (3.4 has the same issue)
  2. Add 1 vCenter6.7 host and link the 'Template VM VMware'
  3. Set the vCenter6.7 host macros with related url/sdk
  4. Zabbix can not discovery the host and  display 'SSL connect error' 
  5. Try the vCenter6.5 with same steps, it's work

Result:
See screenshot...
See log file...
See memory dump...
Expected:
See screenshot....
See attached patch file...



 Comments   
Comment by jack [ 2019 Feb 18 ]

Can anyone help on it? it's blocking my zabbix several months

Comment by Arturs Lontons [ 2019 Feb 18 ]

Hi,

Please increase the zabbix server log-level by executing the following command on the zabbix server twice:

zabbix_server -R log_level_increase='vmware collector'

Next up, please provide a partial zabbix_server.log file with the log entries corresponding to the vCenter connection issues.
Additionally, you could also provide a vCenter log, which holds the information regarding dropped connections.

Comment by jack [ 2019 Feb 18 ]

please see the zabbix logs and vCenter logs

Comment by jack [ 2019 Feb 18 ]

About the another error, I changed the maxQueryMetrics number in vCenter, but zabbix is still report the error.

Error of query maxQueryMetrics: 'config.vpxd.stats.maxQueryMetrics' is invalid or exceeds the maximum number of characters permitted..

Comment by jack [ 2019 Feb 25 ]

Any progress on the ticket? It shows blocker status

Comment by Aigars Kadikis [ 2019 Feb 28 ]

How did you performed both tests: vCenter6.5, vCenter6.7? Do you own 2 environments?

Are you still facing "SSL connect error" message? What is the situation of SSL layer? Are these certificates:

  • valid in current time period
  • self-signed
  • same connection settings (TLSv1.1, TLSv1.2)
  • same domain

Inside the log file I can see a lot of "vmware.*" items going to not supported state. Does all of items in frontend are not supported? Do you at least one of them working?

Please make sure you are using the latest version of VMWare template. Here it is:
https://www.zabbix.org/websvn/wsvn/zabbix.com/tags/4.0.5/templates/classic/template_vm_vmware.xml

 

Comment by jack [ 2019 Feb 28 ]
  1. yes,  actually I have 2 vCenter with 6.5 and 6.7 monitored by zabbix.
  2. yes, for the vCenter6.7,  the zabbix still display the 'ssl connect error', but with same zabbix template for vm_vmware, it has not problem for vCenter6.5
  3. Actually, I installed another zabbix 4.0.5 to monitor these 2 vCenters also, same issue for vCenter.
Comment by Aigars Kadikis [ 2019 Mar 06 ]

 

Hello Jack,

Good to hear you have both instances side by side.

Please try using https://www.soapui.org and communicate with VMware SDK service.

There URL to enter in SoapUI must be in the format.

https://192.168.1.100/sdk/vimService.wsdl

 

This should download the WSDL file and give the list of available methods

Kindly attach the output for both cases?

Comment by jack [ 2019 Mar 07 ]

Attached the 6.5 and 6.7 soap xml. Also attach the interface list in the zip package

Hope it will help.

Comment by Aigars Kadikis [ 2019 Mar 08 ]

Hello Jack,

I looking again into zabbix_server.log and assuming that hosts (coming from host prototype definition) are created in an instance. Only the metrics are not coming regarding each host. Please confirm.

At this point please go to latest data page and search for one host. Select the checkboxes:

  • Show items without data
  • Show details

Do you see an expanded {$URL} macro for each vmware.vm* item?

Comment by jack [ 2019 Mar 11 ]

Yes, Please see the host information

The attached  host template page is I created for my vCenter6.7 which link with vmware template

The attached host marco is I set host macro for my vCenter6.7

The attached 6.5_latest_data is snapshot for my vCenter6.5 in latest data page, You can see the host has right information.

The attached 6.7_latest_data is snapshot for my vCenter6.7 in latest data page, It has error to discover vcenter information.  But I used same  host template and marco url for my 2 vCenter.

 

Comment by Aigars Kadikis [ 2019 Mar 12 ]

Regarding vCenter 6.7 could you please temporary set up a communicating channel with SDK through vCenter super admin (root) user account? Please list your findings.

Comment by jack [ 2019 Mar 13 ]

Could you tell me how to set up it? I am not familiar with that

Comment by jack [ 2019 Mar 13 ]

try the curl to access the sdk, found the issue may help you root cause.

curl access vCenter6.7:

[root@zabbix02 ~]# curl -k -v -u [email protected] https://10.231.66.10/sdk
Enter host password for user '[email protected]':

  • About to connect() to 10.231.66.10 port 443 (#0)
  • Trying 10.231.66.10... connected
  • Connected to 10.231.66.10 (10.231.66.10) port 443 (#0)
  • Initializing NSS with certpath: sql:/etc/pki/nssdb
  • warning: ignoring value of ssl.verifyhost
    * NSS error -5938
  • Closing connection #0
  • SSL connect error
    curl: (35) SSL connect error

 

 

curl access vCenter6.5:
[root@zabbix02 ~]# curl -k -v -u [email protected] https://10.231.66.14/sdk
Enter host password for user '[email protected]':

  • About to connect() to 10.231.66.14 port 443 (#0)
  • Trying 10.231.66.14... connected
  • Connected to 10.231.66.14 (10.231.66.14) port 443 (#0)
  • Initializing NSS with certpath: sql:/etc/pki/nssdb
  • warning: ignoring value of ssl.verifyhost
  • skipping SSL peer certificate verification
  • SSL connection using TLS_RSA_WITH_AES_256_CBC_SHA
  • Server certificate:
  • subject: C=US,CN=10.231.66.14
  • start date: May 01 14:38:00 2018 GMT
  • expire date: Apr 25 14:37:59 2028 GMT
  • common name: 10.231.66.14
  • issuer: OU=VMware Engineering,O=photon-machine,ST=California,C=US,DC=local,DC=vsphere,CN=CA
  • Server auth using Basic with user '[email protected]'
    > GET /sdk HTTP/1.1
    > Authorization: Basic YWRtaW5pc3RyYXRvckB2c3BoZXJlLmxvY2FsOlBhc3N3b3JkMTIzIQ==
    > User-Agent: curl/7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7 NSS/3.19.1 Basic ECC zlib/1.2.3 libidn/1.18 libssh2/1.4.2
    > Host: 10.231.66.14
    > Accept: /
    >
    < HTTP/1.1 404 Not Found
    < Date: Wed, 13 Mar 2019 05:47:45 GMT
    < Connection: close
    < Content-Type: text; charset=plain
    < X-Frame-Options: DENY
    < Content-Length: 0
    <
  • Closing connection #0
Comment by jack [ 2019 Mar 13 ]

aha, I found the issue. I use the tlsv1.2 the suceess to access the vCenter6.7 sdk.

The only question is how should I add the tlsv1.2 parameter to the vCenter host setting?

[root@zabbix02 ~]# curl -k -v --tlsv1.2 -u [email protected] https://10.231.66.10/sdk
Enter host password for user '[email protected]':

  • About to connect() to 10.231.66.10 port 443 (#0)
  • Trying 10.231.66.10... connected
  • Connected to 10.231.66.10 (10.231.66.10) port 443 (#0)
  • Initializing NSS with certpath: sql:/etc/pki/nssdb
  • warning: ignoring value of ssl.verifyhost
  • skipping SSL peer certificate verification
  • Server certificate:
  • subject: C=US,CN=10.231.66.10
  • start date: Jan 21 02:59:27 2019 GMT
  • expire date: Jan 15 02:59:27 2029 GMT
  • common name: 10.231.66.10
  • issuer: OU=VMware Engineering,O=photon-machine,ST=California,C=US,DC=local,DC=vsphere,CN=CA
  • Server auth using Basic with user '[email protected]'
    > GET /sdk HTTP/1.1
    > Authorization: Basic YWRtaW5pc3RyYXRvckB2c3BoZXJlLmxvY2FsOlBhc3N3b3JkMTIzIQ==
    > User-Agent: curl/7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7 NSS/3.27.1 zlib/1.2.3 libidn/1.18 libssh2/1.4.2
    > Host: 10.231.66.10
    > Accept: /
    >
    < HTTP/1.1 404 Not Found
    < Date: Wed, 13 Mar 2019 06:04:06 GMT
    < Connection: close
    < Content-Type: text/plain; charset=utf-8
    < Content-Length: 0
    <
  • Closing connection #0
Comment by jack [ 2019 Mar 18 ]

@aigars.kadikis

Seem the RCA is clear that vSphere6.7 is using tlsv1.2 , Could we have quickly fix on the issue?

Comment by jack [ 2019 Mar 27 ]

Hi Experts,

Any response on the ticket?

Comment by Aigars Kadikis [ 2019 Mar 28 ]

Hello Jack,

I'm sorry it took me so long.

A solution/workaround would be to change the curl crypto library from GnuTLS to OpenSSL. Kindly recompile libcurl library or install a precompiled package which is based on OpenSSL.

I'm wondering what OS you are using where the curl actually gets executed? You mention Zabbix 4.0 in the description, what is the base system in this example?

Comment by jack [ 2019 Mar 29 ]

I am using CentOS6.7 for both Zabbix3.4 and Zabbix 4.0.

That's the version:

2.6.32-573.el6.x86_64

Comment by jack [ 2019 Mar 29 ]

The issue is urgent for me. 

Do you have the instruction on how to replace the curl library?

Comment by Aigars Kadikis [ 2019 Apr 03 ]

Hello Jack,

This instruction can work: https://unix.stackexchange.com/questions/84283/how-can-i-get-tlsv1-2-support-in-apache-on-rhel6-centos-sl6

Also, a workaround can be to bring up a new proxy with CentOS7 and configure the monitoring through this machine.

Comment by jack [ 2019 Apr 03 ]

Thank you point it out.

It seems hard to me recompile the openssl lib in my host.

Does Zabbix will install the fix in future release?

Comment by Luca Carangelo [ 2019 Apr 11 ]

Problem seems to be related to the usage of SSL instead of TLS.

Example 

FAIL  (-3 OPTION IS FOR SSLV3)

curl -k -v -3 -u [email protected] https://10.30.3.135/sdk

Enter host password for user '[email protected]':

  • About to connect() to 10.30.3.135 port 443 (#0)
  • Trying 10.30.3.135...
  • Connected to 10.30.3.135 (10.30.3.135) port 443 (#0)
  • Initializing NSS with certpath: sql:/etc/pki/nssdb
  • NSS error -5938 (PR_END_OF_FILE_ERROR)
  • Encountered end of file
  • Closing connection 0
    curl: (35) Encountered end of file

PASS  (-1 OPTION IS FOR TLS): 

curl -k -v -1 -u [email protected] https://10.30.3.135/sdk
Enter host password for user '[email protected]':

  • About to connect() to 10.30.3.135 port 443 (#0)
  • Trying 10.30.3.135...
  • Connected to 10.30.3.135 (10.30.3.135) port 443 (#0)
  • Initializing NSS with certpath: sql:/etc/pki/nssdb
  • skipping SSL peer certificate verification
  • SSL connection using TLS_RSA_WITH_AES_128_GCM_SHA256
  • Server certificate:
  • subject: C=US,CN=**********
  • start date: Mar 25 10:55:47 2019 GMT
  • expire date: Mar 19 10:24:39 2029 GMT
  • common name: *********
  • issuer: OU=******,O=******,ST=****,C=US,DC=local,DC=vsphere,CN=CA
  • Server auth using Basic with user '[email protected]'
    > GET /sdk HTTP/1.1
    > Authorization: Basic emFiYml4QGUtdmlhLWNsb3VkLml0OnJFQUQwTkxZ
    > User-Agent: curl/7.29.0
    > Host: 10.30.3.135
    > Accept: /
    >
    < HTTP/1.1 404 Not Found
    < Date: Thu, 11 Apr 2019 10:14:56 GMT
    < Connection: close
    < Content-Type: text/plain; charset=utf-8
    < Content-Length: 0
    <
  • Closing connection 0

--------------------------------------------------------------------------------------------------------------

Comparing with other VCenter versions , old one are able to accept SSL negotiation , vCenter 6.7 instead does not allow SSL anymore...ONLY TLS.

Template VM VMware need to be modified in order to negotiate TLS too if this is available.

How can I workaround this issue in order to monitor vCenter 6.7?

 

Please let me know ASAP.

 

 

Comment by jack [ 2019 Apr 30 ]

Dear zabbix dev,

Do we have the fix in latest zabbix version?

Comment by Aigars Kadikis [ 2019 May 02 ]

Can you please confirm you are using the Zabbix server on the top of the quite recent operating system?

Comment by Luca Carangelo [ 2019 May 02 ]

CentOS Linux release 7.3.1611 (Core)

Comment by Luca Carangelo [ 2019 May 26 ]

Can you please provide an update ?

Comment by Roberto Cremasco [ 2019 Jun 17 ]

You can check which version of curl you are using, we have a similar situation:

vsphere 6.7 supported:

curl 7.29.0 (x86_64-redhat-linux-gnu) libcurl/7.29.0 NSS/3.28.4 zlib/1.2.7 libidn/1.28 libssh2/1.4.3
Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps pop3 pop3s rtsp scp sftp smtp smtps telnet tftp
Features: AsynchDNS GSS-Negotiate IDN IPv6 Largefile NTLM NTLM_WB SSL libz unix-sockets

vsphere 6.7 not supported:

curl 7.19.7 (x86_64-suse-linux-gnu) libcurl/7.19.7 OpenSSL/0.9.8j zlib/1.2.3 libidn/1.10
Protocols: tftp ftp telnet dict ldap http file https ftps
Features: GSS-Negotiate IDN IPv6 Largefile NTLM SSL libz

 

Comment by Luca Carangelo [ 2019 Jun 17 ]

We have 
curl 7.29.0
but it doesn't work.

Comment by Aigars Kadikis [ 2019 Nov 22 ]

Did you land 7.29.0 curl via rpm packages or compiled from sources on your CentOS 7.3? Please show supported features:

curl --version
Comment by Luca Carangelo [ 2019 Nov 22 ]

Installed via yum  (This is a CentOS 7.x )

 

curl --version
curl 7.29.0 (x86_64-redhat-linux-gnu) libcurl/7.29.0 NSS/3.28.4 zlib/1.2.7 libidn/1.28 libssh2/1.4.3
Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps pop3 pop3s rtsp scp sftp smtp smtps telnet tftp
Features: AsynchDNS GSS-Negotiate IDN IPv6 Largefile NTLM NTLM_WB SSL libz unix-sockets

 

Comment by Aigars Kadikis [ 2019 Nov 25 ]

Provide please more details about OS:

cat /etc/*release* 

Please do the walk from the command line

curl -k -v -u [email protected] https://<ip>/sdk
# replace with valid username and password

 

Comment by Yasumi [ 2019 Dec 06 ]

I am using vCenter6.7.The exact same error event has been confirmed, resulting in an "SSL connection error". In vCenter 6.7 or later, the default settings for disabling SSL, TLSv1.0, and 1.1 are the cause.

Is Zabbix ready to get vCenter6.7 API information using TLSv1.2?

This is a very important issue.Zabbix makes it impossible to LLD monitor VM virtual environments in vCenter6.7 and later.

 

vCenter6.7を使用しています。まったく同じエラーイベントを確認しており、「SSL接続エラー」が発生しています。 vCenter6.7以降では、SSL、TLSv1.0および1.1を無効にするためのデフォルト設定がされていることが原因です。

ZabbixはTLSv1.2を使用してvCenter6.7 API情報を取得する予定がありますか?

非常に重要な問題です。Zabbixでは、vCenter6.7以降のVM仮想環境をLLD監視できない状態です。

Comment by Marco Hofmann [ 2019 Dec 06 ]

Just to state a different opinion, and maybe it even helps in some way.

I have a Zabbix 4.4.3 environment. 1 Server and 30+ Proxies. All are Debian 9 stretch amd64 on the latest patch level. I monitor about 15 vCenter from which seven are 6.7. I don't have any problems with Host Prototype Discovery or SSL errors.

root@zabprox:~# curl --version
curl 7.52.1 (x86_64-pc-linux-gnu) libcurl/7.52.1 OpenSSL/1.0.2t zlib/1.2.8 libidn2/0.16 libpsl/0.17.0 (+libidn2/0.16) libssh2/1.7.0 nghttp2/1.18.1 librtmp/2.3
Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps pop3 pop3s rtmp rtsp scp sftp smb smbs smtp smtps telnet tftp 
Features: AsynchDNS IDN IPv6 Largefile GSS-API Kerberos SPNEGO NTLM NTLM_WB SSL libz TLS-SRP HTTP2 UnixSockets HTTPS-proxy PSL
Comment by Yasumi [ 2019 Dec 12 ]

Thank you.

In other words, this is not a problem with Zabbix, but a problem with the curl version of Zabbix Server?

If so, I have been asked to build Zabbix Server on a more modern OS version.

 

[root@ZabbixServer ~]# cat /etc/redhat-release 
Red Hat Enterprise Linux Server release 7.1 (Maipo)

[root@ZabbixServer ~]# curl --version 
curl 7.29.0 (x86_64-redhat-linux-gnu) libcurl/7.29.0 NSS/3.15.4 zlib/1.2.7 libidn/1.28 libssh2/1.4.3
Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps pop3 pop3s rtsp scp sftp smtp smtps telnet tftp
Features: AsynchDNS GSS-Negotiate IDN IPv6 Largefile NTLM NTLM_WB SSL libz

[root@ZabbixServer ~]# curl -k -v -u [email protected] https://***.***.***.****/sdk
Enter host password for user '[email protected]':
* About to connect() to ***.***.***.**** port 443 (#0)
*   Trying ***.***.***.****...
* Connected to ***.***.***.**** (***.***.***.****) port 443 (#0)
* Initializing NSS with certpath: sql:/etc/pki/nssdb
* NSS error -5938 (PR_END_OF_FILE_ERROR)
* Encountered end of file
* Closing connection 0
curl: (35) Encountered end of file

 

Comment by ping [ 2020 Jan 02 ]

My Zabbix OS Platform is Redhat 6.5

I got zabbix error log "became not supported: SSL connect error" and I sloved by the following:

In ESXi 6.7 Web Console -> System -> Advances Setting -> Search UserVars.ESXiVPsDisabledProtocols Default Value is sslv3,tlsv1,tlsv1.1

Edit Option and Value Change to sslv3 than Save.

Log in ESXi Console -> Troubleshooting Options -> Restart Managements Agents

Zabbix 3.4.15 get Zabbix 6.7 Performance Values.

VCenter 6.7 may download VMware-vSphereTlsReconfigurator Tools to Change Values.

Comment by Damien Dye [ 2020 Jan 07 ]

am having the same thing.  with Zabbix 4.4.4

I can see that the XML is being returned by the vcentre but not processed by the system and nothing is created.

all keys are unsupported states with time out error but the setting are quite high for a minimal install

 # 
 ## 
 ### Option: StartVMwareCollectors
 StartVMwareCollectors=2
 ### Option: VMwareFrequency
 # How often Zabbix will connect to VMware service to obtain a new data.
 VMwareFrequency=300
 ## 
 ### Option: VMwarePerfFrequency
 # How often Zabbix will connect to VMware service to obtain performance data.
 VMwarePerfFrequency=300
 ## 
 ### Option: VMwareCacheSize
 # Size of VMware cache, in bytes.
 # Shared memory size for storing VMware data.
 # Only used if VMware collectors are started.
 VMwareCacheSize=2G
 ## 
 ### Option: VMwareTimeout
 # Specifies how many seconds vmware collector waits for response from VMware service.
 VMwareTimeout=150

 

 

 19170:20200107:093928.578 vmware_service_get_vm_data() SOAP response: <?xml version="1.0" encoding="UTF-8"?>
<soapenv:Envelope xmlns:soapenc="http://schemas.xmlsoap.org/soap/encoding/"
 xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/"
 xmlns:xsd="http://www.w3.org/2001/XMLSchema"
 xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
<soapenv:Body>
<RetrievePropertiesExResponse xmlns="urn:vim25"><returnval><objects><obj type="VirtualMachine">vm-731</obj><propSet><name>config.hardware</name><val xsi:type="VirtualHardware"><numCPU>2</numCPU><memoryMB>4096</memoryMB><device xsi:type="VirtualPCIController"><key>100</key><deviceInfo><label>PCI controller 0</label><summary>PCI controller 0</summary></deviceInfo><busNumber>0</busNumber><device>500</device><device>12000</device><device>14000</device><device>1000</device><device>15000</device><device>4000</device></device><device xsi:type="VirtualIDEController"><key>200</key><deviceInfo><label>IDE 0</label><summary>IDE 0</summary></deviceInfo><busNumber>0</busNumber></device><device xsi:type="VirtualIDEController"><key>201</key><deviceInfo><label>IDE 1</label><summary>IDE 1</summary></deviceInfo><busNumber>1</busNumber></device><device xsi:type="VirtualPS2Controller"><key>300</key><deviceInfo><label>PS2 controller 0</label><summary>PS2 controller 0</summary></deviceInfo><busNumber>0</busNumber><device>600</device><device>700</device></device><device xsi:type="VirtualSIOController"><key>400</key><deviceInfo><label>SIO controller 0</label><summary>SIO controller 0</summary></deviceInfo><busNumber>0</busNumber></device><device xsi:type="VirtualMachineVideoCard"><key>500</key><deviceInfo><label>Video card </label><summary>Video card</summary></deviceInfo><controllerKey>100</controllerKey><unitNumber>0</unitNumber><videoRamSizeInKB>8192</videoRamSizeInKB><numDisplays>1</numDisplays><useAutoDetect>false</useAutoDetect><enable3DSupport>false</enable3DSupport></device><device xsi:type="VirtualKeyboard"><key>600</key><deviceInfo><label>Keyboard </label><summary>Keyboard</summary></deviceInfo><controllerKey>300</controllerKey><unitNumber>0</unitNumber></device><device xsi:type="VirtualPointingDevice"><key>700</key><deviceInfo><label>Pointing device</label><summary>Pointing device; Device</summary></deviceInfo><backing xsi:type="VirtualPointingDeviceDeviceBackingInfo"><deviceName></deviceName><useAutoDetect>false</useAutoDetect><hostPointingDevice>autodetect</hostPointingDevice></backing><controllerKey>300</controllerKey><unitNumber>1</unitNumber></device><device xsi:type="VirtualLsiLogicSASController"><key>1000</key><deviceInfo><label>SCSI controller 0</label><summary>LSI Logic SAS</summary></deviceInfo><controllerKey>100</controllerKey><unitNumber>3</unitNumber><busNumber>0</busNumber><device>2000</device><hotAddRemove>true</hotAddRemove><sharedBus>noSharing</sharedBus><scsiCtlrUnitNumber>7</scsiCtlrUnitNumber></device><device xsi:type="VirtualDisk"><key>2000</key><deviceInfo><label>Hard disk 1</label><summary>41,943,040 KB</summary></deviceInfo><backing xsi:type="VirtualDiskFlatVer2BackingInfo"><fileName>[BI-HP502002_DS02] TMP-SERVER2019/TMP-SERVER2019.vmdk</fileName><datastore type="Datastore">datastore-89</datastore><diskMode>persistent</diskMode><split>false</split><writeThrough>false</writeThrough><thinProvisioned>false</thinProvisioned><uuid>6000C29c-fc11-2ed1-6bb5-76222c094610</uuid><contentId>56406faf5a4cb2ce64dbd8052e46ac39</contentId></backing><controllerKey>1000</controllerKey><unitNumber>0</unitNumber><capacityInKB>41943040</capacityInKB><shares><shares>1000</shares><level>normal</level></shares><storageIOAllocation><limit>-1</limit><shares><shares>1000</shares><level>normal</level></shares></storageIOAllocation></device><device xsi:type="VirtualEthernetCard"><key>4000</key><deviceInfo><label>Network adapter 1</label><summary>DVSwitch: 50 14 5e 94 63 0d b9 13-f0 0e b3 ec 38 d4 8d 24</summary></deviceInfo><backing xsi:type="VirtualEthernetCardDistributedVirtualPortBackingInfo"><port><switchUuid>50 14 5e 94 63 0d b9 13-f0 0e b3 ec 38 d4 8d 24</switchUuid><portgroupKey>dvportgroup-48</portgroupKey><portKey>38</portKey><connectionCookie>1850717010</connectionCookie></port></backing><connectable><startConnected>true</startConnected><allowGuestControl>true</allowGuestControl><connected>false</connected><status>untried</status></connectable><controllerKey>100</controllerKey><unitNumber>7</unitNumber><addressType>assigned</addressType><macAddress>00:50:56:94:a9:e3</macAddress><wakeOnLanEnabled>true</wakeOnLanEnabled></device><device xsi:type="VirtualMachineVMCIDevice"><key>12000</key><deviceInfo><label>VMCI device</label><summary>Device on the virtual machine PCI bus that provides support for the virtual machine communication interface</summary></deviceInfo><controllerKey>100</controllerKey><unitNumber>17</unitNumber><id>379848677</id><allowUnrestrictedCommunication>false</allowUnrestrictedCommunication></device><device xsi:type="VirtualController"><key>14000</key><deviceInfo><label>USB xHCI controller </label><summary>USB xHCI controller</summary></deviceInfo><controllerKey>100</controllerKey><unitNumber>23</unitNumber><busNumber>0</busNumber></device><device xsi:type="VirtualController"><key>15000</key><deviceInfo><label>SATA controller 0</label><summary>AHCI</summary></deviceInfo><controllerKey>100</controllerKey><unitNumber>24</unitNumber><busNumber>0</busNumber><device>16000</device></device><device xsi:type="VirtualCdrom"><key>16000</key><deviceInfo><label>CD/DVD drive 1</label><summary>ISO [BI-HP502002_LIB01] ISO/SW_DVD9_Win_Server_STD_CORE_2019_1809.1_64Bit_English_DC_STD_MLF_X22-02970.ISO</summary></deviceInfo><backing xsi:type="VirtualCdromIsoBackingInfo"><fileName>[BI-HP502002_LIB01] ISO/SW_DVD9_Win_Server_STD_CORE_2019_1809.1_64Bit_English_DC_STD_MLF_X22-02970.ISO</fileName><datastore type="Datastore">datastore-194</datastore></backing><connectable><startConnected>true</startConnected><allowGuestControl>true</allowGuestControl><connected>false</connected><status>untried</status></connectable><controllerKey>15000</controllerKey><unitNumber>0</unitNumber></device></val></propSet><propSet><name>config.instanceUuid</name><val xsi:type="xsd:string">5014c150-d017-b9a3-16ba-11964e654613</val></propSet><propSet><name>config.uuid</name><val xsi:type="xsd:string">4214e2e1-bb31-ae04-fc5f-7ffb16a407e5</val></propSet><propSet><name>guest.disk</name><val xsi:type="ArrayOfGuestDiskInfo"><GuestDiskInfo xsi:type="GuestDiskInfo"><diskPath>C:\</diskPath><capacity>42303746048</capacity><freeSpace>28170870784</freeSpace></GuestDiskInfo></val></propSet><propSet><name>summary.config.memorySizeMB</name><val xsi:type="xsd:int">4096</val></propSet><propSet><name>summary.config.name</name><val xsi:type="xsd:string">TEMPLATE-SERVER2019-July19</val></propSet><propSet><name>summary.config.numCpu</name><val xsi:type="xsd:int">2</val></propSet><propSet><name>summary.quickStats.balloonedMemory</name><val xsi:type="xsd:int">0</val></propSet><propSet><name>summary.quickStats.compressedMemory</name><val xsi:type="xsd:long">0</val></propSet><propSet><name>summary.quickStats.guestMemoryUsage</name><val xsi:type="xsd:int">0</val></propSet><propSet><name>summary.quickStats.hostMemoryUsage</name><val xsi:type="xsd:int">0</val></propSet><propSet><name>summary.quickStats.overallCpuUsage</name><val xsi:type="xsd:int">0</val></propSet><propSet><name>summary.quickStats.privateMemory</name><val xsi:type="xsd:int">0</val></propSet><propSet><name>summary.quickStats.sharedMemory</name><val xsi:type="xsd:int">0</val></propSet><propSet><name>summary.quickStats.swappedMemory</name><val xsi:type="xsd:int">0</val></propSet><propSet><name>summary.quickStats.uptimeSeconds</name><val xsi:type="xsd:int">0</val></propSet><propSet><name>summary.runtime.powerState</name><val xsi:type="VirtualMachinePowerState">poweredOff</val></propSet><propSet><name>summary.storage.committed</name><val xsi:type="xsd:long">42950471415</val></propSet><propSet><name>summary.storage.uncommitted</name><val xsi:type="xsd:long">4518056444</val></propSet><propSet><name>summary.storage.unshared</name><val xsi:type="xsd:long">42949672960</val></propSet></objects></returnval></RetrievePropertiesExResponse>
</soapenv:Body>
</soapenv:Envelope>
 19170:20200107:093928.579 End of vmware_service_get_vm_data():SUCCEED
 19170:20200107:093928.581 In vmware_vm_get_nic_devices()
 19170:20200107:093928.581 End of vmware_vm_get_nic_devices() found:1
 19170:20200107:093928.581 In vmware_vm_get_disk_devices()
 19170:20200107:093928.582 End of vmware_vm_get_disk_devices() found:1
 19170:20200107:093928.582 In vmware_vm_get_file_systems()
 19170:20200107:093928.582 End of vmware_vm_get_file_systems() found:1
 19170:20200107:093928.582 End of vmware_service_create_vm():SUCCEED
 19170:20200107:093928.582 In vmware_service_create_vm() vmid:'vm-10787'
 19170:20200107:093928.582 In vmware_service_get_vm_data() vmid:'vm-10787'

 

Comment by Aigars Kadikis [ 2020 Jan 08 ]

Hello damo2929, Ysm

From the workstation which has access to vcenter:443, can you use a Google Chrome or Firefox and hit the path where SDK is located:

https://vcenter/sdk/vimService.wsdl

Show the security tab and show us what type of encryption it uses:

The workaround posted by pingtw a couple of days ago looks promising, can you try it?

Comment by Damien Dye [ 2020 Jan 08 ]

Hi the details are as listed.

 

Comment by Aigars Kadikis [ 2020 Jan 08 ]

Thank you damo2929

I think the issue is 'TLS 1.2'. CentOS 6, CentOS 7 does not pick it up by default.

Please try the workaround provided by pingtw.

As a second workaround if you have a docker platform running, then I would suggest to kickstart a zabbix-proxy-sqlite3 build on of alpine image and try to perform the monitoring through this proxy. Does not know yet if it helps, need to try it out.

docker run --name proxy4vcenter -e ZBX_HOSTNAME=proxy4vcenter -e ZBX_SERVER_HOST=ip.of.zabbix.server -d zabbix/zabbix-proxy-sqlite3:alpine-4.4-latest
# in GUI under Administration -> Proxies, create a proxy with title 'proxy4vcenter'

For more like a solution: use Debian Stretch, Debian Buster, Ubuntu Bionic, CentOS 8 as a middle man (Zabbix proxy).

Comment by Damien Dye [ 2020 Jan 08 ]

the platform is centos 8, zabbix 4.4.4 and was logging into the vcentre and getting the XML responses from the centre in the log so am not sure it was TLS related.

however the nodes have started capturing this morning 24 hours after they was added. 

I know it's not normal to take that amount of time for 12 hosts on one vcentre and don't see why it kicked after that long. but am going to keep an eye on if the values are updating correctly.

Comment by Yasumi [ 2020 Jan 09 ]

Thank you everyone.

After all, I understand that enabling "sslv3" is an effective means.
However, this is not a good security practice.
It is recommended that Zabbix's LLD monitoring function properly even if only "TLS v1.2" is used.

I haven't tested Zabbix on RHEL8 yet, can the latest OS overcome this problem?

Comment by Aigars Kadikis [ 2020 Jan 09 ]

Ysm

On recent operating systems it must work out from the box. But if it does not work, we will increase the priority of this bug report and find a way to work it out.

damo2929

Are you sure the log file is free of VMware related errors in the next 3 hours when you just registered vCenter host in Zabbix? For the statistics can you please mention how many hosts, data stores, hypervisors you are running inside vCenter?

Comment by Yasumi [ 2020 Feb 05 ]

I will report.

After building Zabbix4.4 on RHEL8.1
Successfully acquired VMware API information.

 

This is because older OSs do not support TLSv1.2.
In order to support TLSv1.2, the OS Curl version must be 7.34.0 or higher and the OpenSSL version must be 1.0.1 or higher.

 

Thanks.

[root@ZabbixServer ~]# cat /etc/redhat-release
Red Hat Enterprise Linux release 8.1 (Ootpa)

[root@ZabbixServer ~]# curl --version
curl 7.61.1 (x86_64-redhat-linux-gnu) libcurl/7.61.1 OpenSSL/1.1.1c zlib/1.2.11 brotli/1.0.6 libidn2/2.2.0 libpsl/0.20.2 (+libidn2/2.0.5) libssh/0.9.0/openssl/zlib nghttp2/1.33.0
Release-Date: 2018-09-05
Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps pop3 pop3s rtsp scp sftp smb smbs smtp smtps telnet tftp
Features: AsynchDNS IDN IPv6 Largefile GSS-API Kerberos SPNEGO NTLM NTLM_WB SSL libz brotli TLS-SRP HTTP2 UnixSockets HTTPS-proxy PSL Metalink

[root@ZabbixServer ~]# curl -k -v -u [email protected] https://***.***.***.***/sdk
Enter host password for user '[email protected]':
* Unwillingly accepted illegal URL using 1 slash!
*   Trying ***.***.***.***...
* TCP_NODELAY set
* Connected to ***.***.***.*** (***.***.***.***) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/pki/tls/certs/ca-bundle.crt
  CApath: none
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384
* ALPN, server did not agree to a protocol
* Server certificate:
*  subject: CN=ZabbixServer; C=US
*  start date: Aug 18 03:00:29 2016 GMT
*  expire date: Aug 13 03:00:29 2026 GMT
*  issuer: CN=CA; DC=vsphere; DC=local; C=US; ST=California; O=ZabbixServer; OU=VMware
*  SSL certificate verify result: unable to get local issuer certificate (20), continuing anyway.
* Server auth using Basic with user '[email protected]'
> GET /sdk HTTP/1.1
> Host: ***.***.***.***
> Authorization: Basic XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
> User-Agent: curl/7.61.1
> Accept: */*
>
< HTTP/1.1 404 Not Found
< Date: XXXXXXXXXXXXXXXXXX GMT
< Connection: close
< Content-Type: text/plain; charset=utf-8
< Content-Length: 0
<
* Closing connection 0
* TLSv1.2 (OUT), TLS alert, close notify (256):
Comment by Aigars Kadikis [ 2020 Mar 25 ]

In later versions of vCenter (for example 6.7), it offers to establish a transport using TLSv1.2.
To have this communication it's required to have a newer libcurl version for the zabbix backend server.
This is working out from box on CentOS 8, Debian 10 (buster).

If the master server is hard to update, then can use a Zabbix proxy server with a modern OS to perform these checks. Kickstarting a Zabbix proxy via docker container could be handy.

The second workaround is to modify settings at vCenter

In ESXi 6.7 Web Console -> System -> Advances Setting -> Search UserVars.ESXiVPsDisabledProtocols Default Value is sslv3,tlsv1,tlsv1.1

Edit Option and Value Change to sslv3 than Save.

Log in ESXi Console -> Troubleshooting Options -> Restart Managements Agents

Generated at Sun Apr 27 10:58:14 EEST 2025 using Jira 9.12.4#9120004-sha1:625303b708afdb767e17cb2838290c41888e9ff0.