Additionally, some items are duplicated. First are discovered with VMware template and second with OS template, e.g. filesystems, network interfaces etc.

Very interresting. I would like to add VMWare informations on a host not created with autodiscovery.
For example, I have my host "Test" configuring with an IP 192.168.2.2 with a zabbix agent. It's a VM, so I attach my VMWare information about this VM on this host. It depends of the ESXI (trigger dependency), have 2 processors, etc.
If I move or delete my VM, my host doesn't disapear because has informations about zabbix agent (or another way).

Maybe some fusion with two hosts... (the discovered host, the VM and the "fixed" host with zabbix agent informations)

+1

Two major issues :

• the VMWare discovery feature needs to collect the IP address or DNS name of the VMs, in order to be able to poll a Zabbix agent inside the VM for example.
• the created VMs are mostly readonly : you can't link them to a template, except at the VM discovery. This means that you can link a template, but only to all discovered VMs, thus not allowing to differentiate between prod/dev servers, app/db servers, etc.

or at least, it would be great to add a second IP address for the discovered VM to link the zabbix agent.

+1

This is pretty critical to me actually. Most of our server infrastructure exists as VM's now and I kinda need to be able to tie multiple templates to a discovered host so I can make use of the Zabbix agent within the VM's. We have a mix of Windows and UNIX VM's so the hosts need to be unique.

A way of converting an autodiscovered host to a standard host in an easy fashion would be nice.

+1

Same Problem here. I have a lot of host duplicates right now, because I cannot edit individual discovered virtual machines (Macros, Agent and SNMP interfaces) and link additional templates to them. Unfortunately the workaround with the duplicates causes confusion and it decreases the usability of the VMware monitoring solution.
Besides Zabbix Agent we also need some SNMP and External Checks for singles virtual machines.

Hi,

it's open source and we are all encouraged to fix issues we encounter. But before reinventing a wheel for an issue that's 18 months old: has anyone already started to work on this issue or established an acceptable workaround? Does this still affect Zabbix 3? Or ist there a solution we could possibly backport?

Thanks so far!
Yves

yves.vogl, considering current ticket status and the fact of not being found on the roadmap, I'd not expect it to be done for 3.0 or any other release.
If someone else has started to work on this, this should actually be found in ZABBIX Forums or on http://zabbix.org.

Because an implementation might not be trivial, I believe it's unlikely to be done without proper (co-)sponsoring.

(1) both active and passive agent operations should be verified (active specifically requested in ZBXNEXT-2876)

Also looking for a workable solution.
Now I'm full cloning the discovered VM's so it allows me to add templates like zabbix agent and windows os.
Just leave the hostname/visible name and add "new" to it. After that you can delete the Discovered original VM.

I'm wondering if this can be solved by using Host prototypes.

Another solution could be to solely rely on passive checks. Here you need to modify the existing templates to enable those items you'd like to trap.

Unfortunately I don't think I can use the VMware monitoring feature because of this:

• All hosts in Zabbix use the FQDN as their hostname
• Our VMware names don't match up with the FQDN.
• I definitely don't want two Zabbix host objects that refer to the same system

I'm definitely curious if Host prototypes would work, instead of using the UUID use the FQDN.

Hi Doug,

I'm not sure if host prototypes would solve your problem. As far as I can see there's no IP detection (which also requires the VMware Guest Tools to be installed). But you need at least the IP (or a resolvable FQDN) that will be used as service endpoint for the agent.

I wanted to add a technical thought, to see if this could help with this specific issue.

Currently, the VMs are tracked by their instance UUID ('InstanceUUID'). This information is not immediately available to the agent in the VM. However, I am finding that the agent and vCenter both have access to the UUID property ('UUID') of the VM. It appears as a UUID formatted string in vCenter and as 'VMware-xx xx xx xx xx xx xx xx-xx xx xx xx xx xx xx xx' as the guest's serial number. I have confirmed this on CentOS 7 (file: /sys/devices/virtual/dmi/id/product_serial) and in Windows (WMI: Win32_BIOS.SerialNumber).

I know this doesn't directly address being able to add additional agents for the hosts created with the templates, but I'm hoping it can help with identifying a linkage between the agent and the VM.
Greetings,
Mike

It would be possible to use the hostmetadata to report the UUID info back to zabbix server if we're using autoregistration.
Or, you could make it an item and use network discovery to query it.

We'll still need a way to merge the two though.

Hi,
you can add a template to a discovered by using the mass modification tool from host list.
Select the host you want to change, use the mass modification button to add your new template.
The last problem is about the interface, if we can tweak zabbix to use the discovered ip adress as agent interface then this problem should be solved.

The trick with mass modifiication button might be seen as a bug.

After a lot of research the solution proposed by Michael Wilmes is not working for vCenter implemtation because "The instanceUUID is a new property that was introduced in the vSphere 4.0 API to provide an easy way to uniquely identify a VM" (https://blogs.vmware.com/vsphere/2012/02/uniquely-identifying-virtual-machines-in-vsphere-and-vcloud-part-1-overview.html). Actually Zabbix is using InstanceUUID and not SMBIOS UUID for the hostname VM.
Someone have other possible solutions?
Thanks

I don't think you should stop using the instanceUUID for identifying VMs, but what I was suggesting is that both the VM and vCenter have access to the UUID and can this might be used as a way to chain together the host agent running in the VM and the vCenter. Maybe add a field to the host data in the Zabbix DB and insert the shared identifier there? Update the Zabbix agent to track and send the BIOS UUID as a tracked value? Possibly use the BIOS UUID as a third option for uniquely identifying hosts during discovery?

If you point me to a code repository and let me know what problems you're seeing, I would like to help. I have some coding experience, and it may be more productive for me to look at what you're seeing instead of just offering random solutions.

+1

@Michael Wilmes - can you propose a way to extract the UUID using the hostmetadata value in zabbix_agentd.conf to enable this in auto-registration of agents?

Can't seem to fathom a way to achieve this and find it crucial for our planned use case.

Kind regards

According to post https://communities.vmware.com/thread/204268?start=0&tstart=0 dmidecode can show UUID in Serial number.
> # dmidecode |grep 'Serial Number: VM'
> Serial Number: VMware-42 3c c8 5c c2 c1 4e ec-e7 ff 0f fd ec 5a 0d 6d
And looks like this number is hidden in /sys/devices/virtual/dmi/id/product_serial
Unfortunately it requires root access

@boreasops - I cannot find an option that I am happy with. Based on my research, the serial number is root-readable only. We would need to write a program with setuid to root in order to read the number, or run the Zabbix agent as root. Writing the program is probably the least painful and most secure option. It would effectively need to output the data read from /sys/devices/virtual/dmi/id/product_serial, be set uid root, but be executable by only the zabbix user group. The executable, imo, should only read from that one file and take no arguments. At that point, the agent can get the UUID and we can begin to use it.

Another option might be to talk with open-vm-tools/VMware tools and try to get the information through that, but I have no idea if it would work or how to do it.

As a test pilot case, I made a copy of cat that I setuid and changed the group on that executable. I DO NOT RECOMMEND ANYONE DO THIS FOR ACTUAL USE - THIS IS A PROOF OF CONCEPT. Here's my output:
[root@zabbix-p1]# cp -p /bin/cat /usr/local/bin/scat
[root@zabbix-p1]# chgrp zabbix /usr/local/bin/scat
[root@zabbix-p1]# chmod o-rx /usr/local/bin/scat
[root@zabbix-p1]# chmod u+s /usr/local/bin/scat
[root@zabbix-p1]# exit

login as: user1
user1@zabbix-p1's password:
Last login: Wed Feb 15 11:06:40 2017 from xxxx
[user1@zabbix-p1 ~]$ /usr/local/bin/scat /sys/devices/virtual/dmi/id/product_serial
-bash: /usr/local/bin/scat: Permission denied
[user1@zabbix-p1 ~]$ sudo su - zabbix -s /bin/bash
[sudo] password for user1:
[zabbix@zabbix-p1 ~]$ /usr/local/bin/scat /sys/devices/virtual/dmi/id/product_serial
VMware-42 0b 35 58 ce d6 d4 53-db ba 1c e9 59 ff b1 b5

Greetings,
Mike

Hi Guys,
I think i have missed something, as i wrote above:
citation
Actually Zabbix is using InstanceUUID and not SMBIOS UUID for the hostname VM.
citation

Is not possible to use the workaround of "dmidecode" or "cat /sys/devices/virtual/dmi/id/product_serial", maybe i'm wrong or I have missed some parts.
Please give me some hint about that or some Zabbix dev could confirm my sentence, only to be sure that this is not possible.
Thanks,
Best regards

@Dimitri Bellini,
We are not disagreeing with you that the VM is registered by InstanceUUID. However, we are trying to identify ways that the VM can be identified from both sides. I think I may have a working solution.

The problem with SMBIOS, as @Dimitri Bellini has mentioned, is that multiple VMs in the same instance of vCenter may have the same SMBIOS value. This happens when a VM is cloned without being customized. The only truly exact manner to identify a VM is by its instance ID.

I've come up with a way to work around that- the guestinfo VM configuration values. This is a mechanism supported by VMware tools and open-vm-tools that allows the VM and vCenter to exchange data. The guestinfo configuration paramaters are shared between the VM and vCenter and can be read and set by both. We can pass a value around that uniquely identifies the VM that is accessible by both the agent and through vCetner. The Instance ID would be required to get the value from vCenter, completing the loop.

I would not advise placing the InstanceID into the guestinfo space. While it would be easiest, the blogs I have been reading hint that VMware intentionally did not make this value available in the host specifically to isolate the VM from the host- the VM knows nothing of the host it is running on. Also, this would require that Zabbix have write access to vCenter, which IMO is a bad thing.

We can, however, have the agent write its own UUID to the guestinfo space and send that over as a value directly to Zabbix. Then Zabbix can read the value from the VM's configuration settings and match the two up. The only challenge with this is that the guestinfo value is writeable by any user account that can execute vmtoosd, which is world executable. Any user on the VM could change the stored value.

Here is the command I used to set the value on my Linux VM:

[user1@zabbix-p1 ~]$ vmtoolsd --cmd 'info-set guestinfo.zabbix aaaa-bbbb-cccc-dddd-eeee'

Here is the PowerCLI commands I ran to retrieve the value. This can be directly translated over into an API call in Zabbix:

 C:\> $vcenter = Connect-VIServer -Server vcenter-l1.avc.edu -Credential $vmcred 
 C:\> $vm = get-vm zabbix-p1 
 C:\> $vm.ExtensionData.Config.ExtraConfig.GetEnumerator()|?{$_.Key -eq 'guestinfo.zabbix'} 
 Key              Value                   
 ---              -----                   
 guestinfo.zabbix aaaa-bbbb-cccc-dddd-eeee 

@Michael Wilmes
Hi,
I'm sorry but my post was related to clarify that Zabbix is using InstanceUUID as "HostName" on and not other UUID. If you have a good workaround please post it because we need it
The solution you posted using VMWare Tools is not viable on our customer because, as you wrote, it require some not enterprise practice.
Thanks so much for you digging on problem!

I think that I have suggested something that may work well, if we can identify a way to securely use the guestinfo mechanism as a means to complete the loop from Zabbix server to Zabbix Agent to guestinfo to vCenter to Zabbix server. This would involve confirming the data written to the guestinfo mechanism as authentic.

My suggested process works like this:
1. The Zabbix agent generates its own agent UUID. On first run, this would be saved for reuse. On later runs, the UUID would be loaded when the agent is started up.
2. The Zabbix agent would try to update the guestinfo.zabbix key on the host by running the vmtoolsd command on the host. This relies on VMware tools or open-vm-tools being installed, but given that this is a VM running in VMware, this is acceptable expectation and requirement. This step should run at regular intervals, but not necessarily every time the agent sends data to the Zabbix server.
3. If the agent was able to update the guestinfo value in step 2 (the vmtoolsd executable was able to run and update the value successfuly), then the Zabbix agent transmits a flag that this agent is running in a VM and the value of this UUID to the Zabbix server. This could be a part of any regular data submission to the Zabbix server, or a special data transmission. Otherwise the flag is sent that this agent is not able to be associated in the VM environment.
4. The Zabbix server would receive the agent UUID from an update and check to see if this is new or changed compared to any previous InstanceUUID associations.
4a. If the agent UUID is not already associated with any existing vCenter instance, the Zabbix Server checks against VM instances through the vCenter API looking for a VM that has the matching UUID in its guestinfo values. If there is exactly one match, Zabbix associates the Zabbix agent UUID with the vCenter InstanceUUID. If there is not exactly one match, the Zabbix server sends a trigger to the Zabbix agent requesting that a new UUID be generated. This process then restarts at step 1 on the next data submission from the Zabbix agent.
4b. If the agent UUID is already associated with an Instance UUID, the Zabbix server confirms that the agent UUID is still what is expected at the VM with the associated Instance UUID. If the expected agent UUID matches what is in the VMs guestinfo data, then the Zabbix server proceeds as normal. If the agent UUID and instance UUID do not match, then the Zabbix server breaks any internal associations involving both the agent UUID and InstanceUUID, and sends a command to the Zabbix agent to regenerate a new agent UUID. This process restarts with step 1.

What this is doing:

If the agent is not associated with a VM on the Zabbix server, the Zabbix server tries to identify what VM the agent is running on. If the value has been tampered with or the server has been cloned, creating duplicate agent UUIDs, the Zabbix server forces the agent to generate a new UUID so it can be reassociated.

Otherwise the Zabbix server can associate the data coming in from the agent with the VM, and the agent's data can be stored with the VM's host data on the Zabbix server.

When an association is made, any previous data saved by the agent as a host can be migrated over to the host object identified by the VM for continuity. Any created host for the agent can be deleted after assigned templates and saved data is migrated into the VM's host object.

If it appears that the UUID is tampered with, the server does not try to continue to associate the agent's data with the VM. This should then act like a new, unknown Zabbix agent is connecting to the server. If a new host object is created, it would be cleaned up when then agent is reassociated with a VM.

While I like the idea of merging the configs, is there merit in instead allowing people to extend the vmware discovery templates to add guest hostname and IP fields? This seems like it would be a relatively easy partial fix to implement. Hostname and IP data should be available via the sdk and VMWare tools for most VMs.

I am trying to avoid running agents on every machine, so in my environment it would be sufficient for us to simply target ping, WMI, etc checks against a guest IP or hostname. Perhaps my setup is unique enough that this is not of value but I thought I'd throw it out there.

As of Zabbix 5.4 we now are able to monitor just well VMWare guest and also it could be possible to "merge" Guest VMs and traditional Agent based hosts using Zabbix Agent Active and

or

Could be nice to provide an option to decide to leave "opened" the discovered hosts, so in this way we can easily add new templates, macros etc..
As i have tested, there is no issues to leave "VMWare guest Template" on a of cloned VM Guest host and the new host is still receive the VMWare metrics.
@ZabbixDev Is it possibile?
Thanks

See also this reddit discussions about the same problem: https://www.reddit.com/r/zabbix/comments/oh49mg/zabbix_vmware_agent_monitoring/h4ok6t5/?context=8&depth=9

In 5.0 we introduced LLD overrides, which allow linking different templates based on LLD macros. In 5.2 we added custom interfaces for host prototypes and also added #VM.IP and VMWare FQDN Template. All this should address initial concerns (which date back to 2.2) about having two hosts to monitor a single VM. Any objections if we consider it done and close it?

Alex, are you saying we can modify the Vmware template to add a standard agent interface on discovered VMs? If so that's great, although I'd love to see this built in by default because I don't want to have to maintain a customized version of that template and merge my changes in for every release. 

Hi Alex,
from my point of view "no" you can not say the problem is closed, why? Although a lot of work as been made since Zabbix 2.2 we still need some features to implemented to make a "good complete" VMWare monitoring/Zabbix Agent monitoring.
For instance in a lot of customers we faced this simple scenario.
They would like to discover all VMs of VMware Cluster maybe using the default Zabbix Template for the guests VMs
The second request is: "I can monitoring Application like Apache,Oracle, MSSQL or some other metrics?"
At the moment is not possible or maybe you can using "overrides" but is not good for all cases and could be tricky.
The only viable option is to clone the "discovered VM" and assigned a template.
Why is not possible because at the moment the "VMs" is not editable for Interfaces and Templates.
My final considerations are, thanks for all the great work until now and please try to find a possible solution for this such scenarios
Thanks so much

Kalimulin In my very personal opinion, this issue can be closed, after ZBXNEXT-5517, ZBXNEXT-7591 and ZBXNEXT-7523 are resolved. Adding the ability to alter templates, macros and tags of Host prototype will be the ultimate solution.

I am going to close this ticket since Zabbix 6.2 covering requested use cases is out. Any objections?

I'm happy. If everything works as promised with 6.2, I'm currently not aware of any more issues.
EDIT: Actually that's not true, there are two more documentation issues, that need to be adressed, now more than before, because of all the additions from 6.2:
ZBXNEXT-7794: Zabbix Documentation doesn't cover Host prototypes
ZBXNEXT-7794: Add example to Host prototypes

So, just to confirm - in 6.2 if I link a template to a discovered host, can I have agent-based checks that are separate from the VMware discovery template checks? If so, that looks like a perfect solution.

I haven't had a chance to update to 6.2 yet so I'm just looking for clarification.

astadnick Yes. In Zabbix 6.2 you can link a new template with agent-based checks to a discovered hosts having other templates. Moreover you may also add new host tags and fully control macros.

Awesome, that's great news! I agree that this can be closed. Looking forward to 6.2 in my production environment!

I am closing it then. Thanks for the feedback!

Apologies for commenting on a closed topic, but I'm wondering if the inverse situation is possible. We have an existing 6.2.2 environment with thousands of Zabbix Agents. If I add the VMware FQDN template, is it possible to have the new VMware Guest metrics merge with an already-existing Zabbix Active Agent?

In other words, we have:

webserver.example.com = Linux or Windows VM, monitored using only Active Agent items. The FQDN is webserver.example.com, BUT the VM name might just be "webserver".

Is there a way to have Zabbix discover this as a VM, and link the VMware Guest template to the webserver.example.com Zabbix host?

Thank you!