[ZBX-22697] VMware tags are not reliably collected during VM discovery Created: 2023 Apr 19 Updated: 2024 Apr 10 Resolved: 2023 Jun 09 |
|
| Status: | Closed |
| Project: | ZABBIX BUGS AND ISSUES |
| Component/s: | Proxy (P), Server (S) |
| Affects Version/s: | 6.4.1 |
| Fix Version/s: | 6.4.4rc1, 7.0.0alpha2, 7.0 (plan) |
| Type: | Problem report | Priority: | Trivial |
| Reporter: | Nicola Mauri | Assignee: | Michael Veksler |
| Resolution: | Fixed | Votes: | 2 |
| Labels: | Tags, VMware | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Environment: |
VMware vCenter Server 7.0.2 build-18455184 |
||
| Attachments: |
LatestData1.png
LatestData2-after-zabbix-restart.png
ZBX-22697-6.4.patch
vm-tags-item-timeout.png
|
| Team: |  Team B |
| Sprint: | Sprint 100 (May 2023), Sprint 101 (Jun 2023) |
| Story Points: | 1 |
| Description |
|
In the JSON generated by vmware.vm.discovery all discovered VMs have an empty "tags" element, even if vmware tags are actually defined in vSphere. It seems taht restarting zabbix_proxy, in most cases, makes appear tags again in the JSON. Since we use vmware tags to filter VMs that need to be discovered and monitored, this issue makes hosts to be randomly created and deleted from zabbix.
Steps to reproduce: 1. Create an host and apply VMware FQDN template 2. In the vmware.vm.discovery[] discovery rule: add a Javascript preprocessing step in order to write lld output to the logs var obj = JSON.parse(value);
3. Observe JSON output in zabbix_proxy.log: "tags" element is empty. 843683:20230418:100956.685 VMdiscovery: .... ]
4. Restart zabbix and look again at zabbix-proxy.log: "tags" element is now properly collected. # systemctl restart zabbix-proxy zabbix-proxy.service - Zabbix Proxy 3819356:20230418:110956.470 VMdiscovery:
|
| Comments |
| Comment by Edgar Akhmetshin [ 2023 Apr 20 ] |
|
Hello Nicola, We need DebugLevel=4 for "vmware collector" process. Also to get the error happened with tags could you add to the VM template this item: vmware.vm.tags.get[url,uuid] https://www.zabbix.com/documentation/current/en/manual/vm_monitoring/vmware_keys This item should return error, if tag is not handled properly. Regards, |
| Comment by Nicola Mauri [ 2023 Apr 20 ] |
|
Hello Edgar, thanks for your suggestions. I'm working to provide a simple, minimale case with DebugLevel. In the meantime: I added the vmware.tags.get item and it returns "Timeout was reached" (see attached screenshot). After restarting Zabbix proxy, the item start to be properly collected, for some time at least. Please note that this installation is also affected by Thanks |
| Comment by Michael Veksler [ 2023 Apr 21 ] |
|
Hello Nicola, case CURLE_OPERATION_TIMEDOUT:
return "Timeout was reached";
Is this possible due to vmware session expiration timeout? Regards, |
| Comment by Nicola Mauri [ 2023 Apr 21 ] |
|
Hello Michael, well, in theory it could be. But I cannot explain why this happens always and only during tags retrieving and not when gathering other vmware metrics. |
| Comment by Michael Veksler [ 2023 Apr 21 ] |
|
Hello Nicola, Tags subsystem is not soap but rest endpoint with different authentication process I still can't imagine what's going on, but still:
|
| Comment by Nicola Mauri [ 2023 Apr 24 ] |
|
Hello Michael
I suspect that it could be caused by SourceIP option and We'll upgrade to 6.4.1 and let you know. Thanks
|
| Comment by Nicola Mauri [ 2023 May 11 ] |
|
Hello, we upgraded to 6.4.2 but unfortunately issue still persists. Although we are not able to consistently reproduce the problem, it always recovers after restarting zabbix-proxy daemon.
|
| Comment by Michael Veksler [ 2023 May 11 ] |
|
Hello bitmind, Next time when we catch the "weird" state, change the debug level to trace only for the "vmware collector" I need to understand - which phase is not working: authentication or data query (I suspect that authentication phase) zabbix_server -R log_level_increase="vmware collector" zabbix_server -R log_level_increase="vmware collector" Try to find "vmware_service_rest_authenticate" string and all subsequent lines. |
| Comment by Nicola Mauri [ 2023 May 18 ] |
|
Hi Michael here are some excerpts from the debug log. As far as I can understand, all calls complete with SUCCESS status, however tags list is still empty in the JSON.
zabbix_proxy.log: ... |
| Comment by Michael Veksler [ 2023 May 19 ] |
|
Hello Nikola, If possible - run proxy with additional logs: diff --git a/src/zabbix_server/vmware/vmware_rest.c b/src/zabbix_server/vmware/vmware_rest.c
index 5d70d400b05..d06d293049f 100644
--- a/src/zabbix_server/vmware/vmware_rest.c
+++ b/src/zabbix_server/vmware/vmware_rest.c
@@ -633,6 +633,9 @@ static int vmware_rest_post(const char *fn_parent, CURL *easyhandle, const char
CURLcode err;
CURLoption opt;
+ if (NULL != fn_parent)
+ zabbix_log(LOG_LEVEL_TRACE, "%s() POST request: %s", fn_parent, request);
+
if (CURLE_OK != (err = curl_easy_setopt(easyhandle, opt = CURLOPT_POST, 1L)) ||
CURLE_OK != (err = curl_easy_setopt(easyhandle, opt = CURLOPT_POSTFIELDS, request)))
{
and show me output of tags from one host (this is example of curl script for tags requests) h=vcenter7.host u='[email protected]' p='password' pu='api' cg="curl -s -k -X GET" cp="curl -s -k -X POST" t=`curl -s -k -X POST https://${h}/api/session -u "${u}":"${p}" | tr -d '"'` echo "--- token:[$t]" echo "--- category list:" $cg "https://${h}/${pu}/cis/tagging/category" -H "vmware-api-session-id: ${t}"|jq echo "--- tags list:" $cg "https://${h}/${pu}/cis/tagging/tag" -H "vmware-api-session-id: ${t}"|jq echo "--- tags list of host-576209" $cp "https://${h}/${pu}/cis/tagging/tag-association?action=list-attached-tags" -H "vmware-api-session-id: ${t}" -H 'Content-type: application/json' -d '{ "object_id": { "id": "host-576209", "type": "HostSystem" } }'|jq echo "--- logout" curl -X DELETE 'https://vcenter7.zabbix.sandbox/api/session' -H "vmware-api-session-id: ${t}" |
| Comment by Nicola Mauri [ 2023 May 19 ] |
|
Hi Michael, thanks four your feedback. 1) No vSphere tags are defined on Hypervisors (vmware "hosts") at the moment. 2) Yes. Asking for tags returns "Timeout was reached" (see attached latest data screen) 3) run proxy with additional logs: at the momento we can't recompile zabbix, as this is a production environment. But, if necessary, we can try to setup a test environment. 4) show me output of tags from one host: This is the result. I'm not sure about that: shouldn't we ask for VM tags instead of host tags? --- token:[a3612aee97d83de7dd62ef1b8*********]
|
| Comment by Michael Veksler [ 2023 May 23 ] |
|
Hello Nikola, Be so kind to test the patch ZBX-22697-6.4.patch I think I found the problem and thanks for the help. (But the cause of the error is unreliable communication network, I recommend checking the network - after testing the patch |
| Comment by Nicola Mauri [ 2023 May 23 ] |
|
Thanks Michael. Please allow a few days to recompile zabbix and setup a proper test environment. I'll update this issue as soon as possible. |
| Comment by Dimitri Bellini [ 2023 May 29 ] |
|
Hi Michael, |
| Comment by Michael Veksler [ 2023 May 30 ] |
|
Hi dimitri.bellini, We do vmware monitoring via https/443 - you are right cause of the error is that it not erased when network issue is restored, but it means that at least 1 time curl got the "timeout" error (not application level)! |
| Comment by Dimitri Bellini [ 2023 May 30 ] |
|
Hi Michael, PS: Maybe I did not understand how it works |
| Comment by Michael Veksler [ 2023 Jun 01 ] |
|
Hi dimitri.bellini, This is the right question. The answer is based on our vmware information collection architecture. We are trying to prevent the vc from being destroyed if any user configures to collect some metrics every second. Therefore, we collect all the data(VMwareFrequency & VMwarePerfFrequency) in our cache and generate a response based on our cache. As for this case, I forgot to remove the error while restoring the network connection from our cache and all responses returned with an error, despite the fact that all data about tags was in memory (our dev network is very reliable |
| Comment by Dimitri Bellini [ 2023 Jun 01 ] |
|
ahhaha, ok understood! |
| Comment by Michael Veksler [ 2023 Jun 09 ] |
|
Available in:
|
)