CentOS 7
vCenter 6.5 and 6.7

looks like this: https://support.zabbix.com/browse/ZBX-14882
but the proxy nor the server are crashing.

Hello,

Can you please reproduce the issue and provide Zabbix proxy logs with increased log level for vmware collector. (zabbix_proxy -R log_level_increase="vmware collector")

You indicate that this issue occurs when adding vCenter for monitoring by proxy, please advice does it means that issue not persists while monitoring vCenter by server?

So, actually have tried that. But the server does not show anything else in relation to these messages. I have tried to increase log level from all components and the message seems unrelated to any function. The server specifically does not have any vmware collectors enabled.

My server does not monitor anything directly, he just work receiving data from the proxies. I have no means to make the server monitor a vCenter directly at this moment.

When raising the verbosity, it does not show where those come from (apparently) and they are also not suppressed from the log if I decrease the log level to 0, which is also weird.

The other thing that raised a suspicion is the vpxd size problem (Error of query maxQueryMetrics: 'config.vpxd.stats.maxQueryMetrics' is invalid or exceeds the maximum number of characters permitted..) which is mitigated in some of the vcenters but not in others. But I am unable to correlate the message on the server, with the vpxd message that appear in some of the proxies. Apparently also, there are much more messages on the server, than those being written on proxies logs.

So you can have an idea:

grep noname.xml zabbix_server.log | wc -l
36370

grep -v noname.xml zabbix_server.log | wc -l
84358

cat zabbix_server.log | wc -l
120818

Let me know if there are any ideas on how to debug this.

In regard to the logs, I would prefer to have a more private channel to send them, as they might contain sensitive data from our environment and it is not a good idea to use this forum to share it. Let me know.

Hi, sorry for confusion but i never requested data from zabbix_server.log, only from zabbix_proxy. If you increase log level for vmware collector on zabbix proxy then it shall show the full response from VMWare API in logs. For this issue it is a first step in troubleshooting. Thanks in advance.

Hi,

Let's try to gather a bit more information and then make a decision how to continue.
1. Do you in parallel with this error receive any data successfully or you don't get data at all ?
2. Can you check your zabbix proxy log by yourself that all information after label "vmware_service_retrieve_perf_counters() SOAP response" is a valid XML? Or maybe somewhere you can an error or other type of data that potentially may not be parsed on zabbix server?
3. If VMWare discovery works can you confirm if you have all VMs controlled by vCenter successfully discovered?
4. Do you have any pre-processing rules for vmware related items and discovery?
5. Do you see from which process this error is coming?

So,

1) Yes, the data comes ok, as far as I can see...

2) Doing some investigation in one of the proxies, there is no indication that broken XMLs are being parsed or shipped to the server. But that will not exclude that some other proxy is doing the same. I would need to research all the proxies to make sure there isn't such problem.

3) I only use the VM discovery for clusters and hypervisors not for VMs, and on this aspect it works perfectly. There is only one case where there is no "cluster" configuration on the vCenter and the host discovery fails to create a group for the hypervisor – "group already exists", but the message in question was appearing earlier than the "no cluster" vcenter.

4) no pre processing there/here.

5) No. As I said earlier, even if I set verbosity to 0 on server, he still spitting that message out. Which also in part, makes this whole thing quite intriguing.

I will try and prepare a better framework to investigate item (2) as much as I can... in the meanwhile, I will try and reproduce this in an ACC environment, lets see.

Hi,

I've been going through this case and came to the conclusion that very unlikely you have some XML parsing errors. Maybe it was caused by some other reasons? I also see in the error message "We trust you have received the usual lecture from the local System" following the xml parse message. This usually appears when some operation is executed with "sudo". Do you think these two rows in logs are related in no? You can actually check it in zabbix_server log by first 4 digits in a row that represent a thread. However there shall be another error/warning message following to the can't parse XML. Can you please check and verify it?

Additionally you have indicated about Error of query maxQueryMetrics, please check this thread for it: https://support.zabbix.com/browse/ZBX-14858 .

Hello!

It's been awhile, but I'd like to clarify if you have managed to reproduce the issue?

If this is no longer relevant, I will be closing this issue within 14 days, in case there's more information on this topic, feel free to submit it here.

Best Regards,