[ZBX-20356] Collecting perf_counter problem on Windows agent2 Created: 2021 Dec 15  Updated: 2024 Apr 10  Resolved: 2022 Dec 12

Status: Closed
Project: ZABBIX BUGS AND ISSUES
Component/s: Agent2 plugin (G)
Affects Version/s: None
Fix Version/s: 5.0.31rc1, 6.0.13rc1, 6.2.7rc1, 6.4.0beta5, 6.4 (plan)

Type: Problem report Priority: Blocker
Reporter: Andrei Gushchin (Inactive) Assignee: Vladislavs Sokurenko
Resolution: Fixed Votes: 16
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: PNG File Bildschirmfoto 2022-03-18 um 16.32.46.png     PNG File Screenshot 2021-12-20 at 14.04.03.png     PNG File image-2022-05-04-09-20-41-930.png     PNG File image-2022-05-04-09-22-20-428.png     PNG File image-2022-05-04-09-24-37-267.png     PNG File image-2022-05-04-09-29-15-818.png     PNG File image-2022-05-04-09-31-36-098.png     PNG File image-2022-05-04-09-32-53-109.png     PNG File image-2022-05-04-09-32-57-067.png     PNG File image-2022-05-04-09-33-58-599.png     Zip Archive zabbix_agent2-6.0.4rc1-windows-amd64-openssl-static.zip     File zabbix_agent2-6.0.4rc1-x64.msi.7z.001     File zabbix_agent2-6.0.4rc1-x64.msi.7z.002     File zabbix_agent2.log    
Issue Links:
Causes
Duplicate
is duplicated by ZBX-20378 Zabbix agent 2 crashes on Windows - p... Closed
is duplicated by ZBX-20013 Windows performance counters not work... Closed
Sub-task
part of ZBXNEXT-6935 Make sure new Zabbix (6) installation... Needs documenting
Team: Team A
Sprint: Sprint 87 (Apr 2022), Sprint 88 (May 2022), Sprint 94 (Nov 2022), Sprint 95 (Dec 2022)
Story Points: 1

 Description   

Steps to reproduce:
Hard to predict but it happen and agent not return accidently data for perf_cunter till restart agent

Here is debug

2021/12/15 14:36:09.353647 failed to execute exporter task for itemid:8513769 key 'perf_counter_en\[\\LogicalDisk(C:)\\Avg. Disk sec/Write]' error: 'No data to return.

'

2021/12/15 14:36:09.354649 check 'perf_counter_en\[\\LogicalDisk(C:)\\Avg. Disk sec/Write]' is not supported: No data to return.

2021/12/15 14:36:09.356650 executed exporter task for itemid:20689209 key 'vfs.fs.size\[C:\\ClusterStorage\\CSV0010,pused]'

2021/12/15 14:36:09.356650 executed exporter task for itemid:21400029 key 'vfs.fs.size\[C:\\ClusterStorage\\New_CSV0015,pused]'

2021/12/15 14:36:09.665915 \[UserParameter] command:' powershell -NoProfile -ExecutionPolicy "RemoteSigned" -File "c:\\mon-agent\\scripts\\microsoft.ps1" -Action "getnode" -Object "CAB-PSP-HV00084" -Key "none"' length:2 output:'Up'

2021/12/15 14:36:09.665915 executed exporter task for itemid:9962159 key 'microsoft\[getnode,"CAB-PSP-HV00084",none]'

2021/12/15 14:36:09.665915 executing exporter task for itemid:9962161 key 'microsoft\[getnode,"CAB-PSP-HV00086",none]'

2021/12/15 14:36:09.665915 \[UserParameter] executing command:' powershell -NoProfile -ExecutionPolicy "RemoteSigned" -File "c:\\mon-agent\\scripts\\microsoft.ps1" -Action "getnode" -Object "CAB-PSP-HV00086" -Key "none"'

2021/12/15 14:36:10.336272 plugin WindowsPerfMon: executing collector task

2021/12/15 14:36:10.336272 plugin Cpu: executing collector task

2021/12/15 14:36:10.336272 executing exporter task for itemid:21400030 key 'vfs.fs.size\[C:\\ClusterStorage\\New_CSV0016,pused]'

2021/12/15 14:36:10.336272 executing exporter task for itemid:20689210 key 'vfs.fs.size\[C:\\ClusterStorage\\DC_CSV_NewTemp1,pused]'

2021/12/15 14:36:10.338275 plugin 'WindowsPerfMon' collector failed: No data to return.

2021/12/15 14:36:10.338275 executing exporter task for itemid:8513770 key 'perf_counter_en\[\\LogicalDisk(C:)\\Disk Reads/sec]'

2021/12/15 14:36:10.338275 parsed performance counter \\LogicalDisk\\Disk Reads/sec

2021/12/15 14:36:10.339275 failed to execute exporter task for itemid:8513770 key 'perf_counter_en\[\\LogicalDisk(C:)\\Disk Reads/sec]' error: 'No data to return.

'

2021/12/15 14:36:10.339275 check 'perf_counter_en\[\\LogicalDisk(C:)\\Disk Reads/sec]' is not supported: No data to return.

2021/12/15 14:36:10.339275 executed exporter task for itemid:20689210 key 'vfs.fs.size\[C:\\ClusterStorage\\DC_CSV_NewTemp1,pused]'

2021/12/15 14:36:10.340275 executed exporter task for itemid:21400030 key 'vfs.fs.size\[C:\\ClusterStorage\\New_CSV0016,pused]'

2021/12/15 14:36:11.284206 \[UserParameter] command:' powershell -NoProfile -ExecutionPolicy "RemoteSigned" -File "c:\\mon-agent\\scripts\\microsoft.ps1" -Action "getnode" -Object "CAB-PSP-HV00085" -Key "none"' length:2 output:'Up'

2021/12/15 14:36:11.284206 executed exporter task for itemid:9962160 key 'microsoft\[getnode,"CAB-PSP-HV00085",none]'

2021/12/15 14:36:11.284206 plugin WindowsPerfMon: executing collector task

2021/12/15 14:36:11.285205 plugin Cpu: executing collector task

2021/12/15 14:36:11.285205 executing exporter task for itemid:20689211 key 'vfs.fs.size\[C:\\ClusterStorage\\CSV0002,pused]'

2021/12/15 14:36:11.285205 executing exporter task for itemid:9962065 key 'microsoft\[getgroup,"CAB-PSP-TDC0012",none]'

2021/12/15 14:36:11.285205 \[UserParameter] executing command:' powershell -NoProfile -ExecutionPolicy "RemoteSigned" -File "c:\\mon-agent\\scripts\\microsoft.ps1" -Action "getgroup" -Object "CAB-PSP-TDC0012" -Key "none"'

2021/12/15 14:36:11.286206 executing exporter task for itemid:21400031 key 'vfs.fs.size\[C:\\ClusterStorage\\New_CSV0017,pused]'

2021/12/15 14:36:11.287208 plugin 'WindowsPerfMon' collector failed: No data to return.

2021/12/15 14:36:11.288207 executing exporter task for itemid:8513771 key 'perf_counter_en\[\\LogicalDisk(C:)\\Disk Writes/sec]'

2021/12/15 14:36:11.288207 executed exporter task for itemid:20689211 key 'vfs.fs.size\[C:\\ClusterStorage\\CSV0002,pused]'

2021/12/15 14:36:11.288207 parsed performance counter \\LogicalDisk\\Disk Writes/sec

2021/12/15 14:36:11.291207 failed to execute exporter task for itemid:8513771 key 'perf_counter_en\[\\LogicalDisk(C:)\\Disk Writes/sec]' error: 'No data to return.

Expected:
Get data fine.

 Comments   
Comment by Colum Flannigan [ 2022 Feb 10 ]

Hey.

   I am seeing the same thing on a zabbix_agent2 5.4.9 and below is what i see in the logs. I also cannot query this data using zabbix_get

 

2022/02/10 08:58:37.046634 check 'perf_counter_en["\PhysicalDisk(0 C:)\Disk Reads/sec",60]' is not supported: No data to return.
2022/02/10 08:58:37.047632 check 'perf_counter_en["\Memory\Pages/sec"]' is not supported: No data to return.
2022/02/10 08:58:38.046633 plugin 'WindowsPerfMon' collector failed: No data to return.

 

It was working fine when i was on zabbix_agent2 5.4.3 but because of all the crashes i thought upgrading i might see less of the crashes.. Also if i run a powershell invoke command i get data

 

Invoke-Command server{Get-Counter -Counter "\PhysicalDisk(0 C\Avg. Disk Queue Length"}

 

Readings       : \\server\physicaldisk(0 c\avg. disk queue length :
                 0.00179048089439996

Comment by Eric [ 2022 Mar 02 ]

I am seeing the following on several Windows VMs running agent 2 5.0.15. An agent restart fixes the issue for a while but I have been unable to find anything in the agent logs or Windows event logs to determine what causes the issue.

2022/03/02 14:39:26.059746 check 'perf_counter[\700(_Total)\702,300]' is not supported: The handle is not a valid PDH object.
2022/03/02 14:39:27.059916 plugin 'WindowsPerfMon' collector failed: The handle is not a valid PDH object.
2022/03/02 14:39:27.060916 check 'perf_counter[\Memory\Page Faults/sec,300]' is not supported: The handle is not a valid PDH object.
2022/03/02 14:39:28.058826 plugin 'WindowsPerfMon' collector failed: The handle is not a valid PDH object.
2022/03/02 14:39:28.058826 check 'perf_counter[\memory\% Committed Bytes In Use]' is not supported: The handle is not a valid PDH object.

Comment by Daniel S. Miyagi [ 2022 Mar 09 ]

Same problem on agent2 5.0.10

Comment by Thomas Oftring [ 2022 Mar 18 ]

The problem still exists in Zabbix 6.0.1 with Active Items.

It is very annoying, the only option is to restart the agent

Comment by Onkel Titus [ 2022 Mar 24 ]

Same issue. None of the performance counters are working. There are several tickets in the past about this which were all closed or can't reproduced but the problem still exist ( Version 6.0.1 ). Seems that the performance counters shouldn't be used anymore in the Windows Templates.

Comment by Ivan Zubov [ 2022 Mar 30 ]

Got same: The handle is not a valid PDH object.

Its easy to reproduce on Zabbix server 6.0.2 and zabbix-agent2 6.0.1 (with ssl): 

Choose any windows server from hosts, unlink and clear template Windows by Zabbix agent active, click Update, choose same server, link template Windows by Zabbix agent active, after couple of minutes you will got The handle is not a valid PDH object on any perf_counter metrics.

This situation can be fixed by restart zabbix-agent2 service on host.

Comment by Aleksejs Sestakovs [ 2022 Mar 30 ]

Eradicator, Thank you for the information. I will try the described scenario.
Edit: Which Windows version do you use?

Comment by Ivan Zubov [ 2022 Apr 05 ]

Tested on Windows Server 2019 (Standart, Enterprise), Windows Server 2012 (Standart).

Comment by Aleksejs Sestakovs [ 2022 Apr 05 ]

Eradicator, Could you also provide the version numbers?

Comment by Ivan Zubov [ 2022 Apr 05 ]

Windows Server 2019 Standart Version 10.0.17763.2366

Zabbix Server 6.0.2

Zabbix Agent2 6.0.1 (With SSL)

Comment by Jeroen [ 2022 Apr 05 ]

I can confirm the exact same problem happening  with the ' Windows by Zabbix agent Active' template with Zabbix agent2 installed. I have the same happening on all of our Windows host (200+). Everything works fine when using the normal Zabbix Agent. Is there some workarround or temporary fix known to someone?

 

This happens on

Server: Windows server 2012, 2016 & 2019
Agent: Zabbix Agent 2 version 6.01 Revision a80cb13868
Zabbix Server: 6.0.2 Revision d726a4d916
Zabbix Proxy: 6.02 Revision d726a4d916

Comment by Ivan Zubov [ 2022 Apr 06 ]

Jeroen, temporary fix is restart zabbix-agent2 service.

Comment by Jmaiv [ 2022 May 04 ]

Hi Guys,

 

I believe I've been struggling with the same bug ever sinds we upgraded from Zabbix 4 LTS to 5 LTS and also made the jump to zabbix_agent2.

We have several hosts that 'out of nothing' seem to go in to the error state and stop (partially/fully) sending data. 
We first noticed the behavior on agent 5.0.14, then upgrade to 5.0.22 but this did not resolve the problem.

 

Here is an example:
Missing CPU stats, until I restarted the Zabbix_Agent2 service on the Windows Server.

Log files show loads of entry's with this;

2022/05/04 08:02:56.007797 plugin 'WindowsPerfMon' collector failed: No data to return.
2022/05/04 08:02:57.997551 plugin 'WindowsPerfMon' collector failed: No data to return.

 

 

On another server we have the same problem but with a additional twist to it.
CPU usage is not being reported at all for the last 2+ hours.

where at the same time, memory utilization is reported (except for the read marked area) 

In addition, network traffic is 'blank' until 7:45 (so failing to monitor?) but after 7:45 network traffic is logged.

Also the logs show the same entry's:

2022/05/04 07:32:23.054959 plugin 'WindowsPerfMon' collector failed: No data to return.
2022/05/04 07:32:24.049690 plugin 'WindowsPerfMon' collector failed: No data to return.

 

 

I hope this information helps.
Would it be useful if I would 'downgrade' from the Zabbix_agent2.exe to Zabbix_agent.exe in order to verify that this is not present in the 'old' agent?

further, can I be of any assist in the matter? 

 

 

Sorry guys, I think I missed the headlines of this report;
It's fixed in 5.0.24rc1 and I'm running 5.0.22 right now.. so I guess I need to wait for the release of 5.0.24 for now 

Comment by Vladislavs Sokurenko [ 2022 May 05 ]

Fixed in pull request feature/ZBX-20356-5.0

Comment by Aigars Kadikis [ 2022 May 06 ]

New 64-bit agent2 6.0.4rc1 for testing: zabbix_agent2-6.0.4rc1-windows-amd64-openssl-static.zip
This one comes with clean code.

Comment by Jmaiv [ 2022 May 09 ]

Any chance I can download the 5.0.24rc1 to test this fix?

I'm kind of desperate to get this going

 

Comment by Vladislavs Sokurenko [ 2022 May 10 ]

Fixed in:

Comment by Jeffrey Descan [ 2022 Nov 18 ]

As this is re-opened. Please take https://support.zabbix.com/browse/ZBX-21703 into consideration, as we're seeing this as well. We've provided some more debug information as requested.

Comment by Vladislavs Sokurenko [ 2022 Dec 07 ]

Fixed in:

Generated at Tue Apr 01 08:34:03 EEST 2025 using Jira 9.12.4#9120004-sha1:625303b708afdb767e17cb2838290c41888e9ff0.