[ZBX-18800] Zabbix agent WMI returns incorrect values Created: 2020 Dec 22 Updated: 2021 Jun 28 |
|
Status: | Confirmed |
Project: | ZABBIX BUGS AND ISSUES |
Component/s: | Agent (G), Agent2 plugin (G) |
Affects Version/s: | 5.0.7, 5.2.2 |
Fix Version/s: | None |
Type: | Incident report | Priority: | Minor |
Reporter: | Daniel | Assignee: | Zabbix Development Team |
Resolution: | Unresolved | Votes: | 0 |
Labels: | None | ||
Remaining Estimate: | Not Specified | ||
Time Spent: | Not Specified | ||
Original Estimate: | Not Specified | ||
Environment: |
Windows Server 2016 |
Description |
Steps to reproduce:
wmi.get[root\cimv2,select AvgDisksecPerRead from Win32_PerfRawData_PerfDisk_Logicaldisk WHERE Name='_Total']
Result: Values are unreliable:
...
Error state can be reproduced on a system: .\zabbix_agentd.exe -t "wmi.get[root\cimv2,select AvgDisksecPerRead from Win32_PerfRawData_PerfDisk_Logicaldisk WHERE Name LIKE '%VD%']" wmi.get[root\cimv2,select AvgDisksecPerRead from Win32_PerfRawData_PerfDisk_Logicaldisk WHERE Name='C:\\mount\\Tank-VD-0'] [u|18446744073578909317] Running wmi query directly (shortly after):
gwmi -Query "select AvgDisksecPerRead from Win32_PerfRawData_PerfDisk_Logicaldisk WHERE Name LIKE '%VD%'"
AvgDisksecPerRead : 4001443676
Same system - 30min later: .\zabbix_agentd.exe -t "wmi.get[root\cimv2,select AvgDisksecPerRead from Win32_PerfRawData_PerfDisk_Logicaldisk WHERE Name LIKE '%VD%']" wmi.get[root\cimv2,select AvgDisksecPerRead from Win32_PerfRawData_PerfDisk_Logicaldisk WHERE Name='C:\\mount\\Tank-VD-0'] [u|82629443]
Running wmi query directly (shortly after):
gwmi -Query "select AvgDisksecPerRead from Win32_PerfRawData_PerfDisk_Logicaldisk WHERE Name LIKE '%VD%'"
AvgDisksecPerRead : 82668265
There's no real starting point here - it just occurs "sometimes". Expectation would be that the values are returned correctly (as the native WMI query does). |
Comments |
Comment by Edgar Akhmetshin [ 2020 Dec 24 ] |
Hello Daniel, Could you please clarify - which exact patch level of the 2016/2019 used, so we can reproduce such behaviour. On Windows 10 Pro 20H2 i'm not able to reproduce with 5.2.3. Also this metrics are from storage pools/spaces or from direct attached storage? Regards, |
Comment by Daniel [ 2020 Dec 24 ] |
Hello Edgar, we're using both with latest patches installed. This issue occurs on many systems, hardware (with storage spaces) and virtual machines. It's hard to find a common component of all systems. It doesn't matter if we're using wmi.get or wmi.getall. The only interesting thing is that when values are approaching int32 signed max (2147483647), the system produces incorrect values (always 20 digits) until some point when it starts over with small values and it repeats. Seems just some sort of overflow. I checked the source code (agent, wmi). May there be an issue with detection of vector type in zabbix agent that results in an incorrect value conversion? Regards, Daniel |
Comment by Aigars Kadikis [ 2020 Dec 28 ] |
While agent reports 20 digit number, try listing same counters through Windows PowerShell:
Get-WmiObject -Query "select AvgDisksecPerRead from Win32_PerfRawData_PerfDisk_Logicaldisk"
Please make sure you are using a 64-bit agent on the 64-bit windows system. Show the output: zabbix_agentd -V |
Comment by Daniel [ 2020 Dec 28 ] |
I've already noted the result of WMI query in the issue itself. There's an example shown when it's correct and incorrect (agent & wmi result). We're already using x64 Version of the agent.
zabbix_agentd Win64 (service) (Zabbix) 5.0.2
Revision 352ca05870 13 July 2020, compilation time: Jul 13 2020 13:49:01
Copyright (C) 2020 Zabbix SIA
License GPLv2+: GNU GPL version 2 or later <http://gnu.org/licenses/gpl.html>.
This is free software: you are free to change and redistribute it according to
the license. There is NO WARRANTY, to the extent permitted by law.
Issue exists with any agent (testet vom 4.4 to latest build). When it occurs, all versions on the same system show the same behavior at the same time - so the issue does not occur related to agent runtime or anything else, it's propably reproducible depending on processed values. |
Comment by Aigars Kadikis [ 2020 Dec 29 ] |
Can reproduce the issue. Windows version: C:\>ver Microsoft Windows [Version 10.0.18363.1256] Powershell output from a command prompt: powershell -nologo -command "Get-WmiObject -Query \"select AvgDisksecPerRead from Win32_PerfRawData_PerfDisk_Logicaldisk\" | foreach { $_.AvgDisksecPerRead }" 2596463301 8261 2596471562 Agent1 version: c:\zabbix>zabbix_agentd.exe -V zabbix_agentd Win64 (service) (Zabbix) 5.0.7 Revision 1b3db7f980 21 December 2020, compilation time: Dec 21 2020 16:00:09 Copyright (C) 2020 Zabbix SIA License GPLv2+: GNU GPL version 2 or later <http://gnu.org/licenses/gpl.html>. This is free software: you are free to change and redistribute it according to the license. There is NO WARRANTY, to the extent permitted by law. This product includes software developed by the OpenSSL Project for use in the OpenSSL Toolkit (http://www.openssl.org/). Compiled with OpenSSL 1.1.1g 21 Apr 2020 Running with OpenSSL 1.1.1g 21 Apr 2020 Output: c:\zabbix>zabbix_get -s 127.0.0.1 -p 10050 -k "wmi.getall[root\cimv2,select AvgDisksecPerRead from Win32_PerfRawData_PerfDisk_Logicaldisk]" [{"AvgDisksecPerRead":18446744072012421710},{"AvgDisksecPerRead":8261},{"AvgDisksecPerRead":18446744072012429971}] Agent2 version: c:\zabbix>"c:\Program Files\Zabbix Agent 2\zabbix_agent2.exe" -V zabbix_agent2 Win64 (Zabbix) 5.0.7 Revision 1b3db7f980 21 December 2020, compilation time: 2020-12-21 16:04:36 Copyright (C) 2020 Zabbix SIA License GPLv2+: GNU GPL version 2 or later <http://gnu.org/licenses/gpl.html>. This is free software: you are free to change and redistribute it according to the license. There is NO WARRANTY, to the extent permitted by law. This product includes software developed by the OpenSSL Project for use in the OpenSSL Toolkit (http://www.openssl.org/). Compiled with OpenSSL 1.1.1d 10 Sep 2019 Running with OpenSSL 1.1.1d 10 Sep 2019 Has negative values in the output: [{"AvgDisksecPerRead":-1697079031,"Name":"C:"},{"AvgDisksecPerRead":8261,"Name":"HarddiskVolume2"},{"AvgDisksecPerRead":-1697070770,"Name":"_Total"}] |
Comment by Daniel [ 2021 Jun 28 ] |
We're still hanging on this issue because building metrics on these values is not possible. On investiation we noticed that we're indeed talking about an overflow - these errors occur seemingly always when the value increases above int32 max value (2147483647). The formula to "fix" these values is: uint32Max - (int64Max - value) or in numbers: 4294967295-(18446744073709500000-value). This works at least for values in range int32max - uint32max but has to be implemented using a calculation on item processing (like javascript) which is a lot of effort. Maybe this helps to fix this issue. |