[ZBX-15162] GPU perf_counter returns 0.000000 Created: 2018 Nov 13  Updated: 2019 Jan 15  Resolved: 2019 Jan 09

Status: Closed
Project: ZABBIX BUGS AND ISSUES
Component/s: Agent (G)
Affects Version/s: None
Fix Version/s: None

Type: Problem report Priority: Major
Reporter: Maksims Edelmans Assignee: Unassigned
Resolution: Incomplete Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Description   

I need to monitor GPU statistics on a Windows machine.

I have found out that there are performance counters that can do it.

So I ran this:

C:\Users\admin>typeperf -q | find "GPU"
\GPU Local Adapter Memory(*)\Local Usage
\GPU Engine(*)\Utilization Percentage
\GPU Engine(*)\Running Time
\GPU Adapter Memory(*)\Shared Usage
\GPU Adapter Memory(*)\Dedicated Usage
\GPU Adapter Memory(*)\Total Committed
\GPU Process Memory(*)\Shared Usage
\GPU Process Memory(*)\Dedicated Usage
\GPU Process Memory(*)\Non Local Usage
\GPU Process Memory(*)\Local Usage
\GPU Process Memory(*)\Total Committed
\GPU Non Local Adapter Memory(*)\Non Local Usage

Now I query "\GPU Engine\Utilization Percentage" from CLI:

 

PS C:\Users\admin> Get-Counter -Counter "\GPU Engine(*)\Utilization Percentage"
Timestamp CounterSamples
--------- --------------
13/11/2018 17:58:38 \\tc-elecard-08\gpu
 engine(pid_16188_luid_0x00000000_0x00006013_phys_0_eng_8_engtype_3d)\utilization percentage :
 0
\\tc-elecard-08\gpu
 engine(pid_16188_luid_0x00000000_0x00006013_phys_0_eng_7_engtype_3d)\utilization percentage :
 0
\\tc-elecard-08\gpu
 engine(pid_16188_luid_0x00000000_0x00006013_phys_0_eng_6_engtype_3d)\utilization percentage :
 0
\\tc-elecard-08\gpu
 engine(pid_16188_luid_0x00000000_0x00006013_phys_0_eng_5_engtype_3d)\utilization percentage :
 0
\\tc-elecard-08\gpu
 engine(pid_16188_luid_0x00000000_0x00006013_phys_0_eng_4_engtype_3d)\utilization percentage :
 0
\\tc-elecard-08\gpu
 engine(pid_16188_luid_0x00000000_0x00006013_phys_0_eng_3_engtype_3d)\utilization percentage :
 0
\\tc-elecard-08\gpu
 engine(pid_16188_luid_0x00000000_0x00006013_phys_0_eng_2_engtype_3d)\utilization percentage :
 0
\\tc-elecard-08\gpu
 engine(pid_16188_luid_0x00000000_0x00006013_phys_0_eng_1_engtype_3d)\utilization percentage :
 0.111372031119455
\\tc-elecard-08\gpu
 engine(pid_16188_luid_0x00000000_0x00006013_phys_0_eng_0_engtype_3d)\utilization percentage :
 0
\\tc-elecard-08\gpu
 engine(pid_1028_luid_0x00000000_0x000060cd_phys_0_eng_9_engtype_videoprocessing)\utilization
 percentage :
 0
\\tc-elecard-08\gpu engine(pid_1028_luid_0x00000000_0x000060cd_phys_0_eng_8_engtype_gdi
 render)\utilization percentage :
 0
\\tc-elecard-08\gpu
 engine(pid_1028_luid_0x00000000_0x000060cd_phys_0_eng_7_engtype_legacyoverlay)\utilization
 percentage :
 0
\\tc-elecard-08\gpu
 engine(pid_1028_luid_0x00000000_0x000060cd_phys_0_eng_6_engtype_)\utilization percentage :
 0
\\tc-elecard-08\gpu
 engine(pid_1028_luid_0x00000000_0x000060cd_phys_0_eng_5_engtype_)\utilization percentage :
 0
\\tc-elecard-08\gpu
 engine(pid_1028_luid_0x00000000_0x000060cd_phys_0_eng_4_engtype_videodecode)\utilization
 percentage :
 0
\\tc-elecard-08\gpu
 engine(pid_1028_luid_0x00000000_0x000060cd_phys_0_eng_3_engtype_videoprocessing)\utilization
 percentage :
 0
\\tc-elecard-08\gpu
 engine(pid_1028_luid_0x00000000_0x000060cd_phys_0_eng_2_engtype_copy)\utilization percentage
 :
 0
\\tc-elecard-08\gpu
 engine(pid_1028_luid_0x00000000_0x000060cd_phys_0_eng_1_engtype_videodecode)\utilization
 percentage :
 0
\\tc-elecard-08\gpu
 engine(pid_1028_luid_0x00000000_0x000060cd_phys_0_eng_0_engtype_3d)\utilization percentage :
 0
\\tc-elecard-08\gpu
 engine(pid_3916_luid_0x00000000_0x00006013_phys_0_eng_8_engtype_3d)\utilization percentage :
 0
\\tc-elecard-08\gpu
 engine(pid_3916_luid_0x00000000_0x00006013_phys_0_eng_7_engtype_3d)\utilization percentage :
 0
\\tc-elecard-08\gpu
 engine(pid_3916_luid_0x00000000_0x00006013_phys_0_eng_6_engtype_3d)\utilization percentage :
 0
\\tc-elecard-08\gpu
 engine(pid_3916_luid_0x00000000_0x00006013_phys_0_eng_5_engtype_3d)\utilization percentage :
 0
\\tc-elecard-08\gpu
 engine(pid_3916_luid_0x00000000_0x00006013_phys_0_eng_4_engtype_3d)\utilization percentage :
 0
\\tc-elecard-08\gpu
 engine(pid_3916_luid_0x00000000_0x00006013_phys_0_eng_3_engtype_3d)\utilization percentage :
 0
\\tc-elecard-08\gpu
 engine(pid_3916_luid_0x00000000_0x00006013_phys_0_eng_2_engtype_3d)\utilization percentage :
 0
\\tc-elecard-08\gpu
 engine(pid_3916_luid_0x00000000_0x00006013_phys_0_eng_1_engtype_3d)\utilization percentage :
 0
\\tc-elecard-08\gpu
 engine(pid_3916_luid_0x00000000_0x00006013_phys_0_eng_0_engtype_3d)\utilization percentage :
 0
\\tc-elecard-08\gpu engine(pid_14632_luid_0x00000000_0x000060cd_phys_0_eng_9_engtype_videopro
 cessing)\utilization percentage :
 0
\\tc-elecard-08\gpu engine(pid_14632_luid_0x00000000_0x000060cd_phys_0_eng_8_engtype_gdi
 render)\utilization percentage :
 0
\\tc-elecard-08\gpu
 engine(pid_14632_luid_0x00000000_0x000060cd_phys_0_eng_7_engtype_legacyoverlay)\utilization
 percentage :
 0
\\tc-elecard-08\gpu
 engine(pid_14632_luid_0x00000000_0x000060cd_phys_0_eng_6_engtype_)\utilization percentage :
 0
\\tc-elecard-08\gpu
 engine(pid_14632_luid_0x00000000_0x000060cd_phys_0_eng_5_engtype_)\utilization percentage :
 0
\\tc-elecard-08\gpu
 engine(pid_14632_luid_0x00000000_0x000060cd_phys_0_eng_4_engtype_videodecode)\utilization
 percentage :
 0
\\tc-elecard-08\gpu engine(pid_14632_luid_0x00000000_0x000060cd_phys_0_eng_3_engtype_videopro
 cessing)\utilization percentage :
 0
\\tc-elecard-08\gpu
 engine(pid_14632_luid_0x00000000_0x000060cd_phys_0_eng_2_engtype_copy)\utilization
 percentage :
 0
\\tc-elecard-08\gpu
 engine(pid_14632_luid_0x00000000_0x000060cd_phys_0_eng_1_engtype_videodecode)\utilization
 percentage :
 0
\\tc-elecard-08\gpu
 engine(pid_14632_luid_0x00000000_0x000060cd_phys_0_eng_0_engtype_3d)\utilization percentage :
 0
\\tc-elecard-08\gpu engine(pid_10556_luid_0x00000000_0x000060cd_phys_0_eng_9_engtype_videopro
 cessing)\utilization percentage :
 0
\\tc-elecard-08\gpu engine(pid_10556_luid_0x00000000_0x000060cd_phys_0_eng_8_engtype_gdi
 render)\utilization percentage :
 0
\\tc-elecard-08\gpu
 engine(pid_10556_luid_0x00000000_0x000060cd_phys_0_eng_7_engtype_legacyoverlay)\utilization
 percentage :
 0
\\tc-elecard-08\gpu
 engine(pid_10556_luid_0x00000000_0x000060cd_phys_0_eng_6_engtype_)\utilization percentage :
 0
\\tc-elecard-08\gpu
 engine(pid_10556_luid_0x00000000_0x000060cd_phys_0_eng_5_engtype_)\utilization percentage :
 0
\\tc-elecard-08\gpu
 engine(pid_10556_luid_0x00000000_0x000060cd_phys_0_eng_4_engtype_videodecode)\utilization
 percentage :
 0
\\tc-elecard-08\gpu engine(pid_10556_luid_0x00000000_0x000060cd_phys_0_eng_3_engtype_videopro
 cessing)\utilization percentage :
 3.96203603891869
\\tc-elecard-08\gpu
 engine(pid_10556_luid_0x00000000_0x000060cd_phys_0_eng_2_engtype_copy)\utilization
 percentage :
 0
\\tc-elecard-08\gpu
 engine(pid_10556_luid_0x00000000_0x000060cd_phys_0_eng_1_engtype_videodecode)\utilization
 percentage :
 3.63973424907668
\\tc-elecard-08\gpu
 engine(pid_10556_luid_0x00000000_0x000060cd_phys_0_eng_0_engtype_3d)\utilization percentage :
 7.10984386144568
 
(output omitted)

As you can see, some processes are utilizing GPU which altogether gives me around 50%.

But if I query with zabbix_get I get this:

[maksimse@zabbix-server-node1 ~]$ zabbix_get -s <host> -k perf_counter["\GPU Engine(*)\Utilization Percentage"]
0.000000

Why? Is it a bug?

 



 Comments   
Comment by Edgars Melveris [ 2018 Dec 27 ]

Hello Maksims!
Can you try to get the value for this item using zabbix_agentd on the affected host itself?
Use this parameter:

-t, --test item-key
    Test single item and exit. See --print for output description. 
Comment by Maksims Edelmans [ 2018 Dec 27 ]

PS C:\zabbix> .\zabbix_agentd.exe -c .\zabbix_agentd.win.conf -t perf_counter["\GPU Engine(*)\Utilization Percentage"]
perf_counter[\GPU Engine(*)\Utilization Percentage] [d|0.000000]
PS C:\zabbix>

Comment by Glebs Ivanovskis [ 2018 Dec 27 ]

Dear maksims.edelmans, do you use a pre-compiled binary from Zabbi website? It is compiled on the old Windows and GPU performance counters seem to be a relatively new addition. Could it be that pdh.dll Zabbix agent is linked to does not support newest counters?

Comment by Arturs Lontons [ 2019 Jan 09 ]

Please provide the information that Glebs requested - it could very well be the cause for the described behavior.

The ticket will be closed for now due to lack of information.

Comment by Maksims Edelmans [ 2019 Jan 15 ]

Glebs,

Yes, I am using pre-compiled agent.

I tried to compile it but with no success ("unsupported architecture x86" error while compiling).

Anyway, I think the problem is that zabbix_get reads first line of the output, which in my case is 0.0000:

#PS C:\Users\admin> Get-Counter -Counter "\GPU Engine\Utilization Percentage"

Timestamp CounterSamples
--------- --------------
15/01/2019 14:43:18 \\tc-elecard-05\gpu
engine(pid_7600_luid_0x00000000_0x00006d06_phys_0_eng_9_engtype_videoprocessing)\utilization
percentage :
0

\\tc-elecard-05\gpu engine(pid_7600_luid_0x00000000_0x00006d06_phys_0_eng_8_engtype_gdi
render)\utilization percentage :
0

\\tc-elecard-05\gpu
engine(pid_7600_luid_0x00000000_0x00006d06_phys_0_eng_7_engtype_legacyoverlay)\utilization
percentage :
0

...

Workaround for me is reading GPU-Z log files.

The end.

 

 

Generated at Fri Mar 29 15:44:33 EET 2024 using Jira 9.12.4#9120004-sha1:625303b708afdb767e17cb2838290c41888e9ff0.