-
Incident report
-
Resolution: Unresolved
-
Trivial
-
None
-
7.0.3
-
None
-
Windows 2022
A newly built windows 7.0.3 agent (built on WIndows 2022) crashes and no data is sent to proxy. This looks like something to do with the WinAPI calls to get perfmon counters. Upping the debug value the logfile contains:-
22048:20240904:144040.208 Starting Zabbix Agent [<redacted>]. Zabbix 7.0.3 (revision d93ce022627). *22048:20240904:144040.210 **** Enabled features ***** 22048:20240904:144040.211 IPv6 support: YES 22048:20240904:144040.212 TLS support: YES *22048:20240904:144040.213 *************************** 22048:20240904:144040.214 using configuration file: c:\Program Files\Zabbix Agent\zabbix_agentd.conf 22048:20240904:144040.219 In zbx_init_collector_data() 22048:20240904:144040.220 logical CPU count 4 22048:20240904:144040.221 End of zbx_init_collector_data() 22048:20240904:144040.222 In zbx_init_perf_collector() 22048:20240904:144040.224 In zbx_init_builtin_counter_indexes() 22048:20240904:144040.225 In zbx_get_all_counter_names() 22048:20240904:144040.415 End of zbx_get_all_counter_names() 22048:20240904:144040.744 End of zbx_init_builtin_counter_indexes():SUCCEED 22048:20240904:144040.745 In set_object_names() 22048:20240904:144041.046 VEH Trap detected exception 6d9 at 0x00007FFB6B0FF14C. Exception information: 22048:20240904:144041.049 ====== Fatal information: ====== 22048:20240904:144041.052 Program counter: 0x6b0ff14c 22048:20240904:144041.053 === Registers: === 22048:20240904:144041.055 r8 = 2e42 = 11842 = 11842 22048:20240904:144041.058 r9 = ac08b612 = 2886252050 = 2886252050 22048:20240904:144041.059 r10 = 1001d0000 = 4296867840 = 4296867840 22048:20240904:144041.061 r11 = 0 = 0 = 0 22048:20240904:144041.062 r12 = 6d9 = 1753 = 1753 22048:20240904:144041.064 r13 = 0 = 0 = 0 22048:20240904:144041.065 r14 = 10ecaf0 = 17746672 = 17746672 22048:20240904:144041.067 r15 = f8d5f0 = 16307696 = 16307696 22048:20240904:144041.068 rdi = 7ffb6cceb4c8 = 140717839004872 = 140717839004872 22048:20240904:144041.070 rsi = 0 = 0 = 0 22048:20240904:144041.071 rbp = 0 = 0 = 0 22048:20240904:144041.073 rbx = 6d9 = 1753 = 1753 22048:20240904:144041.074 rdx = 2dc0 = 11712 = 11712 22048:20240904:144041.075 rax = 400 = 1024 = 1024 22048:20240904:144041.077 rcx = 3ff = 1023 = 1023 22048:20240904:144041.078 rsp = f8d370 = 16307056 = 16307056 22048:20240904:144041.080 efl = 202 = 514 = 514 22048:20240904:144041.081 csgsfs = 332b5300 = 858477312 = 858477312 22048:20240904:144041.083 === Backtrace: === 22048:20240904:144041.154 24: C:\Windows\System32\KERNELBASE.dll(RaiseException+0x6c) [0x6b0ff14c] 22048:20240904:144041.157 23: C:\Windows\System32\RPCRT4.dll(RpcRaiseException+0x50) [0x6d947930] 22048:20240904:144041.160 22: C:\Windows\System32\RPCRT4.dll(RpcRaiseException+0x14) [0x6d9478f4] 22048:20240904:144041.163 21: C:\Windows\System32\RPCRT4.dll(NdrClientCall3+0x2282) [0x6da19e02] 22048:20240904:144041.165 20: C:\Windows\System32\RPCRT4.dll(NdrClientCall3+0xf0) [0x6da17c70] 22048:20240904:144041.167 19: C:\Windows\System32\ADVAPI32.dll(PerfRegSetValue+0x41f4) [0x6ccd9ae4] 22048:20240904:144041.171 18: C:\Windows\System32\ADVAPI32.dll(ElfOldestRecord+0xb101) [0x6ccaba61] 22048:20240904:144041.172 17: C:\Windows\System32\ADVAPI32.dll(EnumServicesStatusExA+0x14f9) [0x6cc97f79] 22048:20240904:144041.175 16: C:\Windows\System32\ADVAPI32.dll(EnumServicesStatusExA+0xc93) [0x6cc97713] 22048:20240904:144041.176 15: C:\Windows\System32\ADVAPI32.dll(EnableTrace+0x27c6) [0x6cc91856] 22048:20240904:144041.177 14: C:\Windows\System32\ADVAPI32.dll(PerfRegQueryValue+0x3b) [0x6cc9202b] 22048:20240904:144041.179 13: C:\Windows\System32\KERNELBASE.dll(GetCurrentThread+0xbd0) [0x6b17ed40] 22048:20240904:144041.181 12: C:\Windows\System32\KERNELBASE.dll(RegQueryValueExW+0x10b) [0x6b107f5b] 22048:20240904:144041.183 11: C:\Windows\SYSTEM32\pdh.dll(PdhGetFormattedCounterArrayW+0x287d) [0x5e9f979d] 22048:20240904:144041.185 10: C:\Windows\SYSTEM32\pdh.dll(PdhSetCounterScaleFactor+0x154a) [0x5ea0d0da] 22048:20240904:144041.186 9: C:\Windows\SYSTEM32\pdh.dll(PdhTranslate009CounterW+0x143f) [0x5ea0793f] 22048:20240904:144041.190 8: C:\Windows\SYSTEM32\pdh.dll(PdhWriteRelogSample+0xc01) [0x5ea2fad1] 22048:20240904:144041.194 7: C:\Windows\SYSTEM32\pdh.dll(PdhEnumObjectsHW+0xa2) [0x5ea31ce2] 22048:20240904:144041.198 6: C:\Windows\SYSTEM32\pdh.dll(PdhEnumObjectsW+0xeb) [0x5ea31e1b] 22048:20240904:144041.201 5: C:\Program Files\Zabbix Agent\zabbix_agentd.exe() [0x40149a5f] 22048:20240904:144041.203 4: C:\Program Files\Zabbix Agent\zabbix_agentd.exe() [0x4014a191] 22048:20240904:144041.207 3: C:\Program Files\Zabbix Agent\zabbix_agentd.exe() [0x40169671] 22048:20240904:144041.209 2: C:\Windows\System32\sechost.dll(BuildSecurityDescriptorForSharingAccessEx+0xb32) [0x6c5e8e32] 22048:20240904:144041.211 1: C:\Windows\System32\KERNEL32.DLL(BaseThreadInitThunk+0x10) [0x6ba04cb0] 22048:20240904:144041.212 0: C:\Windows\SYSTEM32\ntdll.dll(RtlUserThreadStart+0x2b) [0x6db3eceb] 22048:20240904:144041.216 ================================ 22048:20240904:144041.354 VEH Trap detected exception 6ba at 0x00007FFB6B0FF14C. Exception information: 22048:20240904:144041.358 ====== Fatal information: ====== 22048:20240904:144041.360 Program counter: 0x6b0ff14c 22048:20240904:144041.363 === Registers: === 22048:20240904:144041.365 r8 = 0 = 0 = 0 22048:20240904:144041.367 r9 = 0 = 0 = 0 22048:20240904:144041.370 r10 = f8cb30 = 16304944 = 16304944 22048:20240904:144041.372 r11 = f8c9d0 = 16304592 = 16304592 22048:20240904:144041.374 r12 = 6ba = 1722 = 1722 22048:20240904:144041.375 r13 = 0 = 0 = 0 22048:20240904:144041.377 r14 = 1085330 = 17322800 = 17322800 22048:20240904:144041.379 r15 = f8d850 = 16308304 = 16308304 22048:20240904:144041.380 rdi = 7ffb5b6236c0 = 140717546682048 = 140717546682048 22048:20240904:144041.383 rsi = 0 = 0 = 0 22048:20240904:144041.385 rbp = 6 = 6 = 6 22048:20240904:144041.386 rbx = 6ba = 1722 = 1722 22048:20240904:144041.388 rdx = 232000 = 2301952 = 2301952 22048:20240904:144041.389 rax = 1f46857d4 = 8395446228 = 8395446228 22048:20240904:144041.391 rcx = 15177c = 1382268 = 1382268 22048:20240904:144041.392 rsp = f8d5d0 = 16307664 = 16307664 22048:20240904:144041.393 efl = 202 = 514 = 514 22048:20240904:144041.395 csgsfs = 332b5300 = 858477312 = 858477312 22048:20240904:144041.396 === Backtrace: === 22048:20240904:144041.471 25: C:\Windows\System32\KERNELBASE.dll(RaiseException+0x6c) [0x6b0ff14c] 22048:20240904:144041.475 24: C:\Windows\System32\RPCRT4.dll(RpcRaiseException+0x50) [0x6d947930] 22048:20240904:144041.479 23: C:\Windows\System32\RPCRT4.dll(RpcRaiseException+0x14) [0x6d9478f4] 22048:20240904:144041.483 22: C:\Windows\System32\RPCRT4.dll(NdrClientCall3+0x2282) [0x6da19e02] 22048:20240904:144041.486 21: C:\Windows\System32\RPCRT4.dll(NdrClientCall3+0xf0) [0x6da17c70] 22048:20240904:144041.489 20: C:\Windows\System32\INFOADMN.dll(InetInfoQueryStatistics+0x2e) [0x5b62114e] 22048:20240904:144041.491 19: C:\Windows\System32\infoctrs.dll(CollectINFOPerformanceData+0x87) [0x68aa12c7] 22048:20240904:144041.493 18: C:\Windows\System32\ADVAPI32.dll(EnumServicesStatusExA+0x16af) [0x6cc9812f] 22048:20240904:144041.496 17: C:\Windows\System32\ADVAPI32.dll(EnumServicesStatusExA+0xed2) [0x6cc97952] 22048:20240904:144041.498 16: C:\Windows\System32\ADVAPI32.dll(EnumServicesStatusExA+0xc93) [0x6cc97713] 22048:20240904:144041.499 15: C:\Windows\System32\ADVAPI32.dll(EnableTrace+0x27c6) [0x6cc91856] 22048:20240904:144041.501 14: C:\Windows\System32\ADVAPI32.dll(PerfRegQueryValue+0x3b) [0x6cc9202b] 22048:20240904:144041.503 13: C:\Windows\System32\KERNELBASE.dll(GetCurrentThread+0xbd0) [0x6b17ed40] 22048:20240904:144041.505 12: C:\Windows\System32\KERNELBASE.dll(RegQueryValueExW+0x10b) [0x6b107f5b] 22048:20240904:144041.507 11: C:\Windows\SYSTEM32\pdh.dll(PdhGetFormattedCounterArrayW+0x287d) [0x5e9f979d] 22048:20240904:144041.509 10: C:\Windows\SYSTEM32\pdh.dll(PdhSetCounterScaleFactor+0x154a) [0x5ea0d0da] 22048:20240904:144041.511 9: C:\Windows\SYSTEM32\pdh.dll(PdhTranslate009CounterW+0x143f) [0x5ea0793f] 22048:20240904:144041.514 8: C:\Windows\SYSTEM32\pdh.dll(PdhWriteRelogSample+0xc01) [0x5ea2fad1] 22048:20240904:144041.517 7: C:\Windows\SYSTEM32\pdh.dll(PdhEnumObjectsHW+0xa2) [0x5ea31ce2] 22048:20240904:144041.520 6: C:\Windows\SYSTEM32\pdh.dll(PdhEnumObjectsW+0xeb) [0x5ea31e1b] 22048:20240904:144041.523 5: C:\Program Files\Zabbix Agent\zabbix_agentd.exe() [0x40149a5f] 22048:20240904:144041.525 4: C:\Program Files\Zabbix Agent\zabbix_agentd.exe() [0x4014a191] 22048:20240904:144041.527 3: C:\Program Files\Zabbix Agent\zabbix_agentd.exe() [0x40169671] 22048:20240904:144041.529 2: C:\Windows\System32\sechost.dll(BuildSecurityDescriptorForSharingAccessEx+0xb32) [0x6c5e8e32] 22048:20240904:144041.530 1: C:\Windows\System32\KERNEL32.DLL(BaseThreadInitThunk+0x10) [0x6ba04cb0] 22048:20240904:144041.532 0: C:\Windows\SYSTEM32\ntdll.dll(RtlUserThreadStart+0x2b) [0x6db3eceb] 22048:20240904:144041.536 ================================ 22048:20240904:144042.208 In zbx_get_all_counter_names() 22048:20240904:144042.217 End of zbx_get_all_counter_names() 22048:20240904:144042.218 In zbx_get_all_counter_names() 22048:20240904:144042.224 End of zbx_get_all_counter_names() 22048:20240904:144042.482 End of set_object_names():SUCCEED 22048:20240904:144042.486 End of zbx_init_perf_collector():SUCCEED 22048:20240904:144042.492 OpenSSL library (version OpenSSL 3.3.1 4 Jun 2024) initialized 22048:20240904:144042.496 agent #0 started [main process] 10812:20240904:144042.500 agent #1 started [collector] 18748:20240904:144042.502 agent #2 started active checks #1
then further down I get loads of:
18748:20240904:144042.670 In parse_list_of_checks() 18748:20240904:144042.672 cannot retrieve value of tag "itemid" 18748:20240904:144042.672 cannot retrieve value of tag "itemid" 18748:20240904:144042.673 cannot retrieve value of tag "itemid" 18748:20240904:144042.674 cannot retrieve value of tag "itemid"
Searching I can see crashes of similar natures reported. I'm guessing that VEH is just the debugging part of it. I did see a github "fix" about changing the multiplexed WinAPI calls to perfmon to a single stream and wondered if this the problem that I have here.
The agent was built on a Windows 2022 machine exactly as specified in the guide on the zabbix website for compiling the windows agent, though I did have to add crypt32.lib in the make files to statically link openssl.
The previous version of the agent on this machine (6.0.0alpha1) works without issue.
I'll see if I can find the "fix" again and see if I alter the source code as described in the report whether it fixes the issue. I'll also see if previous releases also have the same issue. I do wonder whether the fix was not put into the main branch so has been wiped out.