Uploaded image for project: 'ZABBIX BUGS AND ISSUES'
  1. ZABBIX BUGS AND ISSUES
  2. ZBX-25180

Zabbix 7.0.3 windows agent crashes on Windows Server 2022

XMLWordPrintable

    • Icon: Incident report Incident report
    • Resolution: Unresolved
    • Icon: Trivial Trivial
    • None
    • 7.0.3
    • Agent (G)
    • None
    • Windows 2022

      A newly built windows 7.0.3 agent (built on WIndows 2022) crashes and no data is sent to proxy. This looks like something to do with the WinAPI calls to get perfmon counters. Upping the debug value the logfile contains:-

       22048:20240904:144040.208 Starting Zabbix Agent [<redacted>]. Zabbix 7.0.3 (revision d93ce022627).
       *22048:20240904:144040.210 **** Enabled features *****
       22048:20240904:144040.211 IPv6 support:          YES
       22048:20240904:144040.212 TLS support:           YES
       *22048:20240904:144040.213 ***************************
       22048:20240904:144040.214 using configuration file: c:\Program Files\Zabbix Agent\zabbix_agentd.conf
       22048:20240904:144040.219 In zbx_init_collector_data()
       22048:20240904:144040.220 logical CPU count 4
       22048:20240904:144040.221 End of zbx_init_collector_data()
       22048:20240904:144040.222 In zbx_init_perf_collector()
       22048:20240904:144040.224 In zbx_init_builtin_counter_indexes()
       22048:20240904:144040.225 In zbx_get_all_counter_names()
       22048:20240904:144040.415 End of zbx_get_all_counter_names()
       22048:20240904:144040.744 End of zbx_init_builtin_counter_indexes():SUCCEED
       22048:20240904:144040.745 In set_object_names()
       22048:20240904:144041.046 VEH Trap detected exception 6d9 at 0x00007FFB6B0FF14C. Exception information:
       22048:20240904:144041.049 ====== Fatal information: ======
       22048:20240904:144041.052 Program counter: 0x6b0ff14c
       22048:20240904:144041.053 === Registers: ===
       22048:20240904:144041.055 r8      =             2e42 =                11842 =                11842
       22048:20240904:144041.058 r9      =         ac08b612 =           2886252050 =           2886252050
       22048:20240904:144041.059 r10     =        1001d0000 =           4296867840 =           4296867840
       22048:20240904:144041.061 r11     =                0 =                    0 =                    0
       22048:20240904:144041.062 r12     =              6d9 =                 1753 =                 1753
       22048:20240904:144041.064 r13     =                0 =                    0 =                    0
       22048:20240904:144041.065 r14     =          10ecaf0 =             17746672 =             17746672
       22048:20240904:144041.067 r15     =           f8d5f0 =             16307696 =             16307696
       22048:20240904:144041.068 rdi     =     7ffb6cceb4c8 =      140717839004872 =      140717839004872
       22048:20240904:144041.070 rsi     =                0 =                    0 =                    0
       22048:20240904:144041.071 rbp     =                0 =                    0 =                    0
       22048:20240904:144041.073 rbx     =              6d9 =                 1753 =                 1753
       22048:20240904:144041.074 rdx     =             2dc0 =                11712 =                11712
       22048:20240904:144041.075 rax     =              400 =                 1024 =                 1024
       22048:20240904:144041.077 rcx     =              3ff =                 1023 =                 1023
       22048:20240904:144041.078 rsp     =           f8d370 =             16307056 =             16307056
       22048:20240904:144041.080 efl     =              202 =                  514 =                  514
       22048:20240904:144041.081 csgsfs  =         332b5300 =            858477312 =            858477312
       22048:20240904:144041.083 === Backtrace: ===
       22048:20240904:144041.154 24: C:\Windows\System32\KERNELBASE.dll(RaiseException+0x6c) [0x6b0ff14c]
       22048:20240904:144041.157 23: C:\Windows\System32\RPCRT4.dll(RpcRaiseException+0x50) [0x6d947930]
       22048:20240904:144041.160 22: C:\Windows\System32\RPCRT4.dll(RpcRaiseException+0x14) [0x6d9478f4]
       22048:20240904:144041.163 21: C:\Windows\System32\RPCRT4.dll(NdrClientCall3+0x2282) [0x6da19e02]
       22048:20240904:144041.165 20: C:\Windows\System32\RPCRT4.dll(NdrClientCall3+0xf0) [0x6da17c70]
       22048:20240904:144041.167 19: C:\Windows\System32\ADVAPI32.dll(PerfRegSetValue+0x41f4) [0x6ccd9ae4]
       22048:20240904:144041.171 18: C:\Windows\System32\ADVAPI32.dll(ElfOldestRecord+0xb101) [0x6ccaba61]
       22048:20240904:144041.172 17: C:\Windows\System32\ADVAPI32.dll(EnumServicesStatusExA+0x14f9) [0x6cc97f79]
       22048:20240904:144041.175 16: C:\Windows\System32\ADVAPI32.dll(EnumServicesStatusExA+0xc93) [0x6cc97713]
       22048:20240904:144041.176 15: C:\Windows\System32\ADVAPI32.dll(EnableTrace+0x27c6) [0x6cc91856]
       22048:20240904:144041.177 14: C:\Windows\System32\ADVAPI32.dll(PerfRegQueryValue+0x3b) [0x6cc9202b]
       22048:20240904:144041.179 13: C:\Windows\System32\KERNELBASE.dll(GetCurrentThread+0xbd0) [0x6b17ed40]
       22048:20240904:144041.181 12: C:\Windows\System32\KERNELBASE.dll(RegQueryValueExW+0x10b) [0x6b107f5b]
       22048:20240904:144041.183 11: C:\Windows\SYSTEM32\pdh.dll(PdhGetFormattedCounterArrayW+0x287d) [0x5e9f979d]
       22048:20240904:144041.185 10: C:\Windows\SYSTEM32\pdh.dll(PdhSetCounterScaleFactor+0x154a) [0x5ea0d0da]
       22048:20240904:144041.186 9: C:\Windows\SYSTEM32\pdh.dll(PdhTranslate009CounterW+0x143f) [0x5ea0793f]
       22048:20240904:144041.190 8: C:\Windows\SYSTEM32\pdh.dll(PdhWriteRelogSample+0xc01) [0x5ea2fad1]
       22048:20240904:144041.194 7: C:\Windows\SYSTEM32\pdh.dll(PdhEnumObjectsHW+0xa2) [0x5ea31ce2]
       22048:20240904:144041.198 6: C:\Windows\SYSTEM32\pdh.dll(PdhEnumObjectsW+0xeb) [0x5ea31e1b]
       22048:20240904:144041.201 5: C:\Program Files\Zabbix Agent\zabbix_agentd.exe() [0x40149a5f]
       22048:20240904:144041.203 4: C:\Program Files\Zabbix Agent\zabbix_agentd.exe() [0x4014a191]
       22048:20240904:144041.207 3: C:\Program Files\Zabbix Agent\zabbix_agentd.exe() [0x40169671]
       22048:20240904:144041.209 2: C:\Windows\System32\sechost.dll(BuildSecurityDescriptorForSharingAccessEx+0xb32) [0x6c5e8e32]
       22048:20240904:144041.211 1: C:\Windows\System32\KERNEL32.DLL(BaseThreadInitThunk+0x10) [0x6ba04cb0]
       22048:20240904:144041.212 0: C:\Windows\SYSTEM32\ntdll.dll(RtlUserThreadStart+0x2b) [0x6db3eceb]
       22048:20240904:144041.216 ================================
       22048:20240904:144041.354 VEH Trap detected exception 6ba at 0x00007FFB6B0FF14C. Exception information:
       22048:20240904:144041.358 ====== Fatal information: ======
       22048:20240904:144041.360 Program counter: 0x6b0ff14c
       22048:20240904:144041.363 === Registers: ===
       22048:20240904:144041.365 r8      =                0 =                    0 =                    0
       22048:20240904:144041.367 r9      =                0 =                    0 =                    0
       22048:20240904:144041.370 r10     =           f8cb30 =             16304944 =             16304944
       22048:20240904:144041.372 r11     =           f8c9d0 =             16304592 =             16304592
       22048:20240904:144041.374 r12     =              6ba =                 1722 =                 1722
       22048:20240904:144041.375 r13     =                0 =                    0 =                    0
       22048:20240904:144041.377 r14     =          1085330 =             17322800 =             17322800
       22048:20240904:144041.379 r15     =           f8d850 =             16308304 =             16308304
       22048:20240904:144041.380 rdi     =     7ffb5b6236c0 =      140717546682048 =      140717546682048
       22048:20240904:144041.383 rsi     =                0 =                    0 =                    0
       22048:20240904:144041.385 rbp     =                6 =                    6 =                    6
       22048:20240904:144041.386 rbx     =              6ba =                 1722 =                 1722
       22048:20240904:144041.388 rdx     =           232000 =              2301952 =              2301952
       22048:20240904:144041.389 rax     =        1f46857d4 =           8395446228 =           8395446228
       22048:20240904:144041.391 rcx     =           15177c =              1382268 =              1382268
       22048:20240904:144041.392 rsp     =           f8d5d0 =             16307664 =             16307664
       22048:20240904:144041.393 efl     =              202 =                  514 =                  514
       22048:20240904:144041.395 csgsfs  =         332b5300 =            858477312 =            858477312
       22048:20240904:144041.396 === Backtrace: ===
       22048:20240904:144041.471 25: C:\Windows\System32\KERNELBASE.dll(RaiseException+0x6c) [0x6b0ff14c]
       22048:20240904:144041.475 24: C:\Windows\System32\RPCRT4.dll(RpcRaiseException+0x50) [0x6d947930]
       22048:20240904:144041.479 23: C:\Windows\System32\RPCRT4.dll(RpcRaiseException+0x14) [0x6d9478f4]
       22048:20240904:144041.483 22: C:\Windows\System32\RPCRT4.dll(NdrClientCall3+0x2282) [0x6da19e02]
       22048:20240904:144041.486 21: C:\Windows\System32\RPCRT4.dll(NdrClientCall3+0xf0) [0x6da17c70]
       22048:20240904:144041.489 20: C:\Windows\System32\INFOADMN.dll(InetInfoQueryStatistics+0x2e) [0x5b62114e]
       22048:20240904:144041.491 19: C:\Windows\System32\infoctrs.dll(CollectINFOPerformanceData+0x87) [0x68aa12c7]
       22048:20240904:144041.493 18: C:\Windows\System32\ADVAPI32.dll(EnumServicesStatusExA+0x16af) [0x6cc9812f]
       22048:20240904:144041.496 17: C:\Windows\System32\ADVAPI32.dll(EnumServicesStatusExA+0xed2) [0x6cc97952]
       22048:20240904:144041.498 16: C:\Windows\System32\ADVAPI32.dll(EnumServicesStatusExA+0xc93) [0x6cc97713]
       22048:20240904:144041.499 15: C:\Windows\System32\ADVAPI32.dll(EnableTrace+0x27c6) [0x6cc91856]
       22048:20240904:144041.501 14: C:\Windows\System32\ADVAPI32.dll(PerfRegQueryValue+0x3b) [0x6cc9202b]
       22048:20240904:144041.503 13: C:\Windows\System32\KERNELBASE.dll(GetCurrentThread+0xbd0) [0x6b17ed40]
       22048:20240904:144041.505 12: C:\Windows\System32\KERNELBASE.dll(RegQueryValueExW+0x10b) [0x6b107f5b]
       22048:20240904:144041.507 11: C:\Windows\SYSTEM32\pdh.dll(PdhGetFormattedCounterArrayW+0x287d) [0x5e9f979d]
       22048:20240904:144041.509 10: C:\Windows\SYSTEM32\pdh.dll(PdhSetCounterScaleFactor+0x154a) [0x5ea0d0da]
       22048:20240904:144041.511 9: C:\Windows\SYSTEM32\pdh.dll(PdhTranslate009CounterW+0x143f) [0x5ea0793f]
       22048:20240904:144041.514 8: C:\Windows\SYSTEM32\pdh.dll(PdhWriteRelogSample+0xc01) [0x5ea2fad1]
       22048:20240904:144041.517 7: C:\Windows\SYSTEM32\pdh.dll(PdhEnumObjectsHW+0xa2) [0x5ea31ce2]
       22048:20240904:144041.520 6: C:\Windows\SYSTEM32\pdh.dll(PdhEnumObjectsW+0xeb) [0x5ea31e1b]
       22048:20240904:144041.523 5: C:\Program Files\Zabbix Agent\zabbix_agentd.exe() [0x40149a5f]
       22048:20240904:144041.525 4: C:\Program Files\Zabbix Agent\zabbix_agentd.exe() [0x4014a191]
       22048:20240904:144041.527 3: C:\Program Files\Zabbix Agent\zabbix_agentd.exe() [0x40169671]
       22048:20240904:144041.529 2: C:\Windows\System32\sechost.dll(BuildSecurityDescriptorForSharingAccessEx+0xb32) [0x6c5e8e32]
       22048:20240904:144041.530 1: C:\Windows\System32\KERNEL32.DLL(BaseThreadInitThunk+0x10) [0x6ba04cb0]
       22048:20240904:144041.532 0: C:\Windows\SYSTEM32\ntdll.dll(RtlUserThreadStart+0x2b) [0x6db3eceb]
       22048:20240904:144041.536 ================================
       22048:20240904:144042.208 In zbx_get_all_counter_names()
       22048:20240904:144042.217 End of zbx_get_all_counter_names()
       22048:20240904:144042.218 In zbx_get_all_counter_names()
       22048:20240904:144042.224 End of zbx_get_all_counter_names()
       22048:20240904:144042.482 End of set_object_names():SUCCEED
       22048:20240904:144042.486 End of zbx_init_perf_collector():SUCCEED
       22048:20240904:144042.492 OpenSSL library (version OpenSSL 3.3.1 4 Jun 2024) initialized
       22048:20240904:144042.496 agent #0 started [main process]
       10812:20240904:144042.500 agent #1 started [collector]
       18748:20240904:144042.502 agent #2 started active checks #1

       then further down I get loads of:

      18748:20240904:144042.670 In parse_list_of_checks()
       18748:20240904:144042.672 cannot retrieve value of tag "itemid"
       18748:20240904:144042.672 cannot retrieve value of tag "itemid"
       18748:20240904:144042.673 cannot retrieve value of tag "itemid"
       18748:20240904:144042.674 cannot retrieve value of tag "itemid"

       

      Searching I can see crashes of similar natures reported. I'm guessing that VEH is just the debugging part of it. I did see a github "fix" about changing the multiplexed WinAPI calls to perfmon to a single stream and wondered if this the problem that I have here.

      The agent was built on a Windows 2022 machine exactly as specified in the guide on the zabbix website for compiling the windows agent, though I did have to add crypt32.lib in the make files to statically link openssl.

      The previous version of the agent on this machine (6.0.0alpha1) works without issue.

      I'll see if I can find the "fix" again and see if I alter the source code as described in the report whether it fixes the issue. I'll also see if previous releases also have the same issue. I do wonder whether the fix was not put into the main branch so has been wiped out.{}

       

            pwegrzyn Piotr Wegrzyn
            Paul_Morris Paul Morris
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

              Created:
              Updated: