-
Incident report
-
Resolution: Unresolved
-
Trivial
-
None
-
7.4.3
-
Operating System: Microsoft Windows Server 2022 Standard
OS Version: 10.0.20348 Build 20348
OS Build: 20348.1.amd64fre.fe_release.210507-1500
Architecture: x64-based PC (64-bit)
System Locale: en-AU (English - Australia)
Platform: Virtualized (QEMU/KVM with OVMF BIOS)
Performance Counter Environment:
- Performance Counter Sets: 170
- Total Performance Counters: 2732
Zabbix Agent Version: 7.4.3 (revision 9c308d6f49c)
Installation Path: C:\Program Files\Zabbix Agent\Operating System: Microsoft Windows Server 2022 Standard OS Version: 10.0.20348 Build 20348 OS Build: 20348.1.amd64fre.fe_release.210507-1500 Architecture: x64-based PC (64-bit) System Locale: en-AU (English - Australia) Platform: Virtualized (QEMU/KVM with OVMF BIOS) Performance Counter Environment: - Performance Counter Sets: 170 - Total Performance Counters: 2732 Zabbix Agent Version: 7.4.3 (revision 9c308d6f49c) Installation Path: C:\Program Files\Zabbix Agent\
Summary: Zabbix Agent 7.4.3 crashes immediately during startup with heap corruption (exception 0xc0000374 and 0xc0000005) when initializing the Windows Performance Data Helper (PDH) subsystem. The crash occurs in zbx_get_all_counter_names() before any monitoring checks execute, rendering the agent completely unusable.
Regression: This is a critical regression from version 7.4.2, which works perfectly on the same system with identical configuration.
Root Cause: The crash occurs during PDH initialization when the agent calls PdhEnumObjectsW to enumerate available performance counters. The backtrace indicates heap corruption during memory allocation for PDH counter name buffers, suggesting insufficient buffer size calculation or memory management error introduced in 7.4.3.
Impact:
- Zabbix Agent 7.4.3 (Agent 1) is completely broken on affected Windows Server 2022 systems
- Cannot be worked around via configuration changes (DenyKey, AllowKey ineffective)
- Crash happens before agent enters monitoring loop
- Affects production monitoring capability
Steps to Reproduce
- Install Zabbix Agent 7.4.3 on Windows Server 2022 Standard (Build 20348)
- Use default or any custom configuration
- Start the Zabbix Agent service
- Observe immediate crash during initialization
Configuration tested:
- Minimal configuration (Server, Hostname only)
- Configuration with DenyKey=perf_counter[*]
- Both produce identical crash
Expected Result
Agent initializes successfully, completes PDH setup, and begins monitoring operations.
Actual Result
Agent crashes with heap corruption during zbx_init_perf_collector():
1372:20251006:192318.637 Starting Zabbix Agent [W2T-FS01]. Zabbix 7.4.3 (revision 9c308d6f49c). 1372:20251006:192318.674 In zbx_init_perf_collector() 1372:20251006:192318.678 In zbx_init_builtin_counter_indexes() 1372:20251006:192318.682 In zbx_get_all_counter_names() 1372:20251006:192318.867 End of zbx_get_all_counter_names() 1372:20251006:192319.162 In set_object_names() 1372:20251006:192319.518 In zbx_get_all_counter_names() 1372:20251006:192319.526 End of zbx_get_all_counter_names() 1372:20251006:192319.530 In zbx_get_all_counter_names() 1372:20251006:192319.539 End of zbx_get_all_counter_names() 1372:20251006:192319.543 VEH Trap detected exception c0000005 at 0x00007FF8F82DF0AF
Exception codes observed:
- 0xc0000005 (ACCESS_VIOLATION)
- 0xc0000374 (HEAP_CORRUPTION)
Backtrace
Call stack showing crash origin:
Frame 41: ntdll.dll(RtlAllocateHeap+0x32ff) - Heap corruption detected Frame 40: ntdll.dll(RtlAllocateHeap+0x749) Frame 39: AcLayers.DLL(NotifyShims+0x39ac) Frame 38-30: zabbix_agentd.exe internal functions Frame 12: pdh.dll(PdhCloseLog+0xf8b) Frame 11: pdh.dll(PdhCloseLog+0x812) Frame 10: pdh.dll(PdhWriteRelogSample+0x1ccc) Frame 9: pdh.dll(PdhWriteRelogSample+0x1db3) Frame 8: pdh.dll(PdhWriteRelogSample+0xdd2) Frame 7: pdh.dll(PdhEnumObjectsHW+0xa2) Frame 6: pdh.dll(PdhEnumObjectsW+0xeb) ← PDH API call Frame 5: zabbix_agentd.exe [0x4012e6cc] ← Zabbix PDH wrapper Frame 4: zabbix_agentd.exe [0x4012ecf1] Frame 3: zabbix_agentd.exe [0x4014ebf9]
Key observation: The function zbx_get_all_counter_names() is called three times during initialization. The third call triggers the heap corruption, suggesting cumulative memory corruption or buffer size miscalculation.
Full Debug Output
<details> <summary>Complete crash log with DebugLevel=5</summary> ``` 1372:20251006:192318.637 Starting Zabbix Agent [W2T-FS01]. Zabbix 7.4.3 (revision 9c308d6f49c). 1372:20251006:192318.642 **** Enabled features **** 1372:20251006:192318.645 IPv6 support: YES 1372:20251006:192318.648 TLS support: YES 1372:20251006:192318.651 ************************** 1372:20251006:192318.655 using configuration file: C:\Program Files\Zabbix Agent\zabbix_agentd.conf 1372:20251006:192318.664 In zbx_init_collector_data() 1372:20251006:192318.668 logical CPU count 4 1372:20251006:192318.671 End of zbx_init_collector_data() 1372:20251006:192318.674 In zbx_init_perf_collector() 1372:20251006:192318.678 In zbx_init_builtin_counter_indexes() 1372:20251006:192318.682 In zbx_get_all_counter_names() 1372:20251006:192318.867 End of zbx_get_all_counter_names() 1372:20251006:192319.158 End of zbx_init_builtin_counter_indexes():SUCCEED 1372:20251006:192319.162 In set_object_names() 1372:20251006:192319.518 In zbx_get_all_counter_names() 1372:20251006:192319.526 End of zbx_get_all_counter_names() 1372:20251006:192319.530 In zbx_get_all_counter_names() 1372:20251006:192319.539 End of zbx_get_all_counter_names() 1372:20251006:192319.543 VEH Trap detected exception c0000005 at 0x00007FF8F82DF0AF. Exception information: 1372:20251006:192319.546 ====== Fatal information: ====== 1372:20251006:192319.549 Program counter: 0xf82df0af 1372:20251006:192319.552 === Registers: === 1372:20251006:192319.555 r8 = 1cc9a80 = 30186112 = 30186112 1372:20251006:192319.558 r9 = 0 = 0 = 0 1372:20251006:192319.561 r10 = 0 = 0 = 0 1372:20251006:192319.564 r11 = 4e0000 = 5111808 = 5111808 1372:20251006:192319.566 r12 = 0 = 0 = 0 1372:20251006:192319.569 r13 = 0 = 0 = 0 1372:20251006:192319.572 r14 = 1cc9a70 = 30186096 = 30186096 1372:20251006:192319.575 r15 = 1cbee00 = 30141952 = 30141952 1372:20251006:192319.578 rdi = 9c6 = 2502 = 2502 1372:20251006:192319.581 rsi = 1cbfe10 = 30146064 = 30146064 1372:20251006:192319.590 rbp = 134c779 = 20236153 = 20236153 1372:20251006:192319.594 rbx = 4e0000 = 5111808 = 5111808 1372:20251006:192319.597 rdx = 4e0000 = 5111808 = 5111808 1372:20251006:192319.601 rax = 4b9c0ac7 = 1268517575 = 1268517575 1372:20251006:192319.604 rcx = 0 = 0 = 0 1372:20251006:192319.607 rsp = 134c500 = 20235520 = 20235520 1372:20251006:192319.610 efl = 10246 = 66118 = 66118 1372:20251006:192319.614 csgsfs = 332b5300 = 858477312 = 858477312 1372:20251006:192319.617 === Backtrace: === 1372:20251006:192319.735 41: C:\Windows\SYSTEM32\ntdll.dll(RtlAllocateHeap+0x32ff) [0xf82df0af] 1372:20251006:192319.740 40: C:\Windows\SYSTEM32\ntdll.dll(RtlAllocateHeap+0x749) [0xf82dc4f9] 1372:20251006:192319.743 39: C:\Windows\SYSTEM32\AcLayers.DLL(NotifyShims+0x39ac) [0xa61c5b7c] 1372:20251006:192319.747 38: C:\Program Files\Zabbix Agent\zabbix_agentd.exe() [0x403f3f2b] 1372:20251006:192319.750 37: C:\Program Files\Zabbix Agent\zabbix_agentd.exe() [0x40400d38] 1372:20251006:192319.753 36: C:\Program Files\Zabbix Agent\zabbix_agentd.exe() [0x403fd045] 1372:20251006:192319.756 35: C:\Program Files\Zabbix Agent\zabbix_agentd.exe() [0x403dd8fe] 1372:20251006:192319.759 34: C:\Program Files\Zabbix Agent\zabbix_agentd.exe() [0x403dbe96] 1372:20251006:192319.762 33: C:\Program Files\Zabbix Agent\zabbix_agentd.exe() [0x403dba62] 1372:20251006:192319.765 32: C:\Program Files\Zabbix Agent\zabbix_agentd.exe() [0x403d9712] 1372:20251006:192319.768 31: C:\Program Files\Zabbix Agent\zabbix_agentd.exe() [0x403df6e6] 1372:20251006:192319.771 30: C:\Program Files\Zabbix Agent\zabbix_agentd.exe() [0x4010cffc] 1372:20251006:192319.774 29: C:\Program Files\Zabbix Agent\zabbix_agentd.exe() [0x40122602] 1372:20251006:192319.776 28: C:\Program Files\Zabbix Agent\zabbix_agentd.exe() [0x4010d1ae] 1372:20251006:192319.779 27: C:\Program Files\Zabbix Agent\zabbix_agentd.exe() [0x40153d0e] 1372:20251006:192319.783 26: C:\Windows\SYSTEM32\ntdll.dll(LdrGetDllHandleByName+0x2e2) [0xf833bfc2] 1372:20251006:192319.786 25: C:\Windows\SYSTEM32\ntdll.dll(RtlVirtualUnwind2+0x152) [0xf82f1722] 1372:20251006:192319.789 24: C:\Windows\SYSTEM32\ntdll.dll(RtlRaiseException+0x1f1) [0xf830fb01] 1372:20251006:192319.792 23: C:\Windows\SYSTEM32\ntdll.dll(RtlIsZeroMemory+0x119) [0xf83c3509] 1372:20251006:192319.795 22: C:\Windows\SYSTEM32\ntdll.dll(RtlIsZeroMemory+0xe3) [0xf83c34d3] 1372:20251006:192319.798 21: C:\Windows\SYSTEM32\ntdll.dll(_misaligned_access+0x41a) [0xf83cc2da] 1372:20251006:192319.801 20: C:\Windows\SYSTEM32\ntdll.dll(_misaligned_access+0x6fa) [0xf83cc5ba] 1372:20251006:192319.804 19: C:\Windows\SYSTEM32\ntdll.dll(_misaligned_access+0xad79) [0xf83d6c39] 1372:20251006:192319.807 18: C:\Windows\SYSTEM32\ntdll.dll(_misaligned_access+0x3a0) [0xf83cc260] 1372:20251006:192319.810 17: C:\Windows\SYSTEM32\ntdll.dll(RtlAllocateHeap+0x3a8e) [0xf82df83e] 1372:20251006:192319.813 16: C:\Windows\SYSTEM32\ntdll.dll(RtlAllocateHeap+0x749) [0xf82dc4f9] 1372:20251006:192319.817 15: C:\Windows\SYSTEM32\AcLayers.DLL(NotifyShims+0x39ac) [0xa61c5b7c] 1372:20251006:192319.820 14: C:\Windows\System32\ucrtbase.dll(malloc_base+0x36) [0xf5acffa6] 1372:20251006:192319.823 13: C:\Windows\System32\ucrtbase.dll(o_malloc+0x4c) [0xf5ade4bc] 1372:20251006:192319.827 12: C:\Windows\SYSTEM32\pdh.dll(PdhCloseLog+0xf8b) [0xe901d49b] 1372:20251006:192319.830 11: C:\Windows\SYSTEM32\pdh.dll(PdhCloseLog+0x812) [0xe901cd22] 1372:20251006:192319.833 10: C:\Windows\SYSTEM32\pdh.dll(PdhWriteRelogSample+0x1ccc) [0xe904972c] 1372:20251006:192319.836 9: C:\Windows\SYSTEM32\pdh.dll(PdhWriteRelogSample+0x1db3) [0xe9049813] 1372:20251006:192319.840 8: C:\Windows\SYSTEM32\pdh.dll(PdhWriteRelogSample+0xdd2) [0xe9048832] 1372:20251006:192319.842 7: C:\Windows\SYSTEM32\pdh.dll(PdhEnumObjectsHW+0xa2) [0xe904a862] 1372:20251006:192319.845 6: C:\Windows\SYSTEM32\pdh.dll(PdhEnumObjectsW+0xeb) [0xe904a99b] 1372:20251006:192319.848 5: C:\Program Files\Zabbix Agent\zabbix_agentd.exe() [0x4012e6cc] 1372:20251006:192319.851 4: C:\Program Files\Zabbix Agent\zabbix_agentd.exe() [0x4012ecf1] 1372:20251006:192319.854 3: C:\Program Files\Zabbix Agent\zabbix_agentd.exe() [0x4014ebf9] 1372:20251006:192319.857 2: C:\Windows\System32\sechost.dll(BuildSecurityDescriptorForSharingAccessEx+0xb32) [0xf61f8e32] 1372:20251006:192319.860 1: C:\Windows\System32\KERNEL32.DLL(BaseThreadInitThunk+0x10) [0xf6784cb0] 1372:20251006:192319.862 0: C:\Windows\SYSTEM32\ntdll.dll(RtlUserThreadStart+0x2b) [0xf833edcb] 1372:20251006:192319.869 ================================ ``` </details>
Additional Context
Suspected Code Location: Based on the backtrace and debug output, the issue likely resides in:
- src/libs/zbxsysinfo/win32/perfmon.c - Function zbx_get_all_counter_names()
- Buffer size calculation for PdhEnumObjectsW() API call
- Memory allocation/deallocation pattern in set_object_names()
Potential Root Causes:
- Insufficient buffer allocation for systems with large numbers of performance counters (2732 counters in this environment)
- Off-by-one error or integer overflow in buffer size calculation
- Memory corruption due to repeated calls without proper cleanup
- Unicode/locale-specific string handling issue (en-AU locale)
Why Configuration Workarounds Don't Help: The crash occurs during agent initialization, before configuration directives like DenyKey or AllowKey are evaluated. The PDH subsystem initialization is mandatory and happens regardless of what checks are configured.
Workaround
Temporary Solutions:
- Downgrade to 7.4.2 (confirmed working on same system)
- Use Zabbix Agent 2 (uses different PDH implementation, may not be affected)