The same issue occurs on a pure installation of Windows Server 2022, so in-place upgrade is not the cause of this issue.

Re: "The same issue occurs on a pure installation of Windows Server 2022, so in-place upgrade is not the cause of this issue."

Same here.  New install of Windows Server 2022, Zabbix agent immediately crashes when service is started.  Show stopper.

Using agent version 4.0 LTS, tried earlier versions with no change in outcome.

This appears to be listed as "trivial".  Not sure why when it is the agent is unusable with Windows Server 2022.

Could you please be so kind to increase log level and attach log as this could help to pinpoint the issue much faster, thanks !

Below is the log file with loglevel 4, let me know if you need more information.

4684:20210930:011807.934 ================================
7436:20211019:140801.320 Starting Zabbix Agent [firaidc01.dev.company.com]. Zabbix 5.0.15 (revision 40fb64d970).
7436:20211019:140801.322 **** Enabled features ****
7436:20211019:140801.324 IPv6 support: YES
7436:20211019:140801.326 TLS support: YES
7436:20211019:140801.327 **************************
7436:20211019:140801.328 using configuration file: C:\ProgramData\company\Zabbix\zabbix_agentd.conf
7436:20211019:140801.330 In init_collector_data()
7436:20211019:140801.332 logical CPU count 3
7436:20211019:140801.334 End of init_collector_data()
7436:20211019:140801.335 In init_perf_collector()
7436:20211019:140801.337 In init_builtin_counter_indexes()
7436:20211019:140801.338 In get_all_counter_names()
7436:20211019:140801.455 End of get_all_counter_names()
7436:20211019:140801.688 End of init_builtin_counter_indexes():SUCCEED
7436:20211019:140801.689 In set_object_names()
7436:20211019:140803.113 In get_all_counter_names()
7436:20211019:140803.121 End of get_all_counter_names()
7436:20211019:140803.123 In get_all_counter_names()
7436:20211019:140803.131 End of get_all_counter_names()
7436:20211019:140803.244 End of set_object_names():SUCCEED
7436:20211019:140803.246 End of init_perf_collector():SUCCEED
7436:20211019:140803.248 OpenSSL library (version OpenSSL 1.1.1k 25 Mar 2021) initialized
7436:20211019:140803.250 agent #0 started [main process]
7064:20211019:140803.252 agent #1 started [collector]
4136:20211019:140803.254 agent #2 started listener #1
7372:20211019:140803.256 agent #3 started listener #2
8640:20211019:140803.257 agent #8 started active checks #1
6296:20211019:140803.259 agent #5 started listener #4
5432:20211019:140803.260 agent #6 started listener #5
1568:20211019:140803.262 agent #7 started listener #6
1280:20211019:140803.263 agent #4 started listener #3
7064:20211019:140803.265 In init_cpu_collector()
4136:20211019:140803.266 In zbx_tls_init_child()
7372:20211019:140803.269 In zbx_tls_init_child()
8640:20211019:140803.270 In zbx_tls_init_child()
6296:20211019:140803.272 In zbx_tls_init_child()
5432:20211019:140803.273 In zbx_tls_init_child()
1568:20211019:140803.275 In zbx_tls_init_child()
1280:20211019:140803.276 In zbx_tls_init_child()
7064:20211019:140803.278 In get_counter_name() pdhIndex:238
4136:20211019:140803.280 End of zbx_tls_init_child()
7372:20211019:140803.281 End of zbx_tls_init_child()
8640:20211019:140803.283 End of zbx_tls_init_child()
6296:20211019:140803.284 End of zbx_tls_init_child()
5432:20211019:140803.286 End of zbx_tls_init_child()
1568:20211019:140803.288 End of zbx_tls_init_child()
1280:20211019:140803.289 End of zbx_tls_init_child()
7064:20211019:140803.291 End of get_counter_name():SUCCEED
8640:20211019:140803.292 In init_active_metrics()
7064:20211019:140803.294 In get_counter_name() pdhIndex:6
8640:20211019:140803.295 buffer: first allocation for 100 elements
7064:20211019:140803.297 End of get_counter_name():SUCCEED
8640:20211019:140803.298 End of init_active_metrics()
7064:20211019:140803.300 In add_perf_counter() counter:'\Processor(_Total)% Processor Time' interval:900
8640:20211019:140803.301 In send_buffer() host:'zabbix.dev.company.com' port:10051 entries:0/100
7064:20211019:140803.303 add_perf_counter(): PerfCounter '\Processor(_Total)% Processor Time' successfully added
8640:20211019:140803.305 End of send_buffer():SUCCEED
7064:20211019:140803.306 End of add_perf_counter(): SUCCEED
8640:20211019:140803.308 In refresh_active_checks() host:'zabbix.dev.company.com' port:10051
7064:20211019:140803.309 In add_perf_counter() counter:'\Processor(0)% Processor Time' interval:900
7064:20211019:140803.312 add_perf_counter(): PerfCounter '\Processor(0)% Processor Time' successfully added
7064:20211019:140803.313 End of add_perf_counter(): SUCCEED
7064:20211019:140803.314 In add_perf_counter() counter:'\Processor(1)% Processor Time' interval:900
7064:20211019:140803.315 add_perf_counter(): PerfCounter '\Processor(1)% Processor Time' successfully added
7064:20211019:140803.317 End of add_perf_counter(): SUCCEED
8640:20211019:140803.318 sending [

]
7064:20211019:140803.320 In add_perf_counter() counter:'\Processor(2)% Processor Time' interval:900
8640:20211019:140803.322 before read
7064:20211019:140803.323 add_perf_counter(): PerfCounter '\Processor(2)% Processor Time' successfully added
8640:20211019:140803.325 got [{"response":"success","data":[

,

,

,

,

,

]}]
7064:20211019:140803.326 End of add_perf_counter(): SUCCEED
8640:20211019:140803.328 In parse_list_of_checks()
7064:20211019:140803.329 In get_counter_name() pdhIndex:2
8640:20211019:140803.331 In add_check() key:'eventlog[application,,,ADSync,6100]' refresh:10800 lastlogsize:2087 mtime:0
7064:20211019:140803.332 End of get_counter_name():SUCCEED
8640:20211019:140803.334 End of add_check()
7064:20211019:140803.336 In get_counter_name() pdhIndex:44
8640:20211019:140803.337 In add_check() key:'eventlog[application,,,ADSync,6941]' refresh:10800 lastlogsize:2087 mtime:0
7064:20211019:140803.339 End of get_counter_name():SUCCEED
8640:20211019:140803.340 End of add_check()
7064:20211019:140803.342 In add_perf_counter() counter:'\System\Processor Queue Length' interval:900
8640:20211019:140803.343 In add_check() key:'eventlog[system,,,,133]' refresh:600 lastlogsize:14420 mtime:0
8640:20211019:140803.344 End of add_check()
7064:20211019:140803.346 add_perf_counter(): PerfCounter '\System\Processor Queue Length' successfully added
8640:20211019:140803.348 In add_check() key:'eventlog[system,,,Service Control Manager,7031]' refresh:60 lastlogsize:14423 mtime:0
7064:20211019:140803.349 End of add_perf_counter(): SUCCEED
8640:20211019:140803.351 End of add_check()
7064:20211019:140803.352 End of init_cpu_collector():SUCCEED
8640:20211019:140803.354 In add_check() key:'eventlog[system,,,disk,153]' refresh:120 lastlogsize:14420 mtime:0
7064:20211019:140803.355 In collect_perfstat()
8640:20211019:140803.357 End of add_check()
7064:20211019:140803.358 End of collect_perfstat()
8640:20211019:140803.360 In add_check() key:'eventlog[system,,,Microsoft-Windows-ActiveDirectory_DomainService,2095]' refresh:900 lastlogsize:14420 mtime:0
8640:20211019:140803.361 End of add_check()
8640:20211019:140803.363 End of parse_list_of_checks():SUCCEED
8640:20211019:140803.364 End of refresh_active_checks():SUCCEED
8640:20211019:140803.366 In process_active_checks() server:'zabbix.dev.company.com' port:10051
8640:20211019:140803.367 In initialize_eventlog6() source:'application' previous lastlogsize:2087
8640:20211019:140803.368 In zbx_open_eventlog6()
8640:20211019:140803.371 End of zbx_open_eventlog6():SUCCEED FirstID:1 LastID:16284 numIDs:16283
8640:20211019:140803.373 In zbx_get_handle_eventlog6(), previous lastlogsize:2087
8640:20211019:140803.374 End of zbx_get_handle_eventlog6():SUCCEED
8640:20211019:140803.376 End of initialize_eventlog6():SUCCEED
8640:20211019:140803.377 In process_eventslog6() source: 'application' previous lastlogsize: 2087, FirstID: 1, LastID: 16284
8640:20211019:140803.393 In zbx_parse_eventlog_message6() EventRecordID:2088
8640:20211019:140803.395 In expand_message6()
8640:20211019:140803.451 End of expand_message6():Faulting application name: zabbix_agentd.exe, version: 5.0.9.2400, time stamp: 0x6033a9f4
Faulting module name: ntdll.dll, version: 10.0.20348.112, time stamp: 0x09a64f97
Exception code: 0xc0000005
Fault offset: 0x000000000000fbf6
Faulting process id: 0x1154
Faulting application start time: 0x01d7a2e4f2e4499c
Faulting application path: C:\ProgramData\company\Zabbix\zabbix_agentd.exe
Faulting module path: C:\WINDOWS\SYSTEM32\ntdll.dll
Report Id: b423b889-51fa-4c95-8d64-5c06a50d3c13
Faulting package full name:
Faulting package-relative application ID:
8640:20211019:140803.453 End of zbx_parse_eventlog_message6():SUCCEED
8640:20211019:140803.455 In zbx_parse_eventlog_message6() EventRecordID:2089
8640:20211019:140803.457 In expand_message6()
8640:20211019:140803.489 End of expand_message6():Fault bucket , type 0
Event Name: APPCRASH
Response: Not available
Cab Id: 0

Problem signature:
P1: zabbix_agentd.exe
P2: 5.0.9.2400
P3: 6033a9f4
P4: ntdll.dll
P5: 10.0.20348.112
P6: 09a64f97
P7: c0000005
P8: 000000000000fbf6
P9:
P10:

Attached files:

These files may be available here:
\\?\C:\ProgramData\Microsoft\Windows\WER\ReportQueue\AppCrash_zabbix_agentd.ex_c1aa6490e9eacf4529e5975d0d3c9d0eaca36_9662dbab_e4d1bb36-c089-461a-9446-7feab2d25e9c

Analysis symbol:
Rechecking for solution: 0
Report Id: b423b889-51fa-4c95-8d64-5c06a50d3c13
Report Status: 4
Hashed bucket:
Cab Guid: 0
8640:20211019:140803.492 End of zbx_parse_eventlog_message6():SUCCEED
8640:20211019:140803.494 In zbx_parse_eventlog_message6() EventRecordID:2090
8640:20211019:140803.495 In expand_message6()
8640:20211019:140803.498 End of expand_message6():Fault bucket 2163426895897933667, type 4
Event Name: APPCRASH
Response: Not available
Cab Id: 0

Problem signature:
P1: zabbix_agentd.exe
P2: 5.0.9.2400
P3: 6033a9f4
P4: ntdll.dll
P5: 10.0.20348.112
P6: 09a64f97
P7: c0000005
P8: 000000000000fbf6
P9:
P10:

Attached files:
\\?\C:\ProgramData\Microsoft\Windows\WER\Temp\WER.25eae81a-d4da-4c8e-9ee3-14c2b559b325.tmp.WERInternalMetadata.xml

These files may be available here:
\\?\C:\ProgramData\Microsoft\Windows\WER\ReportArchive\AppCrash_zabbix_agentd.ex_c1aa6490e9eacf4529e5975d0d3c9d0eaca36_9662dbab_e4d1bb36-c089-461a-9446-7feab2d25e9c

Analysis symbol:
Rechecking for solution: 0
Report Id: b423b889-51fa-4c95-8d64-5c06a50d3c13
Report Status: 268435456
Hashed bucket: b18c1039672a8b715e06094cab85b763
Cab Guid: 0
8640:20211019:140803.499 End of zbx_parse_eventlog_message6():SUCCEED
8640:20211019:140803.501 In zbx_parse_eventlog_message6() EventRecordID:2091
8640:20211019:140803.502 In expand_message6()
8640:20211019:140803.504 End of expand_message6():Fault bucket 1368322795528311268, type 5
Event Name: FaultTolerantHeap
Response: Not available
Cab Id: 0

Problem signature:
P1: zabbix_agentd.exe
P2: 5.0.9.2400
P3: 6033A9F4
P4: ffffbaad
P5:
P6:
P7:
P8:
P9:
P10:

Attached files:
\\?\C:\WINDOWS\SERVIC~1\LOCALS~1\AppData\Local\Temp\FTHF1B8.tmp\fthempty.txt
\\?\C:\ProgramData\Microsoft\Windows\WER\Temp\WER.df4ec106-a8bf-4407-928c-c4ac4b7c0280.tmp.WERInternalMetadata.xml
\\?\C:\ProgramData\Microsoft\Windows\WER\Temp\WER.dfdf7122-639e-4204-8c4f-b61a79d5d7d2.tmp.xml
\\?\C:\ProgramData\Microsoft\Windows\WER\Temp\WER.c10bb6d6-c75d-4d12-a1ea-3a416becb7d5.tmp.csv
\\?\C:\ProgramData\Microsoft\Windows\WER\Temp\WER.0269d455-23d1-4433-9b99-f625f8aad446.tmp.txt

These files may be available here:

Analysis symbol:
Rechecking for solution: 0
Report Id: 52002056-948b-44de-bc56-4f367992c831
Report Status: 268435456
Hashed bucket: 58230cbd6dd6107382fd42553588d1e4
Cab Guid: 0
8640:20211019:140803.506 End of zbx_parse_eventlog_message6():SUCCEED
8640:20211019:140803.508 In zbx_parse_eventlog_message6() EventRecordID:2092
8640:20211019:140803.510 In expand_message6()
8640:20211019:140803.512 End of expand_message6():Faulting application name: zabbix_agentd.exe, version: 5.0.9.2400, time stamp: 0x6033a9f4
Faulting module name: zabbix_agentd.exe, version: 5.0.9.2400, time stamp: 0x6033a9f4
Exception code: 0xc0000005
Fault offset: 0x000000000009a3cb
Faulting process id: 0x10a4
Faulting application start time: 0x01d7a2e55f6de7f2
Faulting application path: C:\ProgramData\company\Zabbix\zabbix_agentd.exe
Faulting module path: C:\ProgramData\company\Zabbix\zabbix_agentd.exe
Report Id: 1c9e6d88-acce-4a34-9f10-d81da494f41f
Faulting package full name:
Faulting package-relative application ID:
8640:20211019:140803.513 End of zbx_parse_eventlog_message6():SUCCEED
8640:20211019:140803.515 In zbx_parse_eventlog_message6() EventRecordID:2093
8640:20211019:140803.516 In expand_message6()
8640:20211019:140803.518 End of expand_message6():Fault bucket 1376207129033732263, type 4
Event Name: APPCRASH
Response: Not available
Cab Id: 0

Problem signature:
P1: zabbix_agentd.exe
P2: 5.0.9.2400
P3: 6033a9f4
P4: zabbix_agentd.exe
P5: 5.0.9.2400
P6: 6033a9f4
P7: c0000005
P8: 000000000009a3cb
P9:
P10:

Attached files:
\\?\C:\ProgramData\Microsoft\Windows\WER\Temp\WER.3c6f8154-f260-44ff-a5fe-8c6f64953748.tmp.dmp
\\?\C:\ProgramData\Microsoft\Windows\WER\Temp\WER.043e9cd1-b838-460e-845a-95f90d26e600.tmp.WERInternalMetadata.xml
\\?\C:\ProgramData\Microsoft\Windows\WER\Temp\WER.126bd6a4-5782-4c59-9a63-6121c57c73cb.tmp.xml
\\?\C:\ProgramData\Microsoft\Windows\WER\Temp\WER.96ffeffc-aa65-471e-9129-66bae5fd48d9.tmp.csv
\\?\C:\ProgramData\Microsoft\Windows\WER\Temp\WER.47ae484c-e095-48de-9afb-93e3c0acc3e8.tmp.txt

These files may be available here:
\\?\C:\ProgramData\Microsoft\Windows\WER\ReportArchive\AppCrash_zabbix_agentd.ex_c63887a5f71c952cf93f5ea78e2d9673fa11262_9662dbab_6b026494-6a5b-4cb0-b39a-ea4c7a0fc943

Analysis symbol:
Rechecking for solution: 0
Report Id: 1c9e6d88-acce-4a34-9f10-d81da494f41f
Report Status: 268435456
Hashed bucket: 029ccce5975be2f183194517a76a70a7
Cab Guid: 0
8640:20211019:140803.520 End of zbx_parse_eventlog_message6():SUCCEED
8640:20211019:140803.522 In zbx_parse_eventlog_message6() EventRecordID:2094
8640:20211019:140803.524 In expand_message6()
8640:20211019:140803.525 End of expand_message6():Security policies were propagated with warning. 0x4b8 : An extended error has occurred.

Advanced help for this problem is available on https://support.microsoft.com. Query for "troubleshooting 1202 events".
8640:20211019:140803.527 End of zbx_parse_eventlog_message6():SUCCEED
8640:20211019:140803.529 In zbx_parse_eventlog_message6() EventRecordID:2095
8640:20211019:140803.531 In expand_message6()
8640:20211019:140803.533 End of expand_message6():Security policies were propagated with warning. 0x4b8 : An extended error has occurred.

I am sorry but there is no crash message in the attached log

Advanced help for this problem is available on https://support.microsoft.com. Query for "troubleshooting 1202 events".
4780:20211019:142339.993 End of zbx_parse_eventlog_message6():SUCCEED
4780:20211019:142339.994 In zbx_parse_eventlog_message6() EventRecordID:2168
4780:20211019:142339.996 Unhandled exception c0000005 detected at 0x00007FFC24163AD6. Crashing ...
4780:20211019:142339.996 ====== Fatal information: ======
4780:20211019:142339.997 Program counter: 0x24163ad6
4780:20211019:142339.997 === Registers: ===
4780:20211019:142339.997 r8 = 11e3890 = 18757776 = 18757776
4780:20211019:142339.998 r9 = 11e3898 = 18757784 = 18757784
4780:20211019:142339.998 r10 = 4f0324 = 5178148 = 5178148
4780:20211019:142339.999 r11 = 0 = 0 = 0
4780:20211019:142339.999 r12 = 1e0 = 480 = 480
4780:20211019:142339.999 r13 = 0 = 0 = 0
4780:20211019:142340.000 r14 = 2 = 2 = 2
4780:20211019:142340.000 r15 = 4f0000 = 5177344 = 5177344
4780:20211019:142340.000 rdi = 11e389 = 1172361 = 1172361
4780:20211019:142340.001 rsi = 11e3890 = 18757776 = 18757776
4780:20211019:142340.001 rbp = 11e38a0 = 18757792 = 18757792
4780:20211019:142340.001 rbx = 4f0000 = 5177344 = 5177344
4780:20211019:142340.002 rdx = 4f0a8f0 = 82880752 = 82880752
4780:20211019:142340.002 rax = 0 = 0 = 0
4780:20211019:142340.002 rcx = 11e389f = 18757791 = 18757791
4780:20211019:142340.003 rsp = 31cf1b0 = 52228528 = 52228528
4780:20211019:142340.003 efl = 10286 = 66182 = 66182
4780:20211019:142340.003 csgsfs = 332b5300 = 858477312 = 858477312
4780:20211019:142340.004 === Backtrace: ===
4752:20211019:142340.051 In collect_perfstat()
4752:20211019:142340.053 End of collect_perfstat()
4780:20211019:142340.146 13: C:\ProgramData\company\Zabbix\zabbix_agentd.exe(RtlReAllocateHeap+0x1276) [0x24163ad6]
4780:20211019:142340.146 12: C:\ProgramData\company\Zabbix\zabbix_agentd.exe(RtlReAllocateHeap+0x197) [0x241629f7]
4780:20211019:142340.147 11: C:\ProgramData\company\Zabbix\zabbix_agentd.exe(RtlReAllocateHeap+0x5a) [0x241628ba]
4780:20211019:142340.147 10: C:\ProgramData\company\Zabbix\zabbix_agentd.exe() [0x40220967]
4780:20211019:142340.148 9: C:\ProgramData\company\Zabbix\zabbix_agentd.exe() [0x40075b03]
4780:20211019:142340.148 8: C:\ProgramData\company\Zabbix\zabbix_agentd.exe() [0x4009e9bc]
4780:20211019:142340.148 7: C:\ProgramData\company\Zabbix\zabbix_agentd.exe() [0x4009d40c]
4780:20211019:142340.149 6: C:\ProgramData\company\Zabbix\zabbix_agentd.exe() [0x4009c633]
4780:20211019:142340.149 5: C:\ProgramData\company\Zabbix\zabbix_agentd.exe() [0x4009a348]
4780:20211019:142340.150 4: C:\ProgramData\company\Zabbix\zabbix_agentd.exe() [0x400993ef]
4780:20211019:142340.150 3: C:\ProgramData\company\Zabbix\zabbix_agentd.exe() [0x400844d7]
4780:20211019:142340.151 2: C:\ProgramData\company\Zabbix\zabbix_agentd.exe() [0x4021a148]
4780:20211019:142340.151 1: C:\ProgramData\company\Zabbix\zabbix_agentd.exe(BaseThreadInitThunk+0x10) [0x22bf4ed0]
4780:20211019:142340.152 0: C:\ProgramData\company\Zabbix\zabbix_agentd.exe(RtlUserThreadStart+0x2b) [0x241ce20b]
4780:20211019:142340.156 ================================
4752:20211019:142341.065 In collect_perfstat()
4752:20211019:142341.067 End of collect_perfstat()
4752:20211019:142342.081 In collect_perfstat()
4752:20211019:142342.082 End of collect_perfstat()
4752:20211019:142343.096 In collect_perfstat()
4752:20211019:142343.097 End of collect_perfstat()

Edit:

Seems loglevel 5 doesn't provide any more information.

Thank you, error occurs when reading eventlog, somewhere inside zbx_parse_eventlog_message6() function.

Reproduced with non administrator user, please try as a workaround not to use checks that are not allowed.

  5948:20211022:063027.455 active check "eventlog[Security]" is not supported: cannot open eventlog 'Security':[0x00000005] Access is denied.
  5948:20211022:063028.523 Unhandled exception c0000005 detected at 0x00007FFF60203AD6. Crashing ...
  5948:20211022:063028.524 ====== Fatal information: ======
  5948:20211022:063028.525 Program counter: 0x60203ad6
  5948:20211022:063028.525 === Registers: ===
  5948:20211022:063028.526 r8      =          22936e0 =             36255456 =             36255456
  5948:20211022:063028.527 r9      =          22936e8 =             36255464 =             36255464
  5948:20211022:063028.527 r10     =                3 =                    3 =                    3
  5948:20211022:063028.528 r11     =                0 =                    0 =                    0
  5948:20211022:063028.528 r12     =              178 =                  376 =                  376
  5948:20211022:063028.529 r13     =                0 =                    0 =                    0
  5948:20211022:063028.530 r14     =                2 =                    2 =                    2
  5948:20211022:063028.531 r15     =           4b0000 =              4915200 =              4915200
  5948:20211022:063028.531 rdi     =           22936e =              2265966 =              2265966
  5948:20211022:063028.532 rsi     =          22936e0 =             36255456 =             36255456
  5948:20211022:063028.533 rbp     =          22936f0 =             36255472 =             36255472
  5948:20211022:063028.533 rbx     =           4b0000 =              4915200 =              4915200
  5948:20211022:063028.534 rdx     =          fa249a0 =            262293920 =            262293920
  5948:20211022:063028.535 rax     =                0 =                    0 =                    0
  5948:20211022:063028.535 rcx     =          22936ef =             36255471 =             36255471
  5948:20211022:063028.539 rsp     =          391f1b0 =             59896240 =             59896240
  5948:20211022:063028.540 efl     =            10206 =                66054 =                66054
  5948:20211022:063028.540 csgsfs  =         332b5300 =            858477312 =            858477312
  5948:20211022:063028.541 === Backtrace: ===
  5948:20211022:063028.660 13: C:\Users\vso\Downloads\bin\zabbix_agentd.exe(RtlReAllocateHeap+0x1276) [0x60203ad6]
  5948:20211022:063028.661 12: C:\Users\vso\Downloads\bin\zabbix_agentd.exe(RtlReAllocateHeap+0x197) [0x602029f7]
  5948:20211022:063028.661 11: C:\Users\vso\Downloads\bin\zabbix_agentd.exe(RtlReAllocateHeap+0x5a) [0x602028ba]
  5948:20211022:063028.662 10: C:\Users\vso\Downloads\bin\zabbix_agentd.exe() [0x4021f847]
  5948:20211022:063028.663 9: C:\Users\vso\Downloads\bin\zabbix_agentd.exe() [0x40075b03]
  5948:20211022:063028.663 8: C:\Users\vso\Downloads\bin\zabbix_agentd.exe() [0x4009e8fc]
  5948:20211022:063028.664 7: C:\Users\vso\Downloads\bin\zabbix_agentd.exe() [0x4009d34c]
  5948:20211022:063028.664 6: C:\Users\vso\Downloads\bin\zabbix_agentd.exe() [0x4009c573]
  5948:20211022:063028.665 5: C:\Users\vso\Downloads\bin\zabbix_agentd.exe() [0x4009a2a8]
  5948:20211022:063028.666 4: C:\Users\vso\Downloads\bin\zabbix_agentd.exe() [0x4009934f]
  5948:20211022:063028.666 3: C:\Users\vso\Downloads\bin\zabbix_agentd.exe() [0x40084487]
  5948:20211022:063028.667 2: C:\Users\vso\Downloads\bin\zabbix_agentd.exe() [0x40219028]
  5948:20211022:063028.667 1: C:\Users\vso\Downloads\bin\zabbix_agentd.exe(BaseThreadInitThunk+0x10) [0x5fda4ed0]
  5948:20211022:063028.668 0: C:\Users\vso\Downloads\bin\zabbix_agentd.exe(RtlUserThreadStart+0x2b) [0x6026e20b]
  5948:20211022:063028.675 ================================

C:\Users\vso\Downloads\bin>

Fixed in pull request feature/ZBX-19926-4.0

It is unclear why it crashes on Windows server 2022 and not on any other Windows version, however it seems that documentation cannot be trusted in that exact case so slightly changed code to be exactly the same as example in Windows documentation to avoid the issue.

Fixed in:

• pre-4.0.36rc1 bfa8819aeb0
• pre-5.0.18rc1 049e2001f2f
• pre-5.4.8rc1 5f1d712db98
• pre-6.0.0alpha6 (master) d96c48dc1d4

I don't have a development environment for Windows Server 22 and no update is available from the Zabbix Agent download page.  What does it take to get an MSI installer and how does that happen?

[EDIT]: Version 4.0 LTS

In Windows Server 2019 with Zabbix Agent 6.2.x (6.2.3 and 6.2.4) it came back.

-----------------

  2304:20221031:155029.440 Zabbix Agent stopped. Zabbix 6.2.3 (revision 98ee88fc19d).
  5276:20221031:155034.336 Starting Zabbix Agent [REMOVED]. Zabbix 6.2.3 (revision 98ee88fc19d).
  5276:20221031:155034.337 **** Enabled features ****
  5276:20221031:155034.337 IPv6 support:          YES
  5276:20221031:155034.338 TLS support:           YES
  5276:20221031:155034.339 **************************
  5276:20221031:155034.339 using configuration file: c:\Program Files\Zabbix\zabbix_agentd.conf
  5276:20221031:155035.386 agent #0 started [main process]
  4500:20221031:155035.387 agent #1 started [collector]
   156:20221031:155035.390 agent #2 started [listener #1]
  5860:20221031:155035.391 agent #3 started [listener #2]
  4128:20221031:155035.391 agent #4 started [listener #3]
  4256:20221031:155035.392 agent #5 started [active checks #1]
  4256:20221031:160035.213 no active checks on server [*REMOVED*.cloud:10051]: host [*REMOVED*] not monitored
  5860:20221031:160309.289 Unhandled exception c0000005 detected at 0x0000000140094CC7. Crashing ...
  5860:20221031:160309.290 ====== Fatal information: ======
  5860:20221031:160309.291 Program counter: 0x40094cc7
  5860:20221031:160309.292 === Registers: ===
  5860:20221031:160309.292 r8      =                0 =                    0 =                    0
  5860:20221031:160309.293 r9      =                1 =                    1 =                    1
  5860:20221031:160309.294 r10     =                0 =                    0 =                    0
  5860:20221031:160309.295 r11     =              246 =                  582 =                  582
  5860:20221031:160309.295 r12     =             1000 =                 4096 =                 4096
  5860:20221031:160309.296 r13     =                0 =                    0 =                    0
  5860:20221031:160309.297 r14     =                0 =                    0 =                    0
  5860:20221031:160309.297 r15     =              4fc =                 1276 =                 1276
  5860:20221031:160309.298 rdi     =                0 =                    0 =                    0
  5860:20221031:160309.299 rsi     =          29cec30 =             43838512 =             43838512
  5860:20221031:160309.300 rbp     =                1 =                    1 =                    1
  5860:20221031:160309.300 rbx     =              690 =                 1680 =                 1680
  5860:20221031:160309.301 rdx     =          29ce0f0 =             43835632 =             43835632
  5860:20221031:160309.302 rax     =      1ddb0fd55c5 =        2051668792773 =        2051668792773
  5860:20221031:160309.303 rcx     =              690 =                 1680 =                 1680
  5860:20221031:160309.303 rsp     =          29cd9f0 =             43833840 =             43833840
  5860:20221031:160309.304 efl     =            10246 =                66118 =                66118
  5860:20221031:160309.304 csgsfs  =         332b5300 =            858477312 =            858477312
  5860:20221031:160309.304 === Backtrace: ===
  5860:20221031:160309.345 8: C:\Program Files\Zabbix\zabbix_agentd.exe() [0x40094cc7]
  5860:20221031:160309.346 7: C:\Program Files\Zabbix\zabbix_agentd.exe() [0x40094be8]
  5860:20221031:160309.347 6: C:\Program Files\Zabbix\zabbix_agentd.exe() [0x4009a663]
  5860:20221031:160309.347 5: C:\Program Files\Zabbix\zabbix_agentd.exe() [0x400a478f]
  5860:20221031:160309.348 4: C:\Program Files\Zabbix\zabbix_agentd.exe() [0x400a4632]
  5860:20221031:160309.348 3: C:\Program Files\Zabbix\zabbix_agentd.exe() [0x40085627]
  5860:20221031:160309.349 2: C:\Program Files\Zabbix\zabbix_agentd.exe() [0x4021ffc0]
  5860:20221031:160309.349 1: C:\Program Files\Zabbix\zabbix_agentd.exe(BaseThreadInitThunk+0x14) [0x47ad7974]
  5860:20221031:160309.350 0: C:\Program Files\Zabbix\zabbix_agentd.exe(RtlUserThreadStart+0x21) [0x480aa2f1] 

  5276:20221109:102635.079 Starting Zabbix Agent [*REMOVED*]. Zabbix 6.2.4 (revision 885eaefa6f7).
  5276:20221109:102635.080 **** Enabled features ****
  5276:20221109:102635.081 IPv6 support:          YES
  5276:20221109:102635.081 TLS support:           YES
  5276:20221109:102635.082 **************************
  5276:20221109:102635.083 using configuration file: c:\Program Files\Zabbix\zabbix_agentd.conf
  5276:20221109:102636.212 agent #0 started [main process]
  3400:20221109:102636.213 agent #1 started [collector]
  2504:20221109:102636.213 agent #2 started [listener #1]
  1760:20221109:102636.214 agent #3 started [listener #2]
  5596:20221109:102636.215 agent #4 started [listener #3]
  2260:20221109:102636.215 agent #5 started [active checks #1]
  2504:20221109:102709.434 Unhandled exception c0000005 detected at 0x0000000140094CC7. Crashing ...
  2504:20221109:102709.435 ====== Fatal information: ======
  2504:20221109:102709.436 Program counter: 0x40094cc7
  2504:20221109:102709.436 === Registers: ===
  2504:20221109:102709.437 r8      =                0 =                    0 =                    0
  2504:20221109:102709.438 r9      =                1 =                    1 =                    1
  2504:20221109:102709.438 r10     =                0 =                    0 =                    0
  2504:20221109:102709.439 r11     =              246 =                  582 =                  582
  2504:20221109:102709.439 r12     =             1000 =                 4096 =                 4096
  2504:20221109:102709.440 r13     =                0 =                    0 =                    0
  2504:20221109:102709.440 r14     =                0 =                    0 =                    0
  2504:20221109:102709.441 r15     =              648 =                 1608 =                 1608
  2504:20221109:102709.441 rdi     =                0 =                    0 =                    0
  2504:20221109:102709.442 rsi     =          2b8ec30 =             45673520 =             45673520
  2504:20221109:102709.442 rbp     =                1 =                    1 =                    1
  2504:20221109:102709.443 rbx     =              664 =                 1636 =                 1636
  2504:20221109:102709.443 rdx     =          2b8e0f0 =             45670640 =             45670640
  2504:20221109:102709.444 rax     =     69af11ad3f6c =      116200636759916 =      116200636759916
  2504:20221109:102709.444 rcx     =              664 =                 1636 =                 1636
  2504:20221109:102709.445 rsp     =          2b8d9f0 =             45668848 =             45668848
  2504:20221109:102709.445 efl     =            10246 =                66118 =                66118
  2504:20221109:102709.446 csgsfs  =         332b5300 =            858477312 =            858477312
  2504:20221109:102709.446 === Backtrace: ===
  2504:20221109:102709.511 8: C:\Program Files\Zabbix\zabbix_agentd.exe() [0x40094cc7]
  2504:20221109:102709.511 7: C:\Program Files\Zabbix\zabbix_agentd.exe() [0x40094be8]
  2504:20221109:102709.512 6: C:\Program Files\Zabbix\zabbix_agentd.exe() [0x4009a643]
  2504:20221109:102709.513 5: C:\Program Files\Zabbix\zabbix_agentd.exe() [0x400a47af]
  2504:20221109:102709.514 4: C:\Program Files\Zabbix\zabbix_agentd.exe() [0x400a4652]
  2504:20221109:102709.514 3: C:\Program Files\Zabbix\zabbix_agentd.exe() [0x40085627]
  2504:20221109:102709.515 2: C:\Program Files\Zabbix\zabbix_agentd.exe() [0x4021ffe0]
  2504:20221109:102709.515 1: C:\Program Files\Zabbix\zabbix_agentd.exe(BaseThreadInitThunk+0x14) [0x6bc7974]
  2504:20221109:102709.516 0: C:\Program Files\Zabbix\zabbix_agentd.exe(RtlUserThreadStart+0x21) [0x752a371] 

With Agent in Version: Zabbix 6.0.10 (LTS) all working fine.

I can confirm this also affects Windows 10 EDU 22H2 BUILD 19045.2604.

676:20230720:195847.256 Starting Zabbix Agent [REDACTED]. Zabbix 6.4.4 (revision a749236b3d9).
   676:20230720:195847.261 **** Enabled features ****
   676:20230720:195847.265 IPv6 support:          YES
   676:20230720:195847.266 TLS support:           YES
   676:20230720:195847.268 **************************
   676:20230720:195847.270 using configuration file: C:\Program Files\Zabbix Agent\zabbix_agentd.conf
   676:20230720:195848.419 agent #0 started [main process]
  6964:20230720:195848.430 agent #1 started [collector]
 10576:20230720:195848.430 agent #2 started [listener #1]
 10972:20230720:195848.431 agent #3 started [listener #2]
  9484:20230720:195848.464 agent #4 started [listener #3]
 10288:20230720:195848.465 agent #5 started [active checks #1]
  ...
  SOME 20 Other lines of info about network connection refusal from other zabbix servers (intended)
  ...
  9484:20230721:024534.130 Unhandled exception c0000005 detected at 0x0000000140106E83. Crashing ...
  9484:20230721:024534.131 ====== Fatal information: ======
  9484:20230721:024534.132 Program counter: 0x40106e83
  9484:20230721:024534.132 === Registers: ===
  9484:20230721:024534.133 r8      =           1a3de0 =              1719776 =              1719776
  9484:20230721:024534.134 r9      =                1 =                    1 =                    1
  9484:20230721:024534.134 r10     =             8000 =                32768 =                32768
  9484:20230721:024534.149 r11     =          2e8ea40 =             48818752 =             48818752
  9484:20230721:024534.158 r12     =                0 =                    0 =                    0
  9484:20230721:024534.160 r13     =                0 =                    0 =                    0
  9484:20230721:024534.161 r14     =           d9ecd8 =             14281944 =             14281944
  9484:20230721:024534.162 r15     =               14 =                   20 =                   20
  9484:20230721:024534.163 rdi     =          2e8ecc0 =             48819392 =             48819392
  9484:20230721:024534.164 rsi     =           f0fca0 =             15793312 =             15793312
  9484:20230721:024534.164 rbp     =                0 =                    0 =                    0
  9484:20230721:024534.165 rbx     =           f299b0 =             15899056 =             15899056
  9484:20230721:024534.167 rdx     =        140198480 =           5370381440 =           5370381440
  9484:20230721:024534.167 rax     =               20 =                   32 =                   32
  9484:20230721:024534.168 rcx     =             1a70 =                 6768 =                 6768
  9484:20230721:024534.170 rsp     =          2e8ea80 =             48818816 =             48818816
  9484:20230721:024534.170 efl     =            10202 =                66050 =                66050
  9484:20230721:024534.171 csgsfs  =         332b5300 =            858477312 =            858477312
  9484:20230721:024534.172 === Backtrace: ===
  9484:20230721:024534.320 14: C:\Program Files\Zabbix Agent\zabbix_agentd.exe() [0x40106e83]
  9484:20230721:024534.321 13: C:\Program Files\Zabbix Agent\zabbix_agentd.exe() [0x4019849b]
  9484:20230721:024534.324 12: C:\Program Files\Zabbix Agent\zabbix_agentd.exe() [0x40109a24]
  9484:20230721:024534.325 11: C:\Program Files\Zabbix Agent\zabbix_agentd.exe() [0x400f7502]
  9484:20230721:024534.326 10: C:\Program Files\Zabbix Agent\zabbix_agentd.exe() [0x4011fe29]
  9484:20230721:024534.327 9: C:\Program Files\Zabbix Agent\zabbix_agentd.exe() [0x4011fdca]
  9484:20230721:024534.328 8: C:\Program Files\Zabbix Agent\zabbix_agentd.exe() [0x401f36c0]
  9484:20230721:024534.329 7: C:\Program Files\Zabbix Agent\zabbix_agentd.exe() [0x401d0e73]
  9484:20230721:024534.331 6: C:\Program Files\Zabbix Agent\zabbix_agentd.exe() [0x400b320b]
  9484:20230721:024534.332 5: C:\Program Files\Zabbix Agent\zabbix_agentd.exe() [0x4007d97e]
  9484:20230721:024534.333 4: C:\Program Files\Zabbix Agent\zabbix_agentd.exe() [0x400a63f6]
  9484:20230721:024534.334 3: C:\Program Files\Zabbix Agent\zabbix_agentd.exe() [0x400865f7]
  9484:20230721:024534.335 2: C:\Program Files\Zabbix Agent\zabbix_agentd.exe() [0x40221d50]
  9484:20230721:024534.335 1: C:\Windows\System32\KERNEL32.DLL(BaseThreadInitThunk+0x14) [0x9ce17614]
  9484:20230721:024534.336 0: C:\Windows\SYSTEM32\ntdll.dll(RtlUserThreadStart+0x21) [0x9de826a1]
  9484:20230721:024534.367 ================================ 

I laso noticed, that it does not crash right after the service starts, but some time later.

Experiencing the same issue on Windows Server 2022. Here is the log of the crash:

 10172:20230721:194918.445 Unhandled exception c0000005 detected at 0x00007FFD8620D160. Crashing ...
 10172:20230721:194918.461 ====== Fatal information: ======
 10172:20230721:194918.474 Program counter: 0x8620d160
 10172:20230721:194918.488 === Registers: ===
 10172:20230721:194918.502 r8      =                f =                   15 =                   15
 10172:20230721:194918.516 r9      =           519530 =              5346608 =              5346608
 10172:20230721:194918.530 r10     =               50 =                   80 =                   80
 10172:20230721:194918.543 r11     =               5f =                   95 =                   95
 10172:20230721:194918.555 r12     =           4b6450 =              4940880 =              4940880
 10172:20230721:194918.567 r13     =          1149480 =             18125952 =             18125952
 10172:20230721:194918.582 r14     =              218 =                  536 =                  536
 10172:20230721:194918.594 r15     =                0 =                    0 =                    0
 10172:20230721:194918.606 rdi     =             2440 =                 9280 =                 9280
 10172:20230721:194918.620 rsi     =           519530 =              5346608 =              5346608
 10172:20230721:194918.635 rbp     =          2a2e8d0 =             44230864 =             44230864
 10172:20230721:194918.649 rbx     =                0 =                    0 =                    0
 10172:20230721:194918.661 rdx     =                1 =                    1 =                    1
 10172:20230721:194918.672 rax     =                1 =                    1 =                    1
 10172:20230721:194918.686 rcx     =         fffffff1 =           4294967281 =           4294967281
 10172:20230721:194918.700 rsp     =          2a2e7d0 =             44230608 =             44230608
 10172:20230721:194918.715 efl     =            10202 =                66050 =                66050
 10172:20230721:194918.728 csgsfs  =         332b5300 =            858477312 =            858477312
 10172:20230721:194918.743 === Backtrace: ===
 10172:20230721:194918.922 18: C:\Windows\System32\ADVAPI32.dll(SaferIdentifyLevel+0x1290) [0x8620d160]
 10172:20230721:194918.934 17: C:\Windows\System32\ADVAPI32.dll(SaferIdentifyLevel+0x1078) [0x8620cf48]
 10172:20230721:194918.946 16: C:\Windows\System32\ADVAPI32.dll(SaferIdentifyLevel+0x403d) [0x8620ff0d]
 10172:20230721:194918.958 15: C:\Windows\System32\ADVAPI32.dll(SaferIdentifyLevel+0x380e) [0x8620f6de]
 10172:20230721:194918.970 14: C:\Windows\System32\ADVAPI32.dll(PerfRegQueryValue+0x31) [0x862102a1]
 10172:20230721:194918.981 13: C:\Windows\System32\KERNELBASE.dll(UnsubscribeFeatureUsageFlush+0xc06) [0x85de0f76]
 10172:20230721:194918.993 12: C:\Windows\System32\KERNELBASE.dll(RegQueryValueExW+0x108) [0x85d67df8]
 10172:20230721:194919.006 11: C:\Windows\SYSTEM32\pdh.dll(PdhCloseQuery+0x628) [0x80bd8ae8]
 10172:20230721:194919.018 10: C:\Windows\SYSTEM32\pdh.dll(PdhGetCounterTimeBase+0x63d) [0x80bda37d]
 10172:20230721:194919.030 9: C:\Windows\SYSTEM32\pdh.dll(PdhCollectQueryData+0x31a0) [0x80bd8450]
 10172:20230721:194919.041 8: C:\Windows\SYSTEM32\pdh.dll(PdhGetCounterTimeBase+0x3a75) [0x80bdd7b5]
 10172:20230721:194919.053 7: C:\Windows\SYSTEM32\pdh.dll(PdhCollectQueryData+0xbf) [0x80bd536f]
 10172:20230721:194919.065 6: C:\Windows\SYSTEM32\pdh.dll(PdhCollectQueryData+0x4b) [0x80bd52fb]
 10172:20230721:194919.077 5: C:\Zabbix\zabbix_agentd.exe() [0x400abdf9]
 10172:20230721:194919.088 4: C:\Zabbix\zabbix_agentd.exe() [0x400ad337]
 10172:20230721:194919.100 3: C:\Zabbix\zabbix_agentd.exe() [0x400865f7]
 10172:20230721:194919.112 2: C:\Zabbix\zabbix_agentd.exe() [0x40221d50]
 10172:20230721:194919.124 1: C:\Windows\System32\KERNEL32.DLL(BaseThreadInitThunk+0x14) [0x86487ac4]
 10172:20230721:194919.136 0: C:\Windows\SYSTEM32\ntdll.dll(RtlUserThreadStart+0x21) [0x88fea351]
 10172:20230721:194919.150 =============================== 

What's strange is that not all the Windows Servers in my environment have the issue. Only a select few. The other ones work flawlessly with no crashes. And as Robert stated above, the crash occurs sometime later and not right away.

Hi zak2273,

Be so kind to make a test with the zabbix_agentd.exe build from comment

If will no crash several days - give information

Hi @Michael Veksler,

I am testing one of the servers with frequent crashing using this new build.
Will keep you posted

Hello @MVekslers,

I tried the new build and it crashed at least four times on the testing machine (same as I mentioned before). I also tried that on Windows Server 2019 (17763.3165) with similar amount of crashes.

I Think I might try Zabbix Agent 2 to run on some more machines to check the stability.

My test server also crashed as well. I attached the log file.

I've been digging through the Zabbix code and suspect this error may be somehow related to the zbx_execute_agent_check() function that is called within process_listener. Perhaps the result variable or the in_command variable are accessing some invalid memory location. I haven't been able to determine why though. Maybe this will help lead in the right direction.

Hi zak2273,
Thanks for tests and log,
but the log shows that the agent just catches "unhandled exception" but didn't restart (crashed)
I suspect this noise is related to this issue  ZBX-22494

Please confirm the agent has crashed (stopped)

Hi laneo 

Can you upload log with LogLevel=5

Hi @Michael Veksler,

Agent does indeed crash. See at line 14794:

 13092:20230727:014329.012 Unhandled exception c0000005 detected at 0x00007FFCE3AEA6F2. Crashing ... 

After producing the back trace, few more entries are recorded and then the Agent stops completely. It is confirmed since when I went into Services, the Agent was not running.

Hi,

I attached the logfile with DebugLevel=5. Crash is at line 22701. I have the service configured to start automatically again after crashing, but I can confirm that the process does indeed exit.

laneo_zabbix_agentd.log

Hi laneo,

Be so kind to make the test with DebugLevel=4

The reason has to do with our new VEH handler. Level 4 will switch off this functionality. I suspect that we caught "normal" cpp exception from win api and stopped the agent "without the reason".

Hello @Michael Veksler,

I attached the log with DebugLevel 4. The crash is on line 55507 and subsequent automatic restart of the service is at 55904.

Sorry for the delay, I was on vacation.

laneo_zbx_agnt_log_lvl_4.txt

Hi laneo,

Thank you very much for the logs, I think the problem has been found and the solution has been moved to ZBX-23068