[ZBX-19309] Zabbix Agent 2 on Windows crashes when using external UserParameter script Created: 2021 Apr 29  Updated: 2023 Mar 28  Resolved: 2023 Mar 28

Status: Closed
Project: ZABBIX BUGS AND ISSUES
Component/s: Agent (G)
Affects Version/s: 5.0.10, 5.0.11
Fix Version/s: None

Type: Incident report Priority: Trivial
Reporter: Karel Bělunek Assignee: Zabbix Support Team
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

Windows 2021 server

Attachments: File hyperv_host.ps1     PNG File image-2021-05-05-13-59-04-964.png     File zabbix_agent2.conf     Text File zabbix_agent2.log    

 Description   

Steps to reproduce:

  1. Zabbix Agent2 service process on windows 2012 crashing without any log or eventlog traces even in DEBUG mode, id the agent is configured to execute external UserParameter script. 
  2. The crash occurs cca after 5-20 minutes of Zabbix Agent 2 run, at least some data is sent into Zabbix Sever before the agent 2 crashes.
  3. In our environment, the powershell script for Hyper-V monitoring (https://share.zabbix.com/virtualization/hyper-v/template-microsoft-hyper-v-host) has been triggered by Zabbix Agent2 in active mode.
  4. Zabbix Agent2 on other windows servers without running UserParameter script works well
  5. We tested agent 2 versions 5.0.8, 5.0.10 and 5.0.11 - every version is affected by this bug.
  6. When downgraded to Zabbix Agent 1 (5.0.11) the agent 1 with forementioned powershell script works flawless.

Result:
Zabbix agent2 is crashed
Expected:

 Comments   
Comment by Edgar Akhmetshin [ 2021 May 03 ]

Hello Karel,

Thank you for reporting the issue. Currently trying to reproduce with Windows 10 Pro 20H2 but looks like agent2 is not crashing using template from (you link is broken, but this one should be the same): https://github.com/itmicus/zabbix/tree/master/Template%20Microsoft%20Hyper-V

Also tried to "spam" agent with discovery requests but with no luck to get crash:

bash-5.0$ for i in {1..1000}; do zabbix_get -s 192.168.2.3 -k hyperv.host[discover,va]; done
{
    "data":  [
                 {
                     "{#HYPERV_NETWORKADAPTER_NAME}":  "vEthernet (Home)",
                     "{#HYPERV_NETWORKADAPTER_PERF}":  "Hyper-V Virtual Ethernet Adapter",
                     "{#HYPERV_NETWORKADAPTER_INTERFACEINDEX}":  21
                 },
                 {
                     "{#HYPERV_NETWORKADAPTER_NAME}":  "vEthernet (vEthernet (Home)",
                     "{#HYPERV_NETWORKADAPTER_PERF}":  "Hyper-V Virtual Ethernet Adapter _2",
                     "{#HYPERV_NETWORKADAPTER_INTERFACEINDEX}":  29
                 },
                 {
                     "{#HYPERV_NETWORKADAPTER_NAME}":  "vEthernet (Default Switch)",
                     "{#HYPERV_NETWORKADAPTER_PERF}":  "Hyper-V Virtual Ethernet Adapter _3",
                     "{#HYPERV_NETWORKADAPTER_INTERFACEINDEX}":  35
                 },
                 {
                     "{#HYPERV_NETWORKADAPTER_NAME}":  "vEthernet (Беспроводная се)",
                     "{#HYPERV_NETWORKADAPTER_PERF}":  "Hyper-V Virtual Ethernet Adapter _4",
                     "{#HYPERV_NETWORKADAPTER_INTERFACEINDEX}":  52
                 }
             ]
}
{
    "data":  [

Could you please specify exact Windows version and details like service packs installed. Also if template is different and you have modified PowerShell script from the original, share all details.

And one more check, that happens if you execute and run Agent 2 from powershell/cmd, not as a service? will it crash?

Regards,
Edgar

Comment by Karel Bělunek [ 2021 May 03 ]

Hi Edgar,

 

thanks for the response.

I will get the exact version of windows and the powershell script tomorrow and i will report it as you requested.

In the meantime, it looks that the problem occurs only when Zabbix Agent is in the active mode (as template defines), after some time (cca 15-40 minutes of normal work and pushing data) the agent goes down without trace in log.

My passive tests based on zabbix_gets calls also works without any problem.

I will also test how the zabbix agent 2 behaves when running from cmd.exe instead from service.

 

Thanks and regards

Karel Belunek

 

Comment by Karel Bělunek [ 2021 May 05 ]

Hi Edgar,

 

there is info I promised.

 

Just a quick recap. The agent 2 falls after some time (it varies from minutes to hours) when the agent is runned as system service, uzing zabbix active checks and using UserParameter script calls. Also persistent buffer is enabled but it does not cause any problem on other monitored windows machines. 
The older agent 1 works without any problem in this setup (without persistentbuffer, because not supported on version 1).
The new agent 2 works fine on many other windows machines, that does not host hyper-v or utilising other UserParameter checks, these other windows machines are monitored by simplified default windows template provided by Zabbix.

I started zabbix_agent2 from console as you requested, running from cmd.exe started with Administrator provileges.

There is commnadline used in cmd, the time commnads only shows time of start and fall of the zabbix agent:

time /T & zabbix_agent2.exe -c zabbix_agent2.conf & time /T

There is output after agent fall including exception trace

CMD output, it showt that run started at 09:43 and falled at 13:21

 

09:43
Starting Zabbix Agent 2 [H033001]. (5.0.11)
Press Ctrl+C to exit.
Exception 0xc0000005 0x0 0xffffffffffffffff 0x7ffb29f094db
PC=0x7ffb29f094db
signal arrived during external code executionsyscall.Syscall(0x7ffb29f08e30, 0x1, 0xc0002f0d80, 0x0, 0x0, 0x0, 0x0, 0x0)
        c:/go/src/runtime/syscall_windows.go:188 +0xe9
syscall.(*Proc).Call(0xc00022f180, 0xc0004c0020, 0x1, 0x1, 0x1040c30, 0xc00000f558, 0xa7cbb9, 0x7ffb29ea2d80)
        c:/go/src/syscall/dll_windows.go:171 +0x136
github.com/go-ole/go-ole.VariantClear(0xc0002f0d80, 0x0, 0x0)
        C:/Users/ZABBIX/go/pkg/mod/github.com/go-ole/[email protected]/com.go:248 +0x76
github.com/go-ole/go-ole.(*VARIANT).Clear(...)
        C:/Users/ZABBIX/go/pkg/mod/github.com/go-ole/[email protected]/variant.go:47
zabbix.com/pkg/wmi.clearOle(0xc0002f0d80)
        C:/Users/ZABBIX/build-agents-web/zabbix-5.0.11/src/go/pkg/wmi/wmi.go:54 +0x36
zabbix.com/pkg/wmi.isPropertyKeyProperty(0x2b143048, 0xf02801, 0x0, 0x0)
        C:/Users/ZABBIX/build-agents-web/zabbix-5.0.11/src/go/pkg/wmi/wmi.go:93 +0x17e
zabbix.com/pkg/wmi.(*valueResult).write.func1.1(0xc0002f0d00, 0x0, 0x0)
        C:/Users/ZABBIX/build-agents-web/zabbix-5.0.11/src/go/pkg/wmi/wmi.go:147 +0x224
github.com/go-ole/go-ole/oleutil.ForEach(0x2b142e88, 0xc00000f8b0, 0x0, 0x0)
        C:/Users/ZABBIX/go/pkg/mod/github.com/go-ole/[email protected]/oleutil/oleutil.go:122 +0x20b
zabbix.com/pkg/wmi.(*valueResult).write.func1(0xc0002f0c60, 0x0, 0x0)
        C:/Users/ZABBIX/build-agents-web/zabbix-5.0.11/src/go/pkg/wmi/wmi.go:127 +0x18a
github.com/go-ole/go-ole/oleutil.ForEach(0x2b143208, 0xc00000fa20, 0x0, 0x0)
        C:/Users/ZABBIX/go/pkg/mod/github.com/go-ole/[email protected]/oleutil/oleutil.go:122 +0x20b
zabbix.com/pkg/wmi.(*valueResult).write(0xc00027a990, 0x2b143208, 0x0, 0x0)
        C:/Users/ZABBIX/build-agents-web/zabbix-5.0.11/src/go/pkg/wmi/wmi.go:114 +0x12f
zabbix.com/pkg/wmi.performQuery(0xc00050e050, 0xa, 0xc0003da750, 0x8c, 0x110f860, 0xc00027a990, 0x0, 0x0)
        C:/Users/ZABBIX/build-agents-web/zabbix-5.0.11/src/go/pkg/wmi/wmi.go:298 +0x43f
zabbix.com/pkg/wmi.QueryValue(...)
        C:/Users/ZABBIX/build-agents-web/zabbix-5.0.11/src/go/pkg/wmi/wmi.go:306
zabbix.com/plugins/windows/wmi.(*Plugin).Export(0x1747e00, 0xc00050e060, 0x7, 0xc0002aa1a0, 0x2, 0x2, 0x112b040, 0xc00053d200, 0x0, 0x0, ...)
        C:/Users/ZABBIX/build-agents-web/zabbix-5.0.11/src/go/plugins/windows/wmi/wmi.go:44 +0x136
zabbix.com/internal/agent/scheduler.(*exporterTask).perform.func1(0xc00053d200, 0x1127e40, 0xc0002b8000, 0xc0000b6b40, 0xa0)
        C:/Users/ZABBIX/build-agents-web/zabbix-5.0.11/src/go/internal/agent/scheduler/task.go:156 +0x454
created by zabbix.com/internal/agent/scheduler.(*exporterTask).perform
        C:/Users/ZABBIX/build-agents-web/zabbix-5.0.11/src/go/internal/agent/scheduler/task.go:144 +0x6cgoroutine 1 [select, 218 minutes]:
main.run(0xc0002b8ff0, 0x0)
        C:/Users/ZABBIX/build-agents-web/zabbix-5.0.11/src/go/cmd/zabbix_agent2/zabbix_agent2.go:147 +0x215
main.main()
        C:/Users/ZABBIX/build-agents-web/zabbix-5.0.11/src/go/cmd/zabbix_agent2/zabbix_agent2.go:501 +0x1904goroutine 12 [select]:
zabbix.com/internal/agent/scheduler.(*Manager).run(0xc0002b8000)
        C:/Users/ZABBIX/build-agents-web/zabbix-5.0.11/src/go/internal/agent/scheduler/manager.go:317 +0x1c4
created by zabbix.com/internal/agent/scheduler.(*Manager).Start
        C:/Users/ZABBIX/build-agents-web/zabbix-5.0.11/src/go/internal/agent/scheduler/manager.go:457 +0x5fgoroutine 42 [chan receive]:
zabbix.com/internal/agent/resultcache.(*MemoryCache).run(0xc000144240)
        C:/Users/ZABBIX/build-agents-web/zabbix-5.0.11/src/go/internal/agent/resultcache/memorycache.go:202 +0xb2
created by zabbix.com/internal/agent/resultcache.(*MemoryCache).Start
        C:/Users/ZABBIX/build-agents-web/zabbix-5.0.11/src/go/internal/agent/resultcache/memorycache.go:232 +0x5fgoroutine 43 [select]:
zabbix.com/internal/agent/serverconnector.(*Connector).run(0xc0001fc000)
        C:/Users/ZABBIX/build-agents-web/zabbix-5.0.11/src/go/internal/agent/serverconnector/serverconnector.go:300 +0x207
created by zabbix.com/internal/agent/serverconnector.(*Connector).Start
        C:/Users/ZABBIX/build-agents-web/zabbix-5.0.11/src/go/internal/agent/serverconnector/serverconnector.go:358 +0x77goroutine 44 [IO wait, 4 minutes]:
internal/poll.runtime_pollWait(0x2ad44f08, 0x72, 0x1110780)
        c:/go/src/runtime/netpoll.go:203 +0x5c
internal/poll.(*pollDesc).wait(0xc000136e48, 0x72, 0xe26400, 0x0, 0x0)
        c:/go/src/internal/poll/fd_poll_runtime.go:87 +0x4c
internal/poll.(*ioSrv).ExecIO(0x1744130, 0xc000136c98, 0xc0002007e0, 0x1, 0x0, 0xb34)
        c:/go/src/internal/poll/fd_windows.go:228 +0x121
internal/poll.(*FD).acceptOne(0xc000136c80, 0xb34, 0xc0002a00f0, 0x2, 0x2, 0xc000136c98, 0x5, 0x11, 0x5, 0x2)
        c:/go/src/internal/poll/fd_windows.go:896 +0xa9
internal/poll.(*FD).Accept(0xc000136c80, 0xc0004c9e08, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
        c:/go/src/internal/poll/fd_windows.go:930 +0x15f
net.(*netFD).accept(0xc000136c80, 0xad2fda, 0xc0004c9ef0, 0xad3008)
        c:/go/src/net/fd_windows.go:193 +0x7b
net.(*TCPListener).accept(0xc00019a2e0, 0x7f3, 0xc00046cc50, 0x14)
        c:/go/src/net/tcpsock_posix.go:139 +0x39
net.(*TCPListener).Accept(0xc00019a2e0, 0xc000200780, 0xf028c0, 0xc0006762f0, 0xf028c0)
        c:/go/src/net/tcpsock.go:261 +0x6b
zabbix.com/pkg/zbxcomms.(*Listener).Accept(0xc00019a320, 0x2540be400, 0x1, 0x0, 0x1, 0x0)
        C:/Users/ZABBIX/build-agents-web/zabbix-5.0.11/src/go/pkg/zbxcomms/comms.go:307 +0x3e
zabbix.com/internal/agent/serverlistener.(*ServerListener).run(0xc0002b8ff0)
        C:/Users/ZABBIX/build-agents-web/zabbix-5.0.11/src/go/internal/agent/serverlistener/serverlistener.go:71 +0x169
created by zabbix.com/internal/agent/serverlistener.(*ServerListener).Start
        C:/Users/ZABBIX/build-agents-web/zabbix-5.0.11/src/go/internal/agent/serverlistener/serverlistener.go:111 +0x256goroutine 45 [syscall, 218 minutes]:
os/signal.signal_recv(0x0)
        c:/go/src/runtime/sigqueue.go:147 +0xa3
os/signal.loop()
        c:/go/src/os/signal/signal_unix.go:23 +0x29
created by os/signal.Notify.func1
        c:/go/src/os/signal/signal.go:127 +0x4bgoroutine 46 [syscall, 218 minutes, locked to thread]:
syscall.Syscall(0x7ffb2a0348f0, 0x2, 0x568, 0xffffffff, 0x0, 0x0, 0x0, 0x0)
        c:/go/src/runtime/syscall_windows.go:188 +0xe9
syscall.WaitForSingleObject(0x568, 0xc0ffffffff, 0x0, 0x0, 0x435d2d)
        c:/go/src/syscall/zsyscall_windows.go:738 +0x6b
github.com/natefinch/npipe.waitForCompletion(0x558, 0xc0004c40c0, 0x1110780, 0xc0004c0120, 0x0)
        C:/Users/ZABBIX/go/pkg/mod/github.com/natefinch/[email protected]/npipe_windows.go:182 +0x42
github.com/natefinch/npipe.(*PipeListener).AcceptPipe(0xc00007c000, 0x0, 0x0, 0x0)
        C:/Users/ZABBIX/go/pkg/mod/github.com/natefinch/[email protected]/npipe_windows.go:327 +0x2fa
github.com/natefinch/npipe.(*PipeListener).Accept(0xc00007c000, 0x0, 0x0, 0x0, 0x0)
        C:/Users/ZABBIX/go/pkg/mod/github.com/natefinch/[email protected]/npipe_windows.go:263 +0x36
zabbix.com/internal/agent/remotecontrol.(*Conn).run(0xc00019a360)
        C:/Users/ZABBIX/build-agents-web/zabbix-5.0.11/src/go/internal/agent/remotecontrol/remote.go:64 +0x4f
created by zabbix.com/internal/agent/remotecontrol.(*Conn).Start
        C:/Users/ZABBIX/build-agents-web/zabbix-5.0.11/src/go/internal/agent/remotecontrol/remote.go:80 +0x56
rax     0xcf5974a600000000
rbx     0xc0002f0d80
rcx     0x2b212d98
rdi     0x0
rsi     0x0
rbp     0x9
rsp     0x4d67fd70
r8      0x0
r9      0x0
r10     0xc0004c0020
r11     0xffffffffffffffff
r12     0x3
r13     0x2
r14     0x8
r15     0x200
rip     0x7ffb29f094db
rflags  0x10202
cs      0x33
fs      0x53
gs      0x2b
13:21

There is log file gathered during run, before agent fall agent starts complainig about some PDH object

zabbix_agent2.log

There is exact version of powershell script used

hyperv_host.ps1

^There is config file^

^zabbix_agent2.conf^

The windows version is

 

Thanks!

Please let me know if you need additional info.

 

 

 

 

Comment by Karel Bělunek [ 2023 Mar 28 ]

The forementioned proble is no longer valid in Zabbix 6, closing the issue.

Comment by Karel Bělunek [ 2023 Mar 28 ]

The problem is no longer occurred on newer Zabbixes

Generated at Sun May 11 07:36:55 EEST 2025 using Jira 9.12.4#9120004-sha1:625303b708afdb767e17cb2838290c41888e9ff0.