[ZBX-20371] Crash agent2 on CentOS 7 when log monitoring in 6.0.0beta1 Created: 2021 Dec 17  Updated: 2024 Apr 10  Resolved: 2022 Jan 25

Status: Closed
Project: ZABBIX BUGS AND ISSUES
Component/s: Agent (G), Agent2 plugin (G)
Affects Version/s: 6.0.0beta1
Fix Version/s: 6.0.0beta3, 6.0 (plan)

Type: Problem report Priority: Critical
Reporter: Yuuki Enomoto Assignee: Andris Mednis
Resolution: Fixed Votes: 0
Labels: agent, crash
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

CentOS Linux release 7.9.2009 (Core)
gcc (GCC) 4.8.5 20150623 (Red Hat 4.8.5-44)
go version go1.17.5 linux/amd64

Issue Links:
Causes
caused by ZBXNEXT-6593 Agent re-read log files Closed
Duplicate
is duplicated by ZBX-20414 Agent 2 (6.0.0beta1) not stable Closed
is duplicated by ZBX-20372 Agent2 log monitoring crashes Closed
Team: Team C
Sprint: Sprint 84 (Jan 2022)
Story Points: 1

 Description   

Steps to reproduce:

  1. Build 6.0.0beta1 agent2 in CentOS 7, then install.
    1. ./configure --enable-agent2
    2. make install
  2. Configure, then run agent2.
    1. /usr/local/sbin/zabbix_agent2 -c /usr/local/etc/zabbix_agent2.conf
  3. Setup log monitoring on Web frontend. For example, create active item log[/tmp/zabbix_agent2.log].

Result:

Print the following error into console. Crash Agent2.

fatal error: unexpected signal during runtime execution
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0xb470ff]runtime stack:
runtime.throw({0xcd27c5, 0x17})
        /root/go/src/runtime/panic.go:1198 +0x71
runtime.sigpanic()
        /root/go/src/runtime/signal_unix.go:719 +0x396goroutine 13 [syscall]:
runtime.cgocall(0xa72d60, 0xc0003f9ad8)
        /root/go/src/runtime/cgocall.go:156 +0x5c fp=0xc0003f9ab0 sp=0xc0003f9a78 pc=0x429ebc
zabbix.com/pkg/zbxlib._Cfunc_process_log_check(0x7f48d8000980, 0x7f48e40008c0, 0x7f48d80008f0, 0xa72a70, 0xc0003a4018, 0xc0003a4028, 0xc000010010, 0x7f48d80009c0)
        _cgo_gotypes.go:653 +0x4c fp=0xc0003f9ad8 sp=0xc0003f9ab0 pc=0x58f4ac
zabbix.com/pkg/zbxlib.ProcessLogCheck.func3(0x7f48d80009c0, 0x7f48d8000980, 0xba7c02, 0x25, 0xc0003f9c40, 0x430b7d, 0xc0001cb6c0)
        /root/zabbix-6.0.0beta1/src/go/pkg/zbxlib/logfile.go:298 +0x125 fp=0xc0003f9b70 sp=0xc0003f9ad8 pc=0x591fa5
zabbix.com/pkg/zbxlib.ProcessLogCheck(0xc0003a400c, 0xc0003f9da8, 0xc0002dd440, 0xcf4a40)
        /root/zabbix-6.0.0beta1/src/go/pkg/zbxlib/logfile.go:298 +0x186 fp=0xc0003f9cc8 sp=0xc0003f9b70 pc=0x591266
zabbix.com/plugins/log.(*Plugin).Export(0x14707c0, {0xc0003a400c, 0x14707c0}, {0xc000068040, 0x48852e, 0x45cfe7}, {0xdb31f8, 0xc00022e0c0})
        /root/zabbix-6.0.0beta1/src/go/plugins/log/log.go:109 +0x50d fp=0xc0003f9df8 sp=0xc0003f9cc8 pc=0x74fbcd
zabbix.com/internal/agent/scheduler.(*exporterTask).perform.func1({0xc0000380e0, 0x1b})
        /root/zabbix-6.0.0beta1/src/go/internal/agent/scheduler/task.go:168 +0x1cb fp=0xc0003f9fc0 sp=0xc0003f9df8 pc=0x5a982b
zabbix.com/internal/agent/scheduler.(*exporterTask).perform·dwrap·2()
        /root/zabbix-6.0.0beta1/src/go/internal/agent/scheduler/task.go:204 +0x31 fp=0xc0003f9fe0 sp=0xc0003f9fc0 pc=0x5a9631
runtime.goexit()
        /root/go/src/runtime/asm_amd64.s:1581 +0x1 fp=0xc0003f9fe8 sp=0xc0003f9fe0 pc=0x48a701
created by zabbix.com/internal/agent/scheduler.(*exporterTask).perform
        /root/zabbix-6.0.0beta1/src/go/internal/agent/scheduler/task.go:156 +0x108goroutine 1 [select, 2 minutes]:
main.run()
        /root/zabbix-6.0.0beta1/src/go/cmd/zabbix_agent2/zabbix_agent2.go:178 +0x185
main.main()
        /root/zabbix-6.0.0beta1/src/go/cmd/zabbix_agent2/zabbix_agent2.go:566 +0x1853goroutine 19 [select]:
zabbix.com/internal/agent/scheduler.(*Manager).run(0xc0002af040)
        /root/zabbix-6.0.0beta1/src/go/internal/agent/scheduler/manager.go:352 +0x1cf
created by zabbix.com/internal/agent/scheduler.(*Manager).Start
        /root/zabbix-6.0.0beta1/src/go/internal/agent/scheduler/manager.go:540 +0x6fgoroutine 20 [chan receive]:
zabbix.com/internal/agent/resultcache.(*MemoryCache).run(0xc0002dd440)
        /root/zabbix-6.0.0beta1/src/go/internal/agent/resultcache/memorycache.go:208 +0x85
created by zabbix.com/internal/agent/resultcache.(*MemoryCache).Start
        /root/zabbix-6.0.0beta1/src/go/internal/agent/resultcache/memorycache.go:238 +0x6fgoroutine 21 [select]:
zabbix.com/internal/agent/serverconnector.(*Connector).run(0xc0002b6f30)
        /root/zabbix-6.0.0beta1/src/go/internal/agent/serverconnector/serverconnector.go:310 +0x1d7
created by zabbix.com/internal/agent/serverconnector.(*Connector).Start
        /root/zabbix-6.0.0beta1/src/go/internal/agent/serverconnector/serverconnector.go:372 +0x87goroutine 22 [IO wait, 2 minutes]:
internal/poll.runtime_pollWait(0x7f491e68e6e8, 0x72)
        /root/go/src/runtime/netpoll.go:234 +0x89
internal/poll.(*pollDesc).wait(0xc0002d8400, 0xc0003a4120, 0x0)
        /root/go/src/internal/poll/fd_poll_runtime.go:84 +0x32
internal/poll.(*pollDesc).waitRead(...)
        /root/go/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Accept(0xc0002d8400)
        /root/go/src/internal/poll/fd_unix.go:402 +0x22c
net.(*netFD).accept(0xc0002d8400)
        /root/go/src/net/fd_unix.go:173 +0x35
net.(*TCPListener).accept(0xc000277b60)
        /root/go/src/net/tcpsock_posix.go:140 +0x28
net.(*TCPListener).Accept(0xc000277b60)
        /root/go/src/net/tcpsock.go:262 +0x3d
zabbix.com/pkg/zbxcomms.(*Listener).Accept(0xc000277b78, 0xb2d05e00, 0x1)
        /root/zabbix-6.0.0beta1/src/go/pkg/zbxcomms/comms.go:308 +0x37
zabbix.com/internal/agent/serverlistener.(*ServerListener).run(0xc000304230)
        /root/zabbix-6.0.0beta1/src/go/internal/agent/serverlistener/serverlistener.go:72 +0x167
created by zabbix.com/internal/agent/serverlistener.(*ServerListener).Start
        /root/zabbix-6.0.0beta1/src/go/internal/agent/serverlistener/serverlistener.go:112 +0x1c7goroutine 34 [syscall, 2 minutes]:
os/signal.signal_recv()
        /root/go/src/runtime/sigqueue.go:169 +0x98
os/signal.loop()
        /root/go/src/os/signal/signal_unix.go:24 +0x19
created by os/signal.Notify.func1.1
        /root/go/src/os/signal/signal.go:151 +0x2cgoroutine 5 [IO wait, 2 minutes]:
internal/poll.runtime_pollWait(0x7f491e68e600, 0x72)
        /root/go/src/runtime/netpoll.go:234 +0x89
internal/poll.(*pollDesc).wait(0xc00010e000, 0x0, 0x0)
        /root/go/src/internal/poll/fd_poll_runtime.go:84 +0x32
internal/poll.(*pollDesc).waitRead(...)
        /root/go/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Accept(0xc00010e000)
        /root/go/src/internal/poll/fd_unix.go:402 +0x22c
net.(*netFD).accept(0xc00010e000)
        /root/go/src/net/fd_unix.go:173 +0x35
net.(*UnixListener).accept(0x0)
        /root/go/src/net/unixsock_posix.go:167 +0x1c
net.(*UnixListener).Accept(0xc000032060)
        /root/go/src/net/unixsock.go:260 +0x3d
zabbix.com/internal/agent/remotecontrol.(*Conn).run(0xc00000e030)
        /root/zabbix-6.0.0beta1/src/go/internal/agent/remotecontrol/remote.go:64 +0x48
created by zabbix.com/internal/agent/remotecontrol.(*Conn).Start
        /root/zabbix-6.0.0beta1/src/go/internal/agent/remotecontrol/remote.go:80 +0x65


 Comments   
Comment by Andris Mednis [ 2021 Dec 30 ]

The C function process_log_check() expects the 1st argument 'addrs' to be a vector with element(s) carrying IP address (or hostname) and port number.

When called from Agent2 the 'addrs' vector carries elements of quite different type - 'log_result_t' - and the vector could be with 0 elements. This causes crash when init_persistent_dir_parameter() is called (it assumes at least 1 element in the vector and tries to unpack IP address and port number from it).

process_log_check() has no reliable method to determine is it called from C or from Golang agent. Additional parameter is required.

Comment by Andris Mednis [ 2022 Jan 04 ]

RESOLVED in development branch  feature/ZBX-20371-5.5-2 Pull Request

Comment by Tomáš Heřmánek [ 2022 Jan 11 ]

Hi Zabbix Team,

same problem in Debian 10 i test beta1 and beta2, both version have problems. Last working version is aplha7.

Version:

root@MACHINE:/etc/zabbix# dpkg -l | grep  zabbix-agent
ii  zabbix-agent2                         1:6.0.0~beta2-1+debian10                                                  amd64        

 Test:

root@MACHINE:/etc/zabbix# zabbix_agent2

 

Starting Zabbix Agent 2 (6.0.0beta2)
Zabbix Agent2 hostname: [XXXXXXXXXXX]
Press Ctrl+C to exit.
fatal error: unexpected signal during runtime execution
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0xba2b31]
 
runtime stack:
runtime.throw({0xd4a3ab, 0x7feedfffe844})
/home/packager/go1.17/src/runtime/panic.go:1198 +0x71
runtime.sigpanic()
/home/packager/go1.17/src/runtime/signal_unix.go:719 +0x396
 
goroutine 3 [syscall]:
runtime.cgocall(0xa7f238, 0xc00006dad8)
/home/packager/go1.17/src/runtime/cgocall.go:156 +0x5c fp=0xc00006dab0 sp=0xc00006da78 pc=0x42477c
zabbix.com/pkg/zbxlib._Cfunc_process_log_check(0x7feed8000d40, 0x7feed0000c60, 0x7feed8000ca0, 0xa7ee88, 0xc0000360b0, 0xc0000360c0, 0xc000010018, 0x7feed8000d80)
_cgo_gotypes.go:653 +0x4c fp=0xc00006dad8 sp=0xc00006dab0 pc=0x58f1ec
zabbix.com/pkg/zbxlib.ProcessLogCheck.func3(0x7feed8000d80, 0x7feed8000d40, 0xc1bf22, 0x25, 0xc00006dc40, 0x42b43d, 0xc000001380)
/tmp/build-debian-10-x86_64.o4QvMhjQ/buildroot/zabbix-6.0.0beta2/debian/tmp.build-sqlite3/src/go/pkg/zbxlib/logfile.go:298 +0x125 fp=0xc00006db70 sp=0xc00006dad8 pc=0x591c85
zabbix.com/pkg/zbxlib.ProcessLogCheck(0xc0000360a0, 0xc00006dda8, 0xc0002b6690, 0xd6c920)
/tmp/build-debian-10-x86_64.o4QvMhjQ/buildroot/zabbix-6.0.0beta2/debian/tmp.build-sqlite3/src/go/pkg/zbxlib/logfile.go:298 +0x186 fp=0xc00006dcc8 sp=0xc00006db70 pc=0x590f46
zabbix.com/plugins/log.(*Plugin).Export(0x130fa80, {0xc0000360a0, 0x130fa80}, {0xc000032020, 0x0, 0x0}, {0xe2bc18, 0xc000432000})
/tmp/build-debian-10-x86_64.o4QvMhjQ/buildroot/zabbix-6.0.0beta2/debian/tmp.build-sqlite3/src/go/plugins/log/log.go:109 +0x50d fp=0xc00006ddf8 sp=0xc00006dcc8 pc=0x754ced
zabbix.com/internal/agent/scheduler.(*exporterTask).perform.func1({0xc000430000, 0x28})
/tmp/build-debian-10-x86_64.o4QvMhjQ/buildroot/zabbix-6.0.0beta2/debian/tmp.build-sqlite3/src/go/internal/agent/scheduler/task.go:168 +0x1cb fp=0xc00006dfc0 sp=0xc00006ddf8 pc=0x5c4deb
zabbix.com/internal/agent/scheduler.(*exporterTask).perform·dwrap·2()
/tmp/build-debian-10-x86_64.o4QvMhjQ/buildroot/zabbix-6.0.0beta2/debian/tmp.build-sqlite3/src/go/internal/agent/scheduler/task.go:204 +0x31 fp=0xc00006dfe0 sp=0xc00006dfc0 pc=0x5c4bf1
runtime.goexit()
/home/packager/go1.17/src/runtime/asm_amd64.s:1581 +0x1 fp=0xc00006dfe8 sp=0xc00006dfe0 pc=0x4851a1
created by zabbix.com/internal/agent/scheduler.(*exporterTask).perform
/tmp/build-debian-10-x86_64.o4QvMhjQ/buildroot/zabbix-6.0.0beta2/debian/tmp.build-sqlite3/src/go/internal/agent/scheduler/task.go:156 +0x108
 
goroutine 1 [select]:
main.run()
/tmp/build-debian-10-x86_64.o4QvMhjQ/buildroot/zabbix-6.0.0beta2/debian/tmp.build-sqlite3/src/go/cmd/zabbix_agent2/zabbix_agent2.go:175 +0x1ab
main.main()
/tmp/build-debian-10-x86_64.o4QvMhjQ/buildroot/zabbix-6.0.0beta2/debian/tmp.build-sqlite3/src/go/cmd/zabbix_agent2/zabbix_agent2.go:572 +0x1873
 
goroutine 21 [select]:
zabbix.com/internal/agent/scheduler.(*Manager).run(0xc00024b450)
/tmp/build-debian-10-x86_64.o4QvMhjQ/buildroot/zabbix-6.0.0beta2/debian/tmp.build-sqlite3/src/go/internal/agent/scheduler/manager.go:353 +0x1cf
created by zabbix.com/internal/agent/scheduler.(*Manager).Start
/tmp/build-debian-10-x86_64.o4QvMhjQ/buildroot/zabbix-6.0.0beta2/debian/tmp.build-sqlite3/src/go/internal/agent/scheduler/manager.go:549 +0x6f
 
goroutine 22 [chan receive]:
zabbix.com/internal/agent/resultcache.(*MemoryCache).run(0xc0002b6690)
/tmp/build-debian-10-x86_64.o4QvMhjQ/buildroot/zabbix-6.0.0beta2/debian/tmp.build-sqlite3/src/go/internal/agent/resultcache/memorycache.go:208 +0x85
created by zabbix.com/internal/agent/resultcache.(*MemoryCache).Start
/tmp/build-debian-10-x86_64.o4QvMhjQ/buildroot/zabbix-6.0.0beta2/debian/tmp.build-sqlite3/src/go/internal/agent/resultcache/memorycache.go:238 +0x6f
 
goroutine 23 [select]:
zabbix.com/internal/agent/serverconnector.(*Connector).run(0xc000253560)
/tmp/build-debian-10-x86_64.o4QvMhjQ/buildroot/zabbix-6.0.0beta2/debian/tmp.build-sqlite3/src/go/internal/agent/serverconnector/serverconnector.go:310 +0x1d7
created by zabbix.com/internal/agent/serverconnector.(*Connector).Start
/tmp/build-debian-10-x86_64.o4QvMhjQ/buildroot/zabbix-6.0.0beta2/debian/tmp.build-sqlite3/src/go/internal/agent/serverconnector/serverconnector.go:372 +0x87
 
goroutine 24 [IO wait]:
internal/poll.runtime_pollWait(0x7feee5a1a6f8, 0x72)
/home/packager/go1.17/src/runtime/netpoll.go:229 +0x89
internal/poll.(*pollDesc).wait(0xc000276a00, 0x0, 0x0)
/home/packager/go1.17/src/internal/poll/fd_poll_runtime.go:84 +0x32
internal/poll.(*pollDesc).waitRead(...)
/home/packager/go1.17/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Accept(0xc000276a00)
/home/packager/go1.17/src/internal/poll/fd_unix.go:402 +0x22c
net.(*netFD).accept(0xc000276a00)
/home/packager/go1.17/src/net/fd_unix.go:173 +0x35
net.(*TCPListener).accept(0xc00025cf18)
/home/packager/go1.17/src/net/tcpsock_posix.go:140 +0x28
net.(*TCPListener).Accept(0xc00025cf18)
/home/packager/go1.17/src/net/tcpsock.go:262 +0x3d
zabbix.com/pkg/zbxcomms.(*Listener).Accept(0xc00025cf30, 0xb2d05e00, 0x1)
/tmp/build-debian-10-x86_64.o4QvMhjQ/buildroot/zabbix-6.0.0beta2/debian/tmp.build-sqlite3/src/go/pkg/zbxcomms/comms.go:308 +0x37
zabbix.com/internal/agent/serverlistener.(*ServerListener).run(0xc0002b46e0)
/tmp/build-debian-10-x86_64.o4QvMhjQ/buildroot/zabbix-6.0.0beta2/debian/tmp.build-sqlite3/src/go/internal/agent/serverlistener/serverlistener.go:72 +0x167
created by zabbix.com/internal/agent/serverlistener.(*ServerListener).Start
/tmp/build-debian-10-x86_64.o4QvMhjQ/buildroot/zabbix-6.0.0beta2/debian/tmp.build-sqlite3/src/go/internal/agent/serverlistener/serverlistener.go:112 +0x1c7
 
goroutine 26 [syscall]:
os/signal.signal_recv()
/home/packager/go1.17/src/runtime/sigqueue.go:169 +0x98
os/signal.loop()
/home/packager/go1.17/src/os/signal/signal_unix.go:24 +0x19
created by os/signal.Notify.func1.1
/home/packager/go1.17/src/os/signal/signal.go:151 +0x2c
 
goroutine 27 [IO wait]:
internal/poll.runtime_pollWait(0x7feee5a1a610, 0x72)
/home/packager/go1.17/src/runtime/netpoll.go:229 +0x89
internal/poll.(*pollDesc).wait(0xc000276c00, 0x0, 0x0)
/home/packager/go1.17/src/internal/poll/fd_poll_runtime.go:84 +0x32
internal/poll.(*pollDesc).waitRead(...)
/home/packager/go1.17/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Accept(0xc000276c00)
/home/packager/go1.17/src/internal/poll/fd_unix.go:402 +0x22c
net.(*netFD).accept(0xc000276c00)
/home/packager/go1.17/src/net/fd_unix.go:173 +0x35
net.(*UnixListener).accept(0x0)
/home/packager/go1.17/src/net/unixsock_posix.go:167 +0x1c
net.(*UnixListener).Accept(0xc0002b67b0)
/home/packager/go1.17/src/net/unixsock.go:260 +0x3d
zabbix.com/internal/agent/remotecontrol.(*Conn).run(0xc00025cf48)
/tmp/build-debian-10-x86_64.o4QvMhjQ/buildroot/zabbix-6.0.0beta2/debian/tmp.build-sqlite3/src/go/internal/agent/remotecontrol/remote.go:64 +0x48
created by zabbix.com/internal/agent/remotecontrol.(*Conn).Start
/tmp/build-debian-10-x86_64.o4QvMhjQ/buildroot/zabbix-6.0.0beta2/debian/tmp.build-sqlite3/src/go/internal/agent/remotecontrol/remote.go:80 +0x65

 

Comment by Andris Mednis [ 2022 Jan 11 ]

Yes, the fix is available in development branch "feature/ZBX-20371-5.5-2" (link to source code) but it was not included in beta2.

Comment by Tomáš Heřmánek [ 2022 Jan 11 ]

OK Cool, thanks. 

Comment by Andris Mednis [ 2022 Jan 12 ]

Available in versions:

Generated at Thu Apr 03 02:21:00 EEST 2025 using Jira 9.12.4#9120004-sha1:625303b708afdb767e17cb2838290c41888e9ff0.