[ZBX-16810] zabbix agent terminates with SIGHUP when started in foreground via screen Created: 2019 Oct 22  Updated: 2019 Nov 07  Resolved: 2019 Nov 06

Status: Closed
Project: ZABBIX BUGS AND ISSUES
Component/s: Agent (G)
Affects Version/s: 4.4.0
Fix Version/s: None

Type: Incident report Priority: Minor
Reporter: Scott McGillivray Assignee: Zabbix Support Team
Resolution: Won't fix Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: File sighup_screen.c    

 Description   

For reasons beyond my control i have to run zabbix agent via screen but it keeps immediately exiting with a SIGHUP and i'm not sure why. 

Version

zabbix_agentd (daemon) (Zabbix) 4.4.0
Revision cfac660b25 7 October 2019, compilation time: Oct 7 2019 07:58:57

 

Here is command used to start

screen -DmS testing -L /usr/sbin/zabbix_agentd -f -c /etc/zabbix/zabbix_agentd.conf

 

Here is log output

cat /var/log/zabbix-agent/zabbix_agentd.log

4437:20191022:223448.514 Starting Zabbix Agent [scott-VirtualBox]. Zabbix 4.4.0 (revision cfac660b25).
4437:20191022:223448.514 Got signal [signal:1(SIGHUP),sender_pid:0,sender_uid:0,reason:128]. Exiting ...
4437:20191022:223448.514 zbx_on_exit() called
4437:20191022:223448.514 In zbx_unload_modules()
4437:20191022:223448.514 End of zbx_unload_modules()
4437:20191022:223448.514 Zabbix Agent stopped. Zabbix 4.4.0 (revision cfac660b25).

As you can see, as soon as it starts it gets SIGHUP and exits. I am confused where the SIGHUP is coming from though ?

If using bash, python or golang i try a similar screen command then i don't get any SIGHUP. Example with below code.. it never receives any SIGHUP so why is zabbix agent ?

 

package main
	
	import (
	"fmt"
	"os"
	"os/signal"
	"syscall"
	)
	
	func main() {
	signal_chan := make(chan os.Signal, 1)
	signal.Notify(signal_chan,
	syscall.SIGHUP,
	syscall.SIGINT,
	syscall.SIGTERM,
	syscall.SIGQUIT)
	
	exit_chan := make(chan int)
	go func() {
	for {
	s := <-signal_chan
	switch s {
	case syscall.SIGHUP:
	fmt.Println("SIGHUP")
	
	case syscall.SIGINT:
	fmt.Println("SIGINT")
	
	case syscall.SIGTERM:
	fmt.Println("SIGTERM ")
	exit_chan <- 0
	
	case syscall.SIGQUIT:
	fmt.Println("SIGQUIT ")
	exit_chan <- 0
	
	default:
	fmt.Println("Unknown signal.")
	exit_chan <- 1
	}
	}
	}()
	
	code := <-exit_chan
	os.Exit(code)
	}
go build signal_test.go
screen -DmS testing -L ./signal_test

Attaching to the session screen -r or cat screenlog.0 has nothing about it getting sent a SIGHUP.

Any ideas why the zabbix agent can not be run in foreground via screen without being immediately killed ?

 Comments   
Comment by richlv [ 2019 Oct 22 ]

Random guess - have you tried -dm instead?

-d -m Start screen in "detached" mode. This creates a new session but doesn't attach to it. This is useful for system startup scripts.
-D -m This also starts screen in "detached" mode, but doesn't fork a new process. The command exits if the session terminates.

Comment by Scott McGillivray [ 2019 Oct 23 ]

yup, tried both but same result.

 

I tried adding strace into the mix like this.. only logs signals.

 

root 6344 0.0 0.1 20412 2872 ? Ss 08:40 0:00 \_ SCREEN -dmS testing -L strace -f -e trace=!all /usr/sbin/zabbix_agentd -f -c /etc/zabbix/zabbix_agentd.conf
root 6345 0.0 0.1 18764 3532 pts/2 Ss+ 08:40 0:00 \_ strace -f -e trace=!all /usr/sbin/zabbix_agentd -f -c /etc/zabbix/zabbix_agentd.conf
root 6347 0.0 0.4 26120 8948 pts/2 S+ 08:40 0:00 \_ /usr/sbin/zabbix_agentd -f -c /etc/zabbix/zabbix_agentd.conf
root 6348 0.0 0.1 26120 3432 pts/2 S+ 08:40 0:00 \_ /usr/sbin/zabbix_agentd: collector [idle 1 sec]
root 6349 0.0 0.2 26252 4648 pts/2 S+ 08:40 0:00 \_ /usr/sbin/zabbix_agentd: listener #1 [waiting for connection]
root 6350 0.0 0.2 26252 4648 pts/2 S+ 08:40 0:00 \_ /usr/sbin/zabbix_agentd: listener #2 [waiting for connection]
root 6351 0.0 0.2 26252 4648 pts/2 S+ 08:40 0:00 \_ /usr/sbin/zabbix_agentd: listener #3 [waiting for connection]
root 6352 0.0 0.2 26252 4644 pts/2 S+ 08:40 0:00 \_ /usr/sbin/zabbix_agentd: active checks #1 [idle 1 sec]

And it works in as much as zabbix doesn't terminate. When i check the strace output i only see this

Starting Zabbix Agent [scott]. Zabbix 4.4.0 (revision cfac660b25).
Press Ctrl+C to exit.

strace: Process 6350 attached
strace: Process 6351 attached
strace: Process 6352 attached
strace: Process 6349 attached
strace: Process 6348 attached
[pid 6347] --- SIGWINCH {si_signo=SIGWINCH, si_code=SI_KERNEL} ---
[pid 6351] --- SIGWINCH {si_signo=SIGWINCH, si_code=SI_KERNEL} ---
[pid 6350] --- SIGWINCH {si_signo=SIGWINCH, si_code=SI_KERNEL} ---
[pid 6349] --- SIGWINCH {si_signo=SIGWINCH, si_code=SI_KERNEL} ---
[pid 6348] --- SIGWINCH {si_signo=SIGWINCH, si_code=SI_KERNEL} ---
[pid 6352] --- SIGWINCH {si_signo=SIGWINCH, si_code=SI_KERNEL} ---

But i see no SIGHUP being sent to Zabbix... is it possible that zabbix agent is misinterpreting SIGWINCH as HUP ?

Comment by Scott McGillivray [ 2019 Oct 23 ]

poking around the code, isn't this line supposed to ignore SIGHUP when in foreground mode ?

https://github.com/zabbix/zabbix/blob/2badd19445bf53d123c5a8b7e6a982bb62da0ad4/src/libs/zbxnix/daemon.c#L360

Comment by Vladislavs Sokurenko [ 2019 Nov 04 ]

poking around the code, isn't this line supposed to ignore SIGHUP when in foreground mode ?

https://github.com/zabbix/zabbix/blob/2badd19445bf53d123c5a8b7e6a982bb62da0ad4/src/libs/zbxnix/daemon.c#L360

No, this branch is only active when running as a daemon.

Comment by Vladislavs Sokurenko [ 2019 Nov 04 ]

Closing as a Won't Fix since it works as intended, in this case it is simply required to run as a daemon.

Comment by richlv [ 2019 Nov 04 ]

Curious - why is it required to be run as a daemon process?

Comment by Vladislavs Sokurenko [ 2019 Nov 04 ]

You could also try nohup command:

screen -DmS zabbix -L nohup ./sbin/zabbix_agentd -f -c /etc/zabbix/zabbix_agentd.conf

Daemon is required in this case or nohup command because it does not require terminal so when terminal is closed it will ignore.

Comment by richlv [ 2019 Nov 04 ]

Shouldn't screen count as a terminal?

Comment by Vladislavs Sokurenko [ 2019 Nov 04 ]

Yes, I agree richlv, but shouldn’t it be transparent to the application ?

Comment by richlv [ 2019 Nov 04 ]

Likely - but also see the provided strace that doesn't seem to include SIGHUP.

Comment by Vladislavs Sokurenko [ 2019 Nov 05 ]

Attached C program sighup_screen.c that reproduces the issue

Comment by Scott McGillivray [ 2019 Nov 06 ]

hi, if there is any extra information i can provide or testing i can do to help on this issue please let me know. 

 

Comment by Edgar Akhmetshin [ 2019 Nov 06 ]

Hello Scott,

As already said by vso: "Daemon is required in this case or nohup command because it does not require terminal so when terminal is closed it will ignore".

Suggested command to start agent works:

screen -DmS zabbix -L nohup ./sbin/zabbix_agentd -f -c /etc/zabbix/zabbix_agentd.conf
...
100      16425  0.0  0.1   5256  1396 ?        S    Oct31   0:00 /usr/sbin/zabbix_agentd --foreground -c /etc/zabbix/zabbix_agentd.conf
100      16482  0.0  0.0   5256   528 ?        S    Oct31   1:53 /usr/sbin/zabbix_agentd: collector [idle 1 sec]
100      16483  0.0  0.1   5528  1048 ?        S    Oct31   0:00 /usr/sbin/zabbix_agentd: listener #1 [waiting for connection]
100      16484  0.0  0.1   5528  1048 ?        S    Oct31   0:00 /usr/sbin/zabbix_agentd: listener #2 [waiting for connection]
100      16485  0.0  0.1   5528  1048 ?        S    Oct31   0:00 /usr/sbin/zabbix_agentd: listener #3 [waiting for connection]
100      16486  0.0  0.1   5528  1288 ?        S    Oct31   0:45 /usr/sbin/zabbix_agentd: active checks #1 [idle 1 sec]

Please be advised that this section of the tracker is for bug reports only. Closing as Won't fix. Thank you for understanding.

Regards,
Edgar

Generated at Sat Jun 28 06:57:06 EEST 2025 using Jira 9.12.4#9120004-sha1:625303b708afdb767e17cb2838290c41888e9ff0.