Red Hat 7, CentOS 7 (all systemd) 2.4.3, 2.4.4

On my environment, /var/run is symlink to /run. systemctl status zabbix-agent doesn't show such errors.

It turns out mine is a symlink too but I am still having the same problem. I don't know if it's related, but I was about to report a second, similar issue where RHEL 6 / CentOS 6 says the agent isn't running when it is.

[root@dns1 ~]# ps -eaf |grep zabbix
zabbix   20970     1  0 21:35 ?        00:00:00 zabbix_agentd -c /etc/zabbix/zabbix_agentd.conf
zabbix   20972 20970  0 21:35 ?        00:00:00 zabbix_agentd: collector [idle 1 sec]
zabbix   20973 20970  0 21:35 ?        00:00:00 zabbix_agentd: listener #1 [waiting for connection]
zabbix   20974 20970  0 21:35 ?        00:00:00 zabbix_agentd: listener #2 [waiting for connection]
zabbix   20975 20970  0 21:35 ?        00:00:00 zabbix_agentd: listener #3 [waiting for connection]
zabbix   20976 20970  0 21:35 ?        00:00:00 zabbix_agentd: active checks #1 [idle 1 sec]
root     21355 20464  0 21:37 pts/0    00:00:00 grep zabbix
[root@dns1 ~]# service zabbix-agent status
zabbix_agentd is stopped
[root@dns1 ~]# ls -lha /var/run/zabbix/
total 12K
drwxr-xr-x.  2 zabbix zabbix 4.0K Mar  4 21:35 .
drwxr-xr-x. 14 root   root   4.0K Mar  4 04:26 ..
-rw-rw-r--.  1 zabbix zabbix    5 Mar  4 21:35 zabbix_agentd.pid

Is there something I can do to help troubleshoot this?

This is from a CenOS 7 host just now:

[root@cent7test ~]# systemctl restart zabbix-agent

Job for zabbix-agent.service failed. See 'systemctl status zabbix-agent.service' and 'journalctl -xn' for details.
[root@cent7test ~]#
[root@cent7test ~]# systemctl status zabbix-agent
zabbix-agent.service - Zabbix Agent
   Loaded: loaded (/usr/lib/systemd/system/zabbix-agent.service; enabled)
   Active: activating (start) since Wed 2015-03-04 21:46:04 EST; 33s ago
  Process: 9363 ExecStart=/usr/sbin/zabbix_agentd -c $CONFFILE (code=exited, status=0/SUCCESS)
 Main PID: 26534 (code=exited, status=1/FAILURE)
   CGroup: /system.slice/zabbix-agent.service
           ├─26244 /usr/sbin/zabbix_agentd -c /etc/zabbix/zabbix_agentd.conf
           ├─26245 /usr/sbin/zabbix_agentd: collector [idle 1 sec]
           ├─26246 /usr/sbin/zabbix_agentd: listener #1 [waiting for connection]
           ├─26247 /usr/sbin/zabbix_agentd: listener #2 [waiting for connection]
           ├─26248 /usr/sbin/zabbix_agentd: listener #3 [waiting for connection]
           └─26249 /usr/sbin/zabbix_agentd: active checks #1 [idle 1 sec]

Mar 04 21:46:04 cent7test systemd[1]: zabbix-agent.service holdoff time over, scheduling restart.
Mar 04 21:46:04 cent7test systemd[1]: Stopping Zabbix Agent...
Mar 04 21:46:04 cent7test systemd[1]: Starting Zabbix Agent...
Mar 04 21:46:04 cent7test systemd[1]: PID file /run/zabbix/zabbix_agentd.pid not readable (yet?) after start.
[root@cent7test ~]# journalctl -xn
-- Logs begin at Mon 2015-02-23 10:21:10 EST, end at Wed 2015-03-04 21:47:34 EST. --
Mar 04 21:46:04 cent7test systemd[1]: Starting Zabbix Agent...
-- Subject: Unit zabbix-agent.service has begun with start-up
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit zabbix-agent.service has begun starting up.
Mar 04 21:46:04 cent7test systemd[1]: PID file /run/zabbix/zabbix_agentd.pid not readable (yet?) after start.
Mar 04 21:46:55 cent7test sshd[9373]: Connection closed by 127.0.0.1 [preauth]
Mar 04 21:47:34 cent7test systemd[1]: zabbix-agent.service operation timed out. Terminating.
Mar 04 21:47:34 cent7test systemd[1]: Failed to start Zabbix Agent.
-- Subject: Unit zabbix-agent.service has failed
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit zabbix-agent.service has failed.
--
-- The result is failed.
Mar 04 21:47:34 cent7test systemd[1]: Unit zabbix-agent.service entered failed state.
Mar 04 21:47:34 cent7test systemd[1]: zabbix-agent.service holdoff time over, scheduling restart.
Mar 04 21:47:34 cent7test systemd[1]: Stopping Zabbix Agent...
-- Subject: Unit zabbix-agent.service has begun shutting down
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit zabbix-agent.service has begun shutting down.
Mar 04 21:47:34 cent7test systemd[1]: Starting Zabbix Agent...
-- Subject: Unit zabbix-agent.service has begun with start-up
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit zabbix-agent.service has begun starting up.
Mar 04 21:47:34 cent7test systemd[1]: PID file /run/zabbix/zabbix_agentd.pid not readable (yet?) after start.

Thank you for the info.

It seems that pid file is exists, but zabbix user cannot access the file. Are you using SELinux? If so, it's necessary to add rule for Zabbix.

SELinux is not enforcing.

Hello,
I have the same error. I am using Centos7. I disabled SELinux. Zabbix-agent still not working.
How did you solve this problem please ?

I have not solved this. I'm still hoping for an answer here.

Dear all.

Problem is still up to date.
Here you go some logs:

• zabbix-agent installation source

[root@ansible ~]# yum info -v zabbix-agent
Failed to set locale, defaulting to C
Loading "fastestmirror" plugin
Config time: 0.009
Yum version: 3.4.3
rpmdb time: 0.000
Setting up Package Sacks
Loading mirror speeds from cached hostfile
 * base: centos.hitme.net.pl
 * epel: ftp.pbone.net
 * extras: centos.hitme.net.pl
 * updates: centos1.hti.pl
pkgsack time: 6.468
Installed Packages
Name        : zabbix-agent
Arch        : x86_64
Version     : 2.2.9
Release     : 1.el7
Size        : 525 k
Repo        : installed
From repo   : zabbix
Committer   : Kodai Terashima <[email protected]>
Committime  : Sun Mar 15 13:00:00 2015
Buildtime   : Sun Mar 15 09:23:26 2015
Install time: Sun Mar 29 19:02:38 2015
Installed by: root <root>
Changed by  : root <root>
Summary     : Zabbix Agent
URL         : http://www.zabbix.com/
Licence     : GPLv2+
Description : The Zabbix client agent, to be installed on monitored systems.

[root@ansible ~]# cat /etc/yum
yum/         yum.conf     yum.repos.d/ 
[root@ansible ~]# cat /etc/yum.repos.d/zabbix.repo 
[zabbix]
name=Zabbix Official Repository - $basearch
baseurl=http://repo.zabbix.com/zabbix/2.2/rhel/7/$basearch/
enabled=1
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-ZABBIX
[root@ansible ~]#

• SELinux status:

[root@ansible ~]# sestatus 
SELinux status:                 enabled
SELinuxfs mount:                /sys/fs/selinux
SELinux root directory:         /etc/selinux
Loaded policy name:             targeted
Current mode:                   permissive
Mode from config file:          enforcing
Policy MLS status:              enabled
Policy deny_unknown status:     allowed
Max kernel policy version:      28
[root@ansible ~]#

• service starting:

[root@ansible ~]# systemctl start zabbix-agent
Job for zabbix-agent.service failed. See 'systemctl status zabbix-agent.service' and 'journalctl -xn' for details.
[root@ansible ~]# systemctl status zabbix-agent
zabbix-agent.service - Zabbix Agent
   Loaded: loaded (/usr/lib/systemd/system/zabbix-agent.service; disabled)
   Active: failed (Result: start-limit) since Sun 2015-03-29 19:19:23 CEST; 4s ago
  Process: 9457 ExecStart=/usr/sbin/zabbix_agentd -c $CONFFILE (code=exited, status=0/SUCCESS)

Mar 29 19:19:23 ansible.management.bitsa.organisation systemd[1]: zabbix-agent.service never wrote its PID file. Failing.
Mar 29 19:19:23 ansible.management.bitsa.organisation systemd[1]: Failed to start Zabbix Agent.
Mar 29 19:19:23 ansible.management.bitsa.organisation systemd[1]: Unit zabbix-agent.service entered failed state.
Mar 29 19:19:23 ansible.management.bitsa.organisation systemd[1]: zabbix-agent.service holdoff time over, scheduling restart.
Mar 29 19:19:23 ansible.management.bitsa.organisation systemd[1]: Stopping Zabbix Agent...
Mar 29 19:19:23 ansible.management.bitsa.organisation systemd[1]: Starting Zabbix Agent...
Mar 29 19:19:23 ansible.management.bitsa.organisation systemd[1]: zabbix-agent.service start request repeated too quickly, refusing to start.
Mar 29 19:19:23 ansible.management.bitsa.organisation systemd[1]: Failed to start Zabbix Agent.
Mar 29 19:19:23 ansible.management.bitsa.organisation systemd[1]: Unit zabbix-agent.service entered failed state.
[root@ansible ~]# 

• manual service running:

[root@ansible ~]# zabbix_agent -c /etc/zabbix/zabbix_agentd.conf 
zabbix_agent [9489]: unknown parameter [PidFile] in config file [/etc/zabbix/zabbix_agentd.conf], line 13
[root@ansible ~]# 

• 13th line number in configuration file:

[root@ansible ~]# grep -A10 -B10 Pid /etc/zabbix/zabbix_agentd.conf 
# This is a config file for the Zabbix agent daemon (Unix)
# To get more information about Zabbix, visit http://www.zabbix.com

############ GENERAL PARAMETERS #################

### Option: PidFile
#	Name of PID file.
#
# Mandatory: no
# Default:
# PidFile=/tmp/zabbix_agentd.pid
#
PidFile=/var/run/zabbix/zabbix_agentd.pid

### Option: LogFile
#	Name of log file.
#	If not set, syslog is used.
#
# Mandatory: no
# Default:
# LogFile=
LogFile=/var/log/zabbix/zabbix_agentd.log

[root@ansible ~]# 

• zabix-agent version:

[root@ansible ~]# zabbix_agent -V
Zabbix agent v2.2.9 (revision 52686) (12 March 2015)
Compilation time: Mar 15 2015 08:21:46
[root@ansible ~]# 

Have you got any suggestion?

Workaround:

Commenting out PidFile in /etc/zabbix/zabbix_agentd.conf:

  5 
  6 ### Option: PidFile
  7 #       Name of PID file.
  8 #
  9 # Mandatory: no
 10 # Default:
 11 # PidFile=/tmp/zabbix_agentd.pid
 12 #
 13 #PidFile=/var/run/zabbix/zabbix_agentd.pid

And PIDFile in zabbix_agentd.service allows to run zabbix_agentd service properly.

[Unit]
Description=Zabbix Agent
After=syslog.target
After=network.target

[Service]
Environment="CONFFILE=/etc/zabbix/zabbix_agentd.conf"
EnvironmentFile=-/etc/sysconfig/zabbix-agent
Type=forking
Restart=always
#PIDFile=/run/zabbix/zabbix_agentd.pid
KillMode=process
ExecStart=/usr/sbin/zabbix_agentd -c $CONFFILE

[Install]
WantedBy=multi-user.target

Workflow:

[root@ansible run]# systemctl status zabbix-agent
zabbix-agent.service - Zabbix Agent
   Loaded: loaded (/usr/lib/systemd/system/zabbix-agent.service; disabled)
   Active: inactive (dead)

Mar 30 09:01:52 ansible.management.bitsa.organisation systemd[1]: zabbix-agent.service holdoff time over, scheduling restart.
Mar 30 09:01:52 ansible.management.bitsa.organisation systemd[1]: Stopping Zabbix Agent...
Mar 30 09:01:52 ansible.management.bitsa.organisation systemd[1]: Starting Zabbix Agent...
Mar 30 09:01:52 ansible.management.bitsa.organisation systemd[1]: zabbix-agent.service start request repeated too quickly, refusing to start.
Mar 30 09:01:52 ansible.management.bitsa.organisation systemd[1]: Failed to start Zabbix Agent.
Mar 30 09:01:52 ansible.management.bitsa.organisation systemd[1]: Unit zabbix-agent.service entered failed state.
Mar 30 09:02:20 ansible.management.bitsa.organisation systemd[1]: Starting Zabbix Agent...
Mar 30 09:02:20 ansible.management.bitsa.organisation systemd[1]: Started Zabbix Agent.
Mar 30 09:17:32 ansible.management.bitsa.organisation systemd[1]: Stopping Zabbix Agent...
Mar 30 09:17:32 ansible.management.bitsa.organisation systemd[1]: Stopped Zabbix Agent.
Hint: Some lines were ellipsized, use -l to show in full.
[root@ansible run]# systemctl start zabbix-agent
[root@ansible run]# systemctl status zabbix-agent
zabbix-agent.service - Zabbix Agent
   Loaded: loaded (/usr/lib/systemd/system/zabbix-agent.service; disabled)
   Active: active (running) since Mon 2015-03-30 09:17:49 CEST; 4s ago
  Process: 13836 ExecStart=/usr/sbin/zabbix_agentd -c $CONFFILE (code=exited, status=0/SUCCESS)
   CGroup: /system.slice/zabbix-agent.service
           ├─13501 /usr/sbin/zabbix_agentd -c /etc/zabbix/zabbix_agentd.conf
           ├─13502 /usr/sbin/zabbix_agentd: collector [idle 1 sec]
           ├─13503 /usr/sbin/zabbix_agentd: listener #1 [waiting for connection]
           ├─13504 /usr/sbin/zabbix_agentd: listener #2 [waiting for connection]
           ├─13505 /usr/sbin/zabbix_agentd: listener #3 [waiting for connection]
           └─13506 /usr/sbin/zabbix_agentd: active checks #1 [idle 1 sec]

Mar 30 09:17:49 ansible.management.bitsa.organisation systemd[1]: Starting Zabbix Agent...
Mar 30 09:17:49 ansible.management.bitsa.organisation systemd[1]: Started Zabbix Agent.
[root@ansible run]# systemctl stop zabbix-agent
[root@ansible run]# systemctl status zabbix-agent
zabbix-agent.service - Zabbix Agent
   Loaded: loaded (/usr/lib/systemd/system/zabbix-agent.service; disabled)
   Active: inactive (dead)

Mar 30 09:01:52 ansible.management.bitsa.organisation systemd[1]: Failed to start Zabbix Agent.
Mar 30 09:01:52 ansible.management.bitsa.organisation systemd[1]: Unit zabbix-agent.service entered failed state.
Mar 30 09:02:20 ansible.management.bitsa.organisation systemd[1]: Starting Zabbix Agent...
Mar 30 09:02:20 ansible.management.bitsa.organisation systemd[1]: Started Zabbix Agent.
Mar 30 09:17:32 ansible.management.bitsa.organisation systemd[1]: Stopping Zabbix Agent...
Mar 30 09:17:32 ansible.management.bitsa.organisation systemd[1]: Stopped Zabbix Agent.
Mar 30 09:17:49 ansible.management.bitsa.organisation systemd[1]: Starting Zabbix Agent...
Mar 30 09:17:49 ansible.management.bitsa.organisation systemd[1]: Started Zabbix Agent.
Mar 30 09:18:00 ansible.management.bitsa.organisation systemd[1]: Stopping Zabbix Agent...
Mar 30 09:18:00 ansible.management.bitsa.organisation systemd[1]: Stopped Zabbix Agent.
[root@ansible run]# 

The workaround above works for the agent but not the server. With both removed from the agent the pid is located at /tmp/zabbix_agentd.pid. Part of the problem seems to be that when a PID location is defined, no pid file is actually created.

Also, systemctl status zabbix-server shows the following most, if not all, the time:

Apr 30 07:58:12 giles systemd[1]: Starting Zabbix Server...
Apr 30 07:58:12 giles systemd[1]: zabbix-server.service start request repeated too quickly, refusing to start.
Apr 30 07:58:12 giles systemd[1]: Failed to start Zabbix Server.
Apr 30 07:58:12 giles systemd[1]: Unit zabbix-server.service entered failed state.

In particular, the "repeated too quickly" part stands out.

Also affects zabbix-agent v2.4.5 on CentOS 7.

Also affects zabbix_server on CentOS 7 with selinux disabled.

 -bash-4.2# zabbix_server -V
 Zabbix server v2.4.5 (revision 53282) (21 April 2015)
 Compilation time: Apr 23 2015 03:14:46

 CentOS Linux release 7.1.1503 (Core)

The start service command fails, but the process starts with pid file errors, after that you can't stop/restart the service. Tried to comment out the pid location in config (which defaults to /tmp), without any luck.

-bash-4.2# systemctl start zabbix-server.service
Job for zabbix-server.service failed. See 'systemctl status zabbix-server.service' and 'journalctl -xn' for details.

 zabbix_server [1860]: Is this process already running? Could not lock PID file [/tmp/zabbix_server.pid]: [11] Resource temporarily unavailable
 zabbix_server [1864]: Is this process already running? Could not lock PID file [/tmp/zabbix_server.pid]: [11] Resource temporarily unavailable
 zabbix_server [1868]: Is this process already running? Could not lock PID file [/tmp/zabbix_server.pid]: [11] Resource temporarily unavailable
 zabbix_server [1872]: Is this process already running? Could not lock PID file [/tmp/zabbix_server.pid]: [11] Resource temporarily unavailable
 zabbix_server [1876]: Is this process already running? Could not lock PID file [/tmp/zabbix_server.pid]: [11] Resource temporarily unavailable

The pid file is being created, however it does not seem that Zabbix Server tries to use it when a stop/restart is called.

 -bash-4.2# ls -la /tmp/zabbix_server.pid
 -rw-rw-r-- 1 zabbix zabbix 3 May 12 22:13 /tmp/zabbix_server.pid

-bash-4.2# cat zabbix-agent.service
[Unit]
Description=Zabbix Agent
After=syslog.target
After=network.target

[Service]
Environment="CONFFILE=/etc/zabbix/zabbix_agentd.conf"
EnvironmentFile=-/etc/sysconfig/zabbix-agent
Type=forking
Restart=always
PIDFile=/run/zabbix/zabbix_agentd.pid
KillMode=process
ExecStart=/usr/sbin/zabbix_agentd -c $CONFFILE

[Install]
WantedBy=multi-user.target

-bash-4.2# cat zabbix-server.service
[Unit]
Description=Zabbix Server
After=syslog.target
After=network.target

[Service]
Environment="CONFFILE=/etc/zabbix/zabbix_server.conf"
EnvironmentFile=-/etc/sysconfig/zabbix-server
Type=forking
Restart=always
PIDFile=/run/zabbix/zabbix_server.pid
KillMode=process
ExecStart=/usr/sbin/zabbix_server -c $CONFFILE

[Install]
WantedBy=multi-user.target

-bash-4.2# stat /usr/sbin/zabbix_agentd
  File: ‘/usr/sbin/zabbix_agentd’
  Size: 307088          Blocks: 600        IO Block: 4096   regular file
Device: 3530b6a1h/892384929d    Inode: 531358      Links: 1
Access: (0755/-rwxr-xr-x)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2015-05-12 20:24:38.629667114 +0100
Modify: 2015-04-23 04:20:59.000000000 +0100
Change: 2015-05-12 20:22:30.377230505 +0100
 Birth: -

-bash-4.2# stat /usr/sbin/zabbix_server
  File: ‘/usr/sbin/zabbix_server’ -> ‘/etc/alternatives/zabbix-server’
  Size: 31              Blocks: 0          IO Block: 4096   symbolic link
Device: 3530b6a1h/892384929d    Inode: 531356      Links: 1
Access: (0777/lrwxrwxrwx)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2015-05-12 20:23:30.509372928 +0100
Modify: 2015-05-12 20:22:29.260209282 +0100
Change: 2015-05-12 20:22:29.260209282 +0100
 Birth: -

-bash-4.2# stat /etc/alternatives/zabbix-server
  File: ‘/etc/alternatives/zabbix-server’ -> ‘/usr/sbin/zabbix_server_mysql’
  Size: 29              Blocks: 0          IO Block: 4096   symbolic link
Device: 3530b6a1h/892384929d    Inode: 393640      Links: 1
Access: (0777/lrwxrwxrwx)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2015-05-12 20:23:30.509372928 +0100
Modify: 2015-05-12 20:22:29.260209282 +0100
Change: 2015-05-12 20:22:29.260209282 +0100
 Birth: -

-bash-4.2# stat /usr/sbin/zabbix_server_mysql
  File: ‘/usr/sbin/zabbix_server_mysql’
  Size: 1392344         Blocks: 2720       IO Block: 4096   regular file
Device: 3530b6a1h/892384929d    Inode: 531355      Links: 1
Access: (0755/-rwxr-xr-x)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2015-05-12 20:23:30.509372928 +0100
Modify: 2015-04-23 04:20:59.000000000 +0100
Change: 2015-05-12 20:22:29.102206280 +0100
 Birth: -

Zabbix agent can start/stop/restart without issues, with pid file line commented in conf file, or with a custom path.

-bash-4.2# zabbix_agent -V
Zabbix agent v2.4.5 (revision 53282) (21 April 2015)
Compilation time: Apr 23 2015 03:19:15

Please let us know what else can we give you to fix this issue. Thanks

I've managed to fix this by downgrading the gnutls packages from latest 3.3.8-12.el7 to:

gnutls-3.1.18-10.el7_0.x86_64.rpm
gnutls-c++-3.1.18-10.el7_0.x86_64.rpm
gnutls-dane-3.1.18-10.el7_0.x86_64.rpm
gnutls-devel-3.1.18-10.el7_0.x86_64.rpm

from vault repo at http://vault.centos.org/7.0.1406/updates/x86_64/Packages/

Hopefully you can find the cause and apply a fix for zabbix-2.4.5 init scripts.

As mentioned before, this is also a problem on CentOS 6 / Red Hat 6. Based on the comment above I ran some checks on one of each:

This box is fully patched:

[root@zaakje gliverma]# cat /etc/redhat-release
CentOS release 6.6 (Final)

[root@zaakje gliverma]# ps -eaf|grep zabbix
zabbix   26437     1  0 Feb25 ?        00:00:00 zabbix_agentd -c /etc/zabbix/zabbix_agentd.conf
zabbix   26439 26437  0 Feb25 ?        01:19:38 zabbix_agentd: collector [idle 1 sec]
zabbix   26440 26437  0 Feb25 ?        04:24:58 zabbix_agentd: listener #1 [waiting for connection]
zabbix   26441 26437  0 Feb25 ?        04:25:10 zabbix_agentd: listener #2 [waiting for connection]
zabbix   26442 26437  0 Feb25 ?        04:24:34 zabbix_agentd: listener #3 [waiting for connection]
zabbix   26443 26437  0 Feb25 ?        00:06:54 zabbix_agentd: active checks #1 [idle 1 sec]
root     28267 28255  0 08:01 pts/0    00:00:00 grep zabbix

[root@zaakje gliverma]# service zabbix-agent stop
Shutting down Zabbix agent:                                [FAILED]

[root@zaakje gliverma]# ps -eaf|grep zabbix
zabbix   26437     1  0 Feb25 ?        00:00:00 zabbix_agentd -c /etc/zabbix/zabbix_agentd.conf
zabbix   26439 26437  0 Feb25 ?        01:19:38 zabbix_agentd: collector [idle 1 sec]
zabbix   26440 26437  0 Feb25 ?        04:24:58 zabbix_agentd: listener #1 [waiting for connection]
zabbix   26441 26437  0 Feb25 ?        04:25:10 zabbix_agentd: listener #2 [waiting for connection]
zabbix   26442 26437  0 Feb25 ?        04:24:35 zabbix_agentd: listener #3 [waiting for connection]
zabbix   26443 26437  0 Feb25 ?        00:06:54 zabbix_agentd: active checks #1 [idle 1 sec]
root     28291 28255  0 08:01 pts/0    00:00:00 grep zabbix

[root@zaakje gliverma]# kill 26437

[root@zaakje gliverma]# ps -eaf|grep zabbix
root     28296 28255  0 08:01 pts/0    00:00:00 grep zabbix

[root@zaakje gliverma]# service zabbix-agent start
Starting Zabbix agent:                                     [  OK  ]

[root@zaakje gliverma]# ps -eaf|grep zabbix
zabbix   28313     1  0 08:02 ?        00:00:00 zabbix_agentd -c /etc/zabbix/zabbix_agentd.conf
zabbix   28315 28313  0 08:02 ?        00:00:00 zabbix_agentd: collector [idle 1 sec]
zabbix   28316 28313  0 08:02 ?        00:00:00 zabbix_agentd: listener #1 [waiting for connection]
zabbix   28317 28313  0 08:02 ?        00:00:00 zabbix_agentd: listener #2 [waiting for connection]
zabbix   28318 28313  0 08:02 ?        00:00:00 zabbix_agentd: listener #3 [waiting for connection]
zabbix   28319 28313  0 08:02 ?        00:00:00 zabbix_agentd: active checks #1 [idle 1 sec]
root     28325 28255  0 08:02 pts/0    00:00:00 grep zabbix

[root@zaakje gliverma]# rpm -qa|grep gnutls
gnutls-2.8.5-14.el6_5.x86_64

This box is not patched due vendor requirements:

[root@uwg-dw01 gliverma]# cat /etc/redhat-release
Red Hat Enterprise Linux Server release 6.1 (Santiago)

[root@uwg-dw01 gliverma]# ps -eaf|grep zabbix
zabbix    2729     1  0 Mar10 ?        00:00:00 zabbix_agentd -c /etc/zabbix/zabbix_agentd.conf
zabbix    2735  2729  0 Mar10 ?        04:47:02 zabbix_agentd: collector [idle 1 sec]
zabbix    2736  2729  0 Mar10 ?        04:14:27 zabbix_agentd: listener #1 [waiting for connection]
zabbix    2737  2729  0 Mar10 ?        04:14:57 zabbix_agentd: listener #2 [waiting for connection]
zabbix    2738  2729  0 Mar10 ?        04:14:28 zabbix_agentd: listener #3 [waiting for connection]
zabbix    2740  2729  0 Mar10 ?        00:05:50 zabbix_agentd: active checks #1 [idle 1 sec]
root     26006 25993  0 08:07 pts/1    00:00:00 grep zabbix

[root@uwg-dw01 gliverma]# service zabbix-agent stop
Shutting down Zabbix agent:                                [FAILED]

[root@uwg-dw01 gliverma]# ps -eaf|grep zabbix
zabbix    2729     1  0 Mar10 ?        00:00:00 zabbix_agentd -c /etc/zabbix/zabbix_agentd.conf
zabbix    2735  2729  0 Mar10 ?        04:47:02 zabbix_agentd: collector [idle 1 sec]
zabbix    2736  2729  0 Mar10 ?        04:14:27 zabbix_agentd: listener #1 [waiting for connection]
zabbix    2737  2729  0 Mar10 ?        04:14:57 zabbix_agentd: listener #2 [waiting for connection]
zabbix    2738  2729  0 Mar10 ?        04:14:28 zabbix_agentd: listener #3 [waiting for connection]
zabbix    2740  2729  0 Mar10 ?        00:05:50 zabbix_agentd: active checks #1 [idle 1 sec]
root     26024 25993  0 08:07 pts/1    00:00:00 grep zabbix

[root@uwg-dw01 gliverma]# kill 2729

[root@uwg-dw01 gliverma]# ps -eaf|grep zabbix
root     26030 25993  0 08:07 pts/1    00:00:00 grep zabbix

[root@uwg-dw01 gliverma]# service zabbix-agent start
Starting Zabbix agent:                                     [  OK  ]

[root@uwg-dw01 gliverma]# ps -eaf|grep zabbix
zabbix   26065     1  0 08:07 ?        00:00:00 zabbix_agentd -c /etc/zabbix/zabbix_agentd.conf
zabbix   26067 26065  0 08:07 ?        00:00:00 zabbix_agentd: collector [idle 1 sec]
zabbix   26068 26065  0 08:07 ?        00:00:00 zabbix_agentd: listener #1 [waiting for connection]
zabbix   26069 26065  0 08:07 ?        00:00:00 zabbix_agentd: listener #2 [waiting for connection]
zabbix   26070 26065  0 08:07 ?        00:00:00 zabbix_agentd: listener #3 [waiting for connection]
zabbix   26071 26065  0 08:07 ?        00:00:00 zabbix_agentd: active checks #1 [idle 1 sec]
root     26073 25993  0 08:08 pts/1    00:00:00 grep zabbix

[root@uwg-dw01 gliverma]# rpm -qa|grep gnutls
gnutls-2.8.5-4.el6.x86_64

As you can see, the latter box has gnutls 2.8.5-4 which, according to http://vault.centos.org/6.0/os/x86_64/Packages/, is the oldest one available for CentOS 6.

Still I cannot reproduce this this problem,, but is CentOS6 problem related to ZBX-9626 ?

Hi,
This worked for me on oracle Linux (RHEL 7)

vi /etc/systemd/system/multi-user.target.wants/zabbix-agent.service
#use this PID file path (because only this path works)
PIDFile=/tmp/zabbix_agentd.pid

systemctl daemon-reload

now... unccoment pidfile path in zabbix agent conf file:
vi /etc/zabbix/zabbix_agentd.conf

1. PidFile=/tmp/zabbix_agentd.pid

–
zabbix_agent -V
Zabbix agent v2.4.5 (revision 53282) (21 April 2015)
Compilation time: Apr 23 2015 03:19:15

I have the same issue. After reboot zabbix-agent wont start. SELinux disabled. I have tried setting PID to /tmp/zabbix_agentd.pid but no difference. Im on CentOS7

zabbix_agentd [3239]: cannot create PID file [/var/run/zabbix/zabbix_agentd.pid]: [2] No such file or directory

PID file /var/run/zabbix/zabbix_agentd.pid not readable (yet?) after start.
Jun 24 06:00:21 CHC01R0LBHAPX01 systemd: zabbix-agent.service never wrote its PID file. Failing.
Jun 24 06:00:21 CHC01R0LBHAPX01 systemd: Failed to start Zabbix Agent.
Jun 24 06:00:21 CHC01R0LBHAPX01 systemd: Unit zabbix-agent.service entered failed state.
Jun 24 06:00:21 CHC01R0LBHAPX01 systemd: zabbix-agent.service holdoff time over, scheduling restart.
Jun 24 06:00:21 CHC01R0LBHAPX01 systemd: Stopping Zabbix Agent...
Jun 24 06:00:21 CHC01R0LBHAPX01 systemd: Starting Zabbix Agent...
Jun 24 06:00:21 CHC01R0LBHAPX01 systemd: PID file /var/run/zabbix/zabbix_agentd.pid not readable (yet?) after start.

Installed Packages
Name : zabbix-agent
Arch : x86_64
Version : 2.4.5
Release : 1.el7
Size : 566 k
Repo : installed
From repo : zabbix
Summary : Zabbix Agent
URL : http://www.zabbix.com/
License : GPLv2+
Description : The Zabbix client agent, to be installed on monitored systems.

I was incorrect in post before:
Try to do this : (This worked for me on oracle Linux (RHEL 7))

vi /etc/systemd/system/multi-user.target.wants/zabbix-agent.service
#use this PID file path (because only this path works)
PIDFile=/tmp/zabbix_agentd.pid

systemctl daemon-reload

now... COMMENT pidfile path in zabbix agent conf file:
vi /etc/zabbix/zabbix_agentd.conf

#PidFile=/tmp/zabbix_agentd.pid

shell> setenforce 0

if you do not want disable selinux you can do this:

semodule -l | grep zabbix
semodule -e zabbix

getsebool -a | grep zabbix
setsebool -P zabbix_can_network on

yum provides \*/audit2allow
yum install policycoreutils-python

Now check log

tail -8 /var/log/audit/audit.log

and for each row with
type ="AVC" create
(and system_r:zabbix_agent_t)
create a file.

This was my audit.log, so i create 14 file

echo type=AVC msg=audit(1435075094.231:41126): avc:  denied  { create } for  pid=21207 comm="zabbix_agentd" scontext=system_u:system_r:zabbix_agent_t:s0 tcontext=system_u:system_r:zabbix_agent_t:s0 tclass=unix_dgram_socket' > file1

echo 'type=AVC msg=audit(1435075379.703:42670): avc:  denied  { connect } for  pid=21207 comm="zabbix_agentd" scontext=system_u:system_r:zabbix_agent_t:s0 tcontext=system_u:system_r:zabbix_agent_t:s0 tclass=unix_dgram_socket' > file2

echo 'type=AVC msg=audit(1435075645.627:44293): avc:  denied  { write } for  pid=21206 comm="zabbix_agentd" name="log" dev="devtmpfs" ino=1453 scontext=system_u:system_r:zabbix_agent_t:s0 tcontext=system_u:object_r:devlog_t:s0 tclass=sock_file' > file3

echo 'type=AVC msg=audit(1435075728.386:44707): avc:  denied  { sendto } for  pid=21207 comm="zabbix_agentd" path="/dev/log" scontext=system_u:system_r:zabbix_agent_t:s0 tcontext=system_u:system_r:kernel_t:s0 tclass=unix_dgram_socket' > file4

echo 'type=AVC msg=audit(1435075811.172:45035): avc:  denied  { write } for  pid=24416 comm="zabbix_agentd" name="tmp" dev="dm-0" ino=201326721 scontext=system_u:system_r:zabbix_agent_t:s0 tcontext=system_u:object_r:tmp_t:s0 tclass=dir' > file5

echo 'type=AVC msg=audit(1435075901.284:45041): avc:  denied  { add_name } for  pid=24436 comm="zabbix_agentd" name="zabbix_agentd.pid" scontext=system_u:system_r:zabbix_agent_t:s0 tcontext=system_u:object_r:tmp_t:s0 tclass=dir' > file6

echo 'type=AVC msg=audit(1435076171.596:45087): avc:  denied  { create } for  pid=24530 comm="zabbix_agentd" name="zabbix_agentd.pid" scontext=system_u:system_r:zabbix_agent_t:s0 tcontext=system_u:object_r:tmp_t:s0 tclass=file' > file7

echo 'type=AVC msg=audit(1435076261.705:45093): avc:  denied  { write open } for  pid=24553 comm="zabbix_agentd" path="/tmp/zabbix_agentd.pid" dev="dm-0" ino=201432265 scontext=system_u:system_r:zabbix_agent_t:s0 tcontext=system_u:object_r:tmp_t:s0 tclass=file' > file8

echo 'type=AVC msg=audit(1435076442.140:45113): avc:  denied  { remove_name } for  pid=24587 comm="zabbix_agentd" name="zabbix_agentd.pid" dev="dm-0" ino=201430045 scontext=system_u:system_r:zabbix_agent_t:s0 tcontext=system_u:object_r:tmp_t:s0 tclass=dir' > file9

echo 'type=AVC msg=audit(1435076532.703:45140): avc:  denied  { unlink } for  pid=24616 comm="zabbix_agentd" name="zabbix_agentd.pid" dev="dm-0" ino=201432588 scontext=system_u:system_r:zabbix_agent_t:s0 tcontext=system_u:object_r:tmp_t:s0 tclass=file' > file10

echo 'type=AVC msg=audit(1435076653.319:45169): avc:  denied  { write } for  pid=24683 comm="mysql" name="mysql.sock" dev="dm-0" ino=135227527 scontext=system_u:system_r:zabbix_agent_t:s0 tcontext=system_u:object_r:mysqld_var_run_t:s0 tclass=sock_file' > file11

echo 'type=AVC msg=audit(1435077249.690:45198): avc:  denied  { connectto } for  pid=24794 comm="mysql" path="/var/lib/mysql/mysql.sock" scontext=system_u:system_r:zabbix_agent_t:s0 tcontext=system_u:system_r:mysqld_t:s0 tclass=unix_stream_socket' > file12


echo 'type=AVC msg=audit(1435077243.460:45183): avc:  denied  { read } for  pid=24766 comm="mysql" name="my.cnf" dev="dm-0" ino=68033735 scontext=system_u:system_r:zabbix_agent_t:s0 tcontext=system_u:object_r:mysqld_etc_t:s0 tclass=file' > file13

echo 'type=AVC msg=audit(1435077808.003:45227): avc:  denied  { open } for  pid=24934 comm="mysql" path="/etc/my.cnf" dev="dm-0" ino=68033735 scontext=system_u:system_r:zabbix_agent_t:s0 tcontext=system_u:object_r:mysqld_etc_t:s0 tclass=file' > file14

now install policy:

audit2allow -i file1 -M zabbix_agentd1
semodule -i zabbix_agentd1.pp

audit2allow -i file2 -M zabbix_agentd2
semodule -i zabbix_agentd2.pp

audit2allow -i file3 -M zabbix_agentd3
semodule -i zabbix_agentd3.pp

audit2allow -i file4 -M zabbix_agentd4
semodule -i zabbix_agentd4.pp

audit2allow -i file5 -M zabbix_agentd5
semodule -i zabbix_agentd5.pp

audit2allow -i file6 -M zabbix_agentd6
semodule -i zabbix_agentd6.pp

audit2allow -i file7 -M zabbix_agentd7
semodule -i zabbix_agentd7.pp

audit2allow -i file8 -M zabbix_agentd8
semodule -i zabbix_agentd8.pp

audit2allow -i file9 -M zabbix_agentd9
semodule -i zabbix_agentd9.pp

audit2allow -i file10 -M zabbix_agentd10
semodule -i zabbix_agentd10.pp

audit2allow -i file11 -M zabbix_agentd11
semodule -i zabbix_agentd11.pp

audit2allow -i file12 -M zabbix_agentd12
semodule -i zabbix_agentd12.pp

audit2allow -i file13 -M zabbix_agentd13
semodule -i zabbix_agentd13.pp

audit2allow -i file14 -M zabbix_agentd14
semodule -i zabbix_agentd14.pp

Many thanks for sysops!
My OS is Oracle Linux 7.
His post helped me, but all will work, if you do the following things(in my case):

1. I installed zabbix-agent from repository
2. Create simlink
ln -s /usr/lib/systemd/system/zabbix-agent.service /etc/systemd/system/multi-user.target.wants/zabbix-agent.service
3. Comment line in /usr/lib/systemd/system/zabbix-agent.service
##PIDFile=/run/zabbix/zabbix_agentd.pid
then add
PIDFile=/tmp/zabbix_agentd.pid
4. Do command
systemctl daemon-reload
5. Disable selinux (because zabbix-agent don't want to start) or add rule for zabbix-agent
6. Execute the command
systemctl start zabbix-agent
In my case, I got error in zabbix_agentd.log:
"listener failed: zbx_tcp_listen() fatal error: unable to serve on any address [[-]:10050]"
I executed "netstat -ltn" and I saw that the port is already listening. Then I kill zabbix-agent processes and start zabbix-agent. It's working!

This does serve as a work around for the agent but does not fix the server. With this setup I can get the server's status but I cannot stop it unless I use kill.

Also, on CentOS 6, the agent responds like the server on RHEL 7... status works with this change but stop does not.

We are having the same problem with Zabbix 2.4.5 (both server and agent services) on a fully patched RHEL 7.1 installation. Disabling SELinux simply isn't an option for us.

systemctl status zabbix-server.service shows:
PID file /run/zabbix/zabbix_server.pid not readable (yet?) after start.

systemctl status zabbix-agent.service shows:
PID file /run/zabbix/zabbix_agentd.pid not readable (yet?) after start.

We've enabled the following SELinux booleans:
setsebool -P httpd_can_connect_zabbix 1
setsebool -P zabbix_can_network 1

I'm really surprised about this issue for two reasons:
1) It wasn't caught in testing (you can't even START the service without an error)
2) It's been allowed to languish this long

I use Oracle Linux (RHEL 7) with zabbix agent (not server) and I had the same problem
I resolved by adding rules on selinux.
type:

sudo tail -f  /var/log/audit/audit.log | grep zabbix

and if you find any rows then
try to do what I have reported in the post above:
https://support.zabbix.com/browse/ZBX-9353?focusedCommentId=146782&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-146782

You are mixing up two separate issues here! The fact that a PID file in /run/zabbix/zabbix_server.pid is not allowed by policy, is due to the fact that the policy expects it in /run/zabbixsrv. If you want your PID file in /run/zabbix, you have to run your own policy package.

The fact that you can't disable it in the config, is due to the systemd unit file, which is looking for one. My experience is, if you don't define one, systemd apparently doesn't know which one is the master process. Therefore I left it in for Fedora/EPEL. The comment should be adapted, I know.

Related to https://bugzilla.redhat.com/show_bug.cgi?id=1220392

today i had a similar/related issue. basically zabbix-agent did not want to start. The zabbix-agent log said, it could not create the pidfile and this was due to a missing /run/zabbix directory.

to fix this i

• created a /etc/tmpfiles.d/zabbix.conf file with the following contents:
  D /run/zabbix 0755 zabbix zabbix

• ran systemd-tmpfiles zabbix.conf --create
• now zabbix-agent starts up just fine.

If I am not missing something, it may be necessary to include the tmpfiles.d file in the distribution, as the /run/zabbix directory has to be recreated on every boot (as /run is on tmpfs and all its contents are lost on reboot).

[bangert@franck system]$ cat /etc/redhat-release
CentOS Linux release 7.1.1503 (Core)
[bangert@franck system]$ sestatus
SELinux status: enabled
SELinuxfs mount: /sys/fs/selinux
SELinux root directory: /etc/selinux
Loaded policy name: targeted
Current mode: enforcing
Mode from config file: enforcing
Policy MLS status: enabled
Policy deny_unknown status: allowed
Max kernel policy version: 28

Hi Guys,

This problem is related to the fact that /run directory is tmpfs. That is, it is created at every boot. That's why e.g. permissions set by RPM packages will be lost when the server is booted.

To fix the issue correctly, zabbix service files (under /usr/lib/systemd/system/) have to create the run directories with root authories before the service is executed. This is how it's done:

(This is my modified version of zabbix-agentd.service, but same trick has to be applied to zabbix-server.service file)

[Unit]
Description=Zabbix Monitor Agent
After=syslog.target network.target

[Service]
Type=simple

# Create and chown /var/run/zabbix directory with root authorieis
PermissionsStartOnly=true
ExecStartPre=-/usr/bin/mkdir -p /var/run/zabbix
ExecStartPre=/usr/bin/chown -R zabbix:zabbix /var/run/zabbix

ExecStart=/usr/sbin/zabbix-agentd -f
User=zabbix
Group=zabbix

PrivateTmp=yes

[Install]
WantedBy=multi-user.target

Please add this fix to both, server and agent!

This fix should be applied to any service that tries to creat PID file under /var/run/zabbix directory.

Hi Jari,

i agree with your analysis, however the correct way to fix this is to install a tmpfiles config file - that way systemd will create the run dir upon boot. No need to ExecStartPre..

This is what i have tried to explain in the comment above. Revisiting this just now, i realize that the tmpfiles.d file should not be installed in /etc/tmpfiles.d but in /usr/lib/tmpfiles.d/

See also http://www.freedesktop.org/software/systemd/man/tmpfiles.d.html

Thanks

That sounds like a better fix!

Can you provide an example, what should the config file look like?

This is what i would propose for /usr/lib/tmpfiles.d/zabbix.conf:

  # PID file directory for zabbix 
  D       /run/zabbix     0755    zabbix  zabbix

The above works for me.
As the file is required by multiple packages (both agent and server need it) it should be located in a common dependency. I haven't figured out where that should be.

I just noticed that in my fresh installation (done today), zabbix-server service is run as "zabbixs" user! If the group is set to zabbixs, I guess that would solve it?

Alternatively, if zabbix-server would use different run directory (/run/zabbixs), that would solve it as well, but currently it seems to create its PID file in the same directory as zabbix-agentd.

ah - i see.

personally i'd prefer if all components had their own dedicated run directories, as you describe as an alternative.

The solution with the zabbixs group is not so nice, since it would require the zabbixs group also to be present on machines that install the client only, and that doesnt make any sense...

Yes, I agree, and actually there seems to be in my installation (opensuse 13.2) separate tmpfile for zabbix-agent and zabbix-server, but there were errors in them and also errors in default configuration files and default service files. After fixing them all manually, everything worked like a charm.

Should there be separate bugs related to all these files?

Dont know. My take is that these issues are so simple, they should just get fixed.... but this bug has been lingering for way to long already, so you might just as well try and open a/some new issue(s) - perhaps that gets the ball moving...

Some different things are mixed in comments. I updated systemd service file in 2.2.12rc1, 2.2.8rc1 and 3.0.0rc1. I hope the fix solve original issue.

zalex_ua The change supposedly fixed this message of systemd right after daemon is started:

"Zabbix-server.service: Supervising process 28719 which is not our child. We'll most likely not notice when it exits."

where PID looks like pid of running main daemon process

Guys, also we suggest to change in systemd file next thing:

KillMode=process

to

KillMode=control-group

Today we have found the issue, if you try to restart agent, it doesn't start. And in logs a tons of:

2016-02-15T08:29:30.677018+00:00 sg1-up-a85 zabbix_agentd[51975]: using configuration file: /etc/zabbix_agentd.conf
2016-02-15T08:29:30.677386+00:00 sg1-up-a85 zabbix_agentd[51975]: listener failed: bind() for [[-]:10050] failed: [98] Address already in use
2016-02-15T08:29:30.944696+00:00 sg1-up-a85 zabbix_agentd[51985]: Starting Zabbix Agent [sg1-up-a85]. Zabbix 2.2.9 (revision 52686).
2016-02-15T08:29:30.945185+00:00 sg1-up-a85 zabbix_agentd[51985]: using configuration file: /etc/zabbix_agentd.conf
2016-02-15T08:29:30.945512+00:00 sg1-up-a85 zabbix_agentd[51985]: listener failed: bind() for [[-]:10050] failed: [98] Address already in use
2016-02-15T08:29:31.196747+00:00 sg1-up-a85 zabbix_agentd[52003]: Starting Zabbix Agent [sg1-up-a85]. Zabbix 2.2.9 (revision 52686).
2016-02-15T08:29:31.197314+00:00 sg1-up-a85 zabbix_agentd[52003]: using configuration file: /etc/zabbix_agentd.conf
2016-02-15T08:29:31.197866+00:00 sg1-up-a85 zabbix_agentd[52003]: listener failed: bind() for [[-]:10050] failed: [98] Address already in use
2016-02-15T08:29:31.312681+00:00 sg1-up-a85 zabbix_agentd[52012]: Starting Zabbix Agent [sg1-up-a85]. Zabbix 2.2.9 (revision 52686).
2016-02-15T08:29:31.313097+00:00 sg1-up-a85 zabbix_agentd[52012]: using configuration file: /etc/zabbix_agentd.conf
2016-02-15T08:29:31.313502+00:00 sg1-up-a85 zabbix_agentd[52012]: listener failed: bind() for [[-]:10050] failed: [98] Address already in use

Please change this behavior to default.

For 3.0, my approach for Fedora and consequently EPEL7 will be, to run the daemons in the foreground, remove the PidFile directive and set Type to simple.

Hi,

First post

I too had this problem. Latest CentOS Linux release 7.2.1511 (Core) and Zabbix REPO from

http://repo.zabbix.com/zabbix/3.0/rhel/7/x86_64/zabbix-release-3.0-1.el7.noarch.rpm

I worked around this by editing /etc/zabbix/zabbix_server.conf and switching out the PidFile location to /tmp.

i.e.

PidFile=/tmp/zabbix_server.pid
#PidFile=/var/run/zabbix/zabbix_server.pid

I then edited /usr/lib/systemd/system/zabbix-server.service [Service] section to point to the new PIDFile location.

[Service]
Environment="CONFFILE=/etc/zabbix/zabbix_server.conf"
EnvironmentFile=-/etc/sysconfig/zabbix-server
Type=forking
Restart=on-failure
PIDFile=/tmp/zabbix_server.pid
#PIDFile=/run/zabbix/zabbix_server.pid
KillMode=control-group
ExecStart=/usr/sbin/zabbix_server -c $CONFFILE
ExecStop=/bin/kill -SIGTERM $MAINPID
RestartSec=10s

The service now starts and restarts and I can get to the setup page. It all survives a reboot too. Hope this helps someone else!

Answering to anton.samets's suggestion above:
as I see such changes were implemented in packages:

zabbix-agent-2.4.7-1.el7.x86_64.rpm - KillMode=process
zabbix-agent-2.4.8-1.el7.x86_64.rpm - KillMode=control-group

and added: ExecStop=/bin/kill -SIGTERM $MAINPID

I was having this issue in Ubuntu 16.04 recently as well, to get around it there is a config file named zabbix-agent.conf in /usr/lib/tmpfiles.d it seems there is a missing - - at the end of the line adding the zabbix pid folder

old line

d /var/run/zabbix 0755 zabbix zabbix

new line

d /var/run/zabbix 0755 zabbix zabbix - -

this has solved my issue.

Thank you very much for the pointed it out. I will fix it on 3.0.9 and 2.2.18

Same problem in Zabbix-Agent 3.2.6-1+jessie

root@ntp2:~# /etc/init.d/zabbix-agent status
● zabbix-agent.service - Zabbix Agent
Loaded: loaded (/lib/systemd/system/zabbix-agent.service; enabled; vendor preset: enabled)
Active: active (running) since Thu 2017-06-22 18:25:43 CEST; 2min 10s ago
Process: 30106 ExecStop=/bin/kill -SIGTERM $MAINPID (code=exited, status=0/SUCCESS)
Process: 30146 ExecStart=/usr/sbin/zabbix_agentd -c $CONFFILE (code=exited, status=0/SUCCESS)
Main PID: 30148 (zabbix_agentd)
Tasks: 6 (limit: 4915)
CGroup: /system.slice/zabbix-agent.service
├─30148 /usr/sbin/zabbix_agentd -c /etc/zabbix/zabbix_agentd.conf
├─30150 /usr/sbin/zabbix_agentd: collector [idle 1 sec]
├─30151 /usr/sbin/zabbix_agentd: listener #1 [waiting for connection]
├─30152 /usr/sbin/zabbix_agentd: listener #2 [waiting for connection]
├─30153 /usr/sbin/zabbix_agentd: listener #3 [waiting for connection]
└─30154 /usr/sbin/zabbix_agentd: active checks #1 [idle 1 sec]

cze 22 18:25:43 ntp2 systemd[1]: Starting Zabbix Agent...
cze 22 18:25:43 ntp2 systemd[1]: zabbix-agent.service: PID file /run/zabbix/zabbix_agentd.pid not readable (yet?) after start: No such file or directory
cze 22 18:25:43 ntp2 systemd[1]: Started Zabbix Agent.

root@ntp2:~# ls -al /run/zabbix/zabbix_agentd.pid
rw-rw-r- 1 zabbix zabbix 5 cze 22 18:25 /run/zabbix/zabbix_agentd.pid

We are experiencing the same issue with the zabbix agent. We are running Ubuntu 16.04.1 LTS and zabbix_agentd (daemon) (Zabbix) 3.0.9

zabbix-agent.service - Zabbix Agent
   Loaded: loaded (/lib/systemd/system/zabbix-agent.service; enabled; vendor preset: enabled)
   Active: active (running) since Mon 2017-07-10 16:28:12 BST; 1s ago
  Process: 10197 ExecStop=/bin/kill -SIGTERM $MAINPID (code=exited, status=0/SUCCESS)
  Process: 10280 ExecStart=/usr/sbin/zabbix_agentd -c $CONFFILE (code=exited, status=0/SUCCESS)
 Main PID: 10284 (zabbix_agentd)
    Tasks: 6
   Memory: 2.5M
      CPU: 25ms
   CGroup: /system.slice/zabbix-agent.service
           ├─10284 /usr/sbin/zabbix_agentd -c /etc/zabbix/zabbix_agentd.conf
           ├─10285 /usr/sbin/zabbix_agentd: collector [idle 1 sec]
           ├─10286 /usr/sbin/zabbix_agentd: listener #1 [waiting for connection
           ├─10287 /usr/sbin/zabbix_agentd: listener #2 [waiting for connection
           ├─10288 /usr/sbin/zabbix_agentd: listener #3 [waiting for connection
           └─10289 /usr/sbin/zabbix_agentd: active checks #1 [idle 1 sec]

Jul 10 16:28:12 xvm69256.vps.cloud.tagadab.com systemd[1]: Starting Zabbix Agent...
Jul 10 16:28:12 xvm69256.vps.cloud.tagadab.com systemd[1]: zabbix-agent.service: PID file /run/zabbix/zabbix_agentd.pid not readable (yet?) after start: No such file or directory
Jul 10 16:28:12 xvm69256.vps.cloud.tagadab.com systemd[1]: Started Zabbix Agent.

In the zabbix_agentd.conf the pidfile it set to

PidFile=/var/run/zabbix/zabbix_agentd.pid

In the zabbix-agent.service

[Unit]
Description=Zabbix Agent
After=syslog.target
After=network.target

[Service]
Environment="CONFFILE=/etc/zabbix/zabbix_agentd.conf"
EnvironmentFile=-/etc/default/zabbix-agent
Type=forking
Restart=on-failure
PIDFile=/run/zabbix/zabbix_agentd.pid
KillMode=control-group
ExecStart=/usr/sbin/zabbix_agentd -c $CONFFILE
ExecStop=/bin/kill -SIGTERM $MAINPID
RestartSec=10s

[Install]
WantedBy=multi-user.target

Tried most of the solutions above but cant seem to get zabbix agent to restart

Making a Domain Permissive for zabbix agent
To make a domain permissive, run the semanage permissive -a domain command, where domain is the domain you want to make permissive. For example, run the following command as the Linux root user to make the zabbix_agent_t domain (the domain the Apache HTTP Server runs in) permissive:
semanage permissive -a zabbix_agent_t
systemctl daemon-reload
systemctl restart zabbix-agent
To make sure that semanage run yum must install "policycoreutils-python"
yum install policycoreutils-python
To view a list of domains you have made permissive, run the semodule -l | grep permissive command as the Linux root user. For example:
semodule -l | grep permissive

Same issue with "Oracle Linux Server release 7.4"

srv01 systemd[1]: Failed to start Zabbix Agent.
srv01 systemd[1]: Unit zabbix-agent.service entered failed state.
srv01 systemd[1]: zabbix-agent.service failed.
srv01 systemd[1]: zabbix-agent.service holdoff time over, scheduling restart.
srv01 systemd[1]: Starting Zabbix Agent...
srv01 systemd[1]: PID file /run/zabbix/zabbix_agentd.pid not readable (yet?) after start.

zabbix_agentd (daemon) (Zabbix) 3.2.11
Revision 76339 27 December 2017, compilation time: Dec 27 2017 15:13:43

SELinux status:                 disabled

Agent only works If I manually create /run/zabbix.  

Greetings,

I had the same issue and by looking at logs with this command "tail -1 /var/log/zabbix/zabbix_agentd.log" i have observed:
zabbix_agentd [30615]: cannot open log: cannot create semaphore set: [28] No space left on device

i had gathered info on this from:
https://gerardnico.com/linux/semaphore

and checked the semaphore array max usage:
$>cat /proc/sys/kernel/sem
output: 250 32000 32 128

$>ipcs -s | grep -c "^0x"
output: 128

the last value, 128 is the max limit reached for semaphore arrays, so i needed to increase that, by doing:
$>echo "kernel.sem = 250 32000 32 256" >> /etc/sysctl.conf
$> sysctl -p /etc/sysctl.conf
output: kernel.sem = 250 32000 32 256

After this, i needed to check the new sempahore array:
$> cat /proc/sys/kernel/sem
output: 250 32000 32 256

and restart zabbix:
$> systemctl start zabbix-agent

$> systemctl status zabbix-agent
● zabbix-agent.service - Zabbix Agent
Loaded: loaded (/usr/lib/systemd/system/zabbix-agent.service; enabled; vendor preset: disabled)
Active: active (running) since Sat 2018-05-05 13:20:09 ADT; 26s ago
Process: 30681 ExecStart=/usr/sbin/zabbix_agentd -c $CONFFILE (code=exited, status=0/SUCCESS)
Main PID: 30683 (zabbix_agentd)
CGroup: /system.slice/zabbix-agent.service
├─30683 /usr/sbin/zabbix_agentd -c /etc/zabbix/zabbix_agentd.conf
├─30684 /usr/sbin/zabbix_agentd: collector [idle 1 sec]
├─30685 /usr/sbin/zabbix_agentd: listener #1 [waiting for connection]
├─30686 /usr/sbin/zabbix_agentd: listener #2 [waiting for connection]
├─30687 /usr/sbin/zabbix_agentd: listener #3 [waiting for connection]
└─30688 /usr/sbin/zabbix_agentd: active checks #1 [idle 1 sec]

Thanks to everyone for their help

Zabbix can be stopped with SIGINT, when SIGINT is received by main process it will send SIGTERM to child processes. The problem is that child processes will handle signal by calling exit() function which is not signal-safe and calls functions registered with atexit(3) and on_exit(3). This means that exit() might hang sometimes.

We need to stop zabbix by calling only signal-safe functions from a signal handler or blocking SIGINT signal in places that might call not signal-safe functions.

You can have a look at ZBX-15027 which attempts to solve some issues related to this.

Some info about signal-safety

Is this related to ZBX-3788 (in particular see ZBX-3788_investigation1.odt) or ZBX-9538?

vso proper investigation should be done, however if pkill zabbix_agent -9 helps then it means that old zabbix could not be properly stopped, that’s why it can’t be started again, you can confirm it by checking that only some processes of zabbix have exited.

When install zabbix-agent simple create /usr/lib/tmpfiles.d/zabbix-agent.conf with content

d /var/run/zabbix 0755 zabbix zabbix - -

And error fixed

genebean, is this still an issue or it has been fixed in the newer (>=4.0) versions?

Fixed in feature branch:
https://git.zabbix.com/projects/ZBX/repos/zabbix/pull-requests/191/overview

This issue is affecting our organization. I cannot access the PR from @vso. When will this be made public? Thanks.

Could you please describe the issue davidd ?

The code is available in development branch. If you would like to access it, it's branch named feature/ZBX-9867-4.0, or

https://git.zabbix.com/projects/ZBX/repos/zabbix/commits?until=refs%2Fheads%2Ffeature%2FZBX-9867-4.0

If you are interested in the release which will contain this fix, see "Fix Version/s" in ticket description.

@Vladislavs Sokurenko our issues match the discussion on this page pretty well. We're running CentOS Linux with SELinux enabled, /var/run is a symlink to /run, same inability to restart, etc. @dimir we are running 4.2.0 installed via yum and will look forward to installing 4.2.4 when it is released.

Thank you both for your help and quick response.

Glad to be of help. 4.2.4 packages are planned to be available by the end of this month.

Fixed in:

• pre-4.0.11rc1 9b49cf45405
• pre-4.2.5rc1 5cd02c99f11
• pre-4.4.0alpha1 (trunk) 6668641448c

Do you know in what code does it hang ? Would be nice to connect with gdb and do backtrace