[ZBX-9867] PidFile location in .conf file ignored, can't restart zabbix Created: 2015 Mar 02 Updated: 2019 Sep 03 Resolved: 2019 Jul 22 |
|
Status: | Closed |
Project: | ZABBIX BUGS AND ISSUES |
Component/s: | Agent (G), Proxy (P), Server (S) |
Affects Version/s: | None |
Fix Version/s: | 4.0.11rc1, 4.2.5rc1, 4.4.0alpha1, 4.4 (plan) |
Type: | Problem report | Priority: | Major |
Reporter: | Gene Liverman | Assignee: | Vladislavs Sokurenko |
Resolution: | Fixed | Votes: | 13 |
Labels: | packaging, rhel, systemd | ||
Remaining Estimate: | Not Specified | ||
Time Spent: | Not Specified | ||
Original Estimate: | Not Specified | ||
Environment: |
Red Hat 7, CentOS 7 (all systemd) 2.4.3, 2.4.4 |
Issue Links: |
|
||||||||||||
Team: | Team A | ||||||||||||
Sprint: | Sprint 49 (Feb 2019), Sprint 50 (Mar 2019), Sprint 51 (Apr 2019), Sprint 52 (May 2019), Sprint 53 (Jun 2019), Sprint 54 (Jul 2019) | ||||||||||||
Story Points: | 5 |
Description |
It seems that nothing in /usr/lib/systemd/system/zabbix-agent.service is parsing the [root@giles ~]# systemctl status zabbix-agent zabbix-agent.service - Zabbix Agent Loaded: loaded (/usr/lib/systemd/system/zabbix-agent.service; enabled) Active: active (running) since Fri 2015-02-27 16:26:33 EST; 2min 13s ago Process: 1540 ExecStart=/usr/sbin/zabbix_agentd -c $CONFFILE (code=exited, status=0/SUCCESS) Main PID: 1578 (zabbix_agentd) CGroup: /system.slice/zabbix-agent.service ??1578 /usr/sbin/zabbix_agentd -c /etc/zabbix/zabbix_agentd.conf ??1608 /usr/sbin/zabbix_agentd: collector [idle 1 sec] ??1609 /usr/sbin/zabbix_agentd: listener #1 [waiting for connection] ??1610 /usr/sbin/zabbix_agentd: listener #2 [waiting for connection] ??1611 /usr/sbin/zabbix_agentd: listener #3 [waiting for connection] ??1613 /usr/sbin/zabbix_agentd: active checks #1 [idle 1 sec] ??1615 /usr/sbin/zabbix_agentd: active checks #2 [idle 1 sec] ??1620 /usr/sbin/zabbix_agentd: active checks #3 [idle 1 sec] Feb 27 16:26:33 giles systemd[1]: Starting Zabbix Agent... Feb 27 16:26:33 giles systemd[1]: PID file /run/zabbix/zabbix_agentd.pid not readable (yet?) after start. Feb 27 16:26:33 giles systemd[1]: Started Zabbix Agent. Zabbix Server seems to have the same issue of trying to put the PID under /run. This is installed using the official Zabbix repositories. [root@giles ~]# systemctl status zabbix-server zabbix-server.service - Zabbix Server Loaded: loaded (/usr/lib/systemd/system/zabbix-server.service; enabled) Active: active (running) since Fri 2015-02-27 16:26:33 EST; 2 days ago Main PID: 1566 (zabbix_server) CGroup: /system.slice/zabbix-server.service [trimmed list of processes] Feb 27 16:26:33 giles systemd[1]: Starting Zabbix Server... Feb 27 16:26:33 giles systemd[1]: PID file /run/zabbix/zabbix_server.pid not readable (yet?) after start. Feb 27 16:26:33 giles systemd[1]: Started Zabbix Server. |
Comments |
Comment by Kodai Terashima [ 2015 Mar 04 ] |
On my environment, /var/run is symlink to /run. systemctl status zabbix-agent doesn't show such errors. |
Comment by Gene Liverman [ 2015 Mar 05 ] |
It turns out mine is a symlink too but I am still having the same problem. I don't know if it's related, but I was about to report a second, similar issue where RHEL 6 / CentOS 6 says the agent isn't running when it is. [root@dns1 ~]# ps -eaf |grep zabbix zabbix 20970 1 0 21:35 ? 00:00:00 zabbix_agentd -c /etc/zabbix/zabbix_agentd.conf zabbix 20972 20970 0 21:35 ? 00:00:00 zabbix_agentd: collector [idle 1 sec] zabbix 20973 20970 0 21:35 ? 00:00:00 zabbix_agentd: listener #1 [waiting for connection] zabbix 20974 20970 0 21:35 ? 00:00:00 zabbix_agentd: listener #2 [waiting for connection] zabbix 20975 20970 0 21:35 ? 00:00:00 zabbix_agentd: listener #3 [waiting for connection] zabbix 20976 20970 0 21:35 ? 00:00:00 zabbix_agentd: active checks #1 [idle 1 sec] root 21355 20464 0 21:37 pts/0 00:00:00 grep zabbix [root@dns1 ~]# service zabbix-agent status zabbix_agentd is stopped [root@dns1 ~]# ls -lha /var/run/zabbix/ total 12K drwxr-xr-x. 2 zabbix zabbix 4.0K Mar 4 21:35 . drwxr-xr-x. 14 root root 4.0K Mar 4 04:26 .. -rw-rw-r--. 1 zabbix zabbix 5 Mar 4 21:35 zabbix_agentd.pid Is there something I can do to help troubleshoot this? |
Comment by Gene Liverman [ 2015 Mar 05 ] |
This is from a CenOS 7 host just now: [root@cent7test ~]# systemctl restart zabbix-agent Job for zabbix-agent.service failed. See 'systemctl status zabbix-agent.service' and 'journalctl -xn' for details. [root@cent7test ~]# [root@cent7test ~]# systemctl status zabbix-agent zabbix-agent.service - Zabbix Agent Loaded: loaded (/usr/lib/systemd/system/zabbix-agent.service; enabled) Active: activating (start) since Wed 2015-03-04 21:46:04 EST; 33s ago Process: 9363 ExecStart=/usr/sbin/zabbix_agentd -c $CONFFILE (code=exited, status=0/SUCCESS) Main PID: 26534 (code=exited, status=1/FAILURE) CGroup: /system.slice/zabbix-agent.service ├─26244 /usr/sbin/zabbix_agentd -c /etc/zabbix/zabbix_agentd.conf ├─26245 /usr/sbin/zabbix_agentd: collector [idle 1 sec] ├─26246 /usr/sbin/zabbix_agentd: listener #1 [waiting for connection] ├─26247 /usr/sbin/zabbix_agentd: listener #2 [waiting for connection] ├─26248 /usr/sbin/zabbix_agentd: listener #3 [waiting for connection] └─26249 /usr/sbin/zabbix_agentd: active checks #1 [idle 1 sec] Mar 04 21:46:04 cent7test systemd[1]: zabbix-agent.service holdoff time over, scheduling restart. Mar 04 21:46:04 cent7test systemd[1]: Stopping Zabbix Agent... Mar 04 21:46:04 cent7test systemd[1]: Starting Zabbix Agent... Mar 04 21:46:04 cent7test systemd[1]: PID file /run/zabbix/zabbix_agentd.pid not readable (yet?) after start. [root@cent7test ~]# journalctl -xn -- Logs begin at Mon 2015-02-23 10:21:10 EST, end at Wed 2015-03-04 21:47:34 EST. -- Mar 04 21:46:04 cent7test systemd[1]: Starting Zabbix Agent... -- Subject: Unit zabbix-agent.service has begun with start-up -- Defined-By: systemd -- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel -- -- Unit zabbix-agent.service has begun starting up. Mar 04 21:46:04 cent7test systemd[1]: PID file /run/zabbix/zabbix_agentd.pid not readable (yet?) after start. Mar 04 21:46:55 cent7test sshd[9373]: Connection closed by 127.0.0.1 [preauth] Mar 04 21:47:34 cent7test systemd[1]: zabbix-agent.service operation timed out. Terminating. Mar 04 21:47:34 cent7test systemd[1]: Failed to start Zabbix Agent. -- Subject: Unit zabbix-agent.service has failed -- Defined-By: systemd -- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel -- -- Unit zabbix-agent.service has failed. -- -- The result is failed. Mar 04 21:47:34 cent7test systemd[1]: Unit zabbix-agent.service entered failed state. Mar 04 21:47:34 cent7test systemd[1]: zabbix-agent.service holdoff time over, scheduling restart. Mar 04 21:47:34 cent7test systemd[1]: Stopping Zabbix Agent... -- Subject: Unit zabbix-agent.service has begun shutting down -- Defined-By: systemd -- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel -- -- Unit zabbix-agent.service has begun shutting down. Mar 04 21:47:34 cent7test systemd[1]: Starting Zabbix Agent... -- Subject: Unit zabbix-agent.service has begun with start-up -- Defined-By: systemd -- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel -- -- Unit zabbix-agent.service has begun starting up. Mar 04 21:47:34 cent7test systemd[1]: PID file /run/zabbix/zabbix_agentd.pid not readable (yet?) after start. |
Comment by Kodai Terashima [ 2015 Mar 06 ] |
Thank you for the info. It seems that pid file is exists, but zabbix user cannot access the file. Are you using SELinux? If so, it's necessary to add rule for Zabbix. |
Comment by Gene Liverman [ 2015 Mar 06 ] |
SELinux is not enforcing. |
Comment by islem karoui [ 2015 Mar 13 ] |
Hello, |
Comment by Gene Liverman [ 2015 Mar 13 ] |
I have not solved this. I'm still hoping for an answer here. |
Comment by Kamil Boratyński [ 2015 Mar 29 ] |
Dear all. Problem is still up to date.
[root@ansible ~]# yum info -v zabbix-agent Failed to set locale, defaulting to C Loading "fastestmirror" plugin Config time: 0.009 Yum version: 3.4.3 rpmdb time: 0.000 Setting up Package Sacks Loading mirror speeds from cached hostfile * base: centos.hitme.net.pl * epel: ftp.pbone.net * extras: centos.hitme.net.pl * updates: centos1.hti.pl pkgsack time: 6.468 Installed Packages Name : zabbix-agent Arch : x86_64 Version : 2.2.9 Release : 1.el7 Size : 525 k Repo : installed From repo : zabbix Committer : Kodai Terashima <kodai.terashima@zabbix.com> Committime : Sun Mar 15 13:00:00 2015 Buildtime : Sun Mar 15 09:23:26 2015 Install time: Sun Mar 29 19:02:38 2015 Installed by: root <root> Changed by : root <root> Summary : Zabbix Agent URL : http://www.zabbix.com/ Licence : GPLv2+ Description : The Zabbix client agent, to be installed on monitored systems. [root@ansible ~]# cat /etc/yum yum/ yum.conf yum.repos.d/ [root@ansible ~]# cat /etc/yum.repos.d/zabbix.repo [zabbix] name=Zabbix Official Repository - $basearch baseurl=http://repo.zabbix.com/zabbix/2.2/rhel/7/$basearch/ enabled=1 gpgcheck=1 gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-ZABBIX [root@ansible ~]#
[root@ansible ~]# sestatus SELinux status: enabled SELinuxfs mount: /sys/fs/selinux SELinux root directory: /etc/selinux Loaded policy name: targeted Current mode: permissive Mode from config file: enforcing Policy MLS status: enabled Policy deny_unknown status: allowed Max kernel policy version: 28 [root@ansible ~]#
[root@ansible ~]# systemctl start zabbix-agent Job for zabbix-agent.service failed. See 'systemctl status zabbix-agent.service' and 'journalctl -xn' for details. [root@ansible ~]# systemctl status zabbix-agent zabbix-agent.service - Zabbix Agent Loaded: loaded (/usr/lib/systemd/system/zabbix-agent.service; disabled) Active: failed (Result: start-limit) since Sun 2015-03-29 19:19:23 CEST; 4s ago Process: 9457 ExecStart=/usr/sbin/zabbix_agentd -c $CONFFILE (code=exited, status=0/SUCCESS) Mar 29 19:19:23 ansible.management.bitsa.organisation systemd[1]: zabbix-agent.service never wrote its PID file. Failing. Mar 29 19:19:23 ansible.management.bitsa.organisation systemd[1]: Failed to start Zabbix Agent. Mar 29 19:19:23 ansible.management.bitsa.organisation systemd[1]: Unit zabbix-agent.service entered failed state. Mar 29 19:19:23 ansible.management.bitsa.organisation systemd[1]: zabbix-agent.service holdoff time over, scheduling restart. Mar 29 19:19:23 ansible.management.bitsa.organisation systemd[1]: Stopping Zabbix Agent... Mar 29 19:19:23 ansible.management.bitsa.organisation systemd[1]: Starting Zabbix Agent... Mar 29 19:19:23 ansible.management.bitsa.organisation systemd[1]: zabbix-agent.service start request repeated too quickly, refusing to start. Mar 29 19:19:23 ansible.management.bitsa.organisation systemd[1]: Failed to start Zabbix Agent. Mar 29 19:19:23 ansible.management.bitsa.organisation systemd[1]: Unit zabbix-agent.service entered failed state. [root@ansible ~]#
[root@ansible ~]# zabbix_agent -c /etc/zabbix/zabbix_agentd.conf zabbix_agent [9489]: unknown parameter [PidFile] in config file [/etc/zabbix/zabbix_agentd.conf], line 13 [root@ansible ~]#
[root@ansible ~]# grep -A10 -B10 Pid /etc/zabbix/zabbix_agentd.conf # This is a config file for the Zabbix agent daemon (Unix) # To get more information about Zabbix, visit http://www.zabbix.com ############ GENERAL PARAMETERS ################# ### Option: PidFile # Name of PID file. # # Mandatory: no # Default: # PidFile=/tmp/zabbix_agentd.pid # PidFile=/var/run/zabbix/zabbix_agentd.pid ### Option: LogFile # Name of log file. # If not set, syslog is used. # # Mandatory: no # Default: # LogFile= LogFile=/var/log/zabbix/zabbix_agentd.log [root@ansible ~]#
[root@ansible ~]# zabbix_agent -V Zabbix agent v2.2.9 (revision 52686) (12 March 2015) Compilation time: Mar 15 2015 08:21:46 [root@ansible ~]# Have you got any suggestion? |
Comment by Kamil Boratyński [ 2015 Mar 30 ] |
Workaround: Commenting out PidFile in /etc/zabbix/zabbix_agentd.conf: 5 6 ### Option: PidFile 7 # Name of PID file. 8 # 9 # Mandatory: no 10 # Default: 11 # PidFile=/tmp/zabbix_agentd.pid 12 # 13 #PidFile=/var/run/zabbix/zabbix_agentd.pid And PIDFile in zabbix_agentd.service allows to run zabbix_agentd service properly. [Unit] Description=Zabbix Agent After=syslog.target After=network.target [Service] Environment="CONFFILE=/etc/zabbix/zabbix_agentd.conf" EnvironmentFile=-/etc/sysconfig/zabbix-agent Type=forking Restart=always #PIDFile=/run/zabbix/zabbix_agentd.pid KillMode=process ExecStart=/usr/sbin/zabbix_agentd -c $CONFFILE [Install] WantedBy=multi-user.target Workflow: [root@ansible run]# systemctl status zabbix-agent zabbix-agent.service - Zabbix Agent Loaded: loaded (/usr/lib/systemd/system/zabbix-agent.service; disabled) Active: inactive (dead) Mar 30 09:01:52 ansible.management.bitsa.organisation systemd[1]: zabbix-agent.service holdoff time over, scheduling restart. Mar 30 09:01:52 ansible.management.bitsa.organisation systemd[1]: Stopping Zabbix Agent... Mar 30 09:01:52 ansible.management.bitsa.organisation systemd[1]: Starting Zabbix Agent... Mar 30 09:01:52 ansible.management.bitsa.organisation systemd[1]: zabbix-agent.service start request repeated too quickly, refusing to start. Mar 30 09:01:52 ansible.management.bitsa.organisation systemd[1]: Failed to start Zabbix Agent. Mar 30 09:01:52 ansible.management.bitsa.organisation systemd[1]: Unit zabbix-agent.service entered failed state. Mar 30 09:02:20 ansible.management.bitsa.organisation systemd[1]: Starting Zabbix Agent... Mar 30 09:02:20 ansible.management.bitsa.organisation systemd[1]: Started Zabbix Agent. Mar 30 09:17:32 ansible.management.bitsa.organisation systemd[1]: Stopping Zabbix Agent... Mar 30 09:17:32 ansible.management.bitsa.organisation systemd[1]: Stopped Zabbix Agent. Hint: Some lines were ellipsized, use -l to show in full. [root@ansible run]# systemctl start zabbix-agent [root@ansible run]# systemctl status zabbix-agent zabbix-agent.service - Zabbix Agent Loaded: loaded (/usr/lib/systemd/system/zabbix-agent.service; disabled) Active: active (running) since Mon 2015-03-30 09:17:49 CEST; 4s ago Process: 13836 ExecStart=/usr/sbin/zabbix_agentd -c $CONFFILE (code=exited, status=0/SUCCESS) CGroup: /system.slice/zabbix-agent.service ├─13501 /usr/sbin/zabbix_agentd -c /etc/zabbix/zabbix_agentd.conf ├─13502 /usr/sbin/zabbix_agentd: collector [idle 1 sec] ├─13503 /usr/sbin/zabbix_agentd: listener #1 [waiting for connection] ├─13504 /usr/sbin/zabbix_agentd: listener #2 [waiting for connection] ├─13505 /usr/sbin/zabbix_agentd: listener #3 [waiting for connection] └─13506 /usr/sbin/zabbix_agentd: active checks #1 [idle 1 sec] Mar 30 09:17:49 ansible.management.bitsa.organisation systemd[1]: Starting Zabbix Agent... Mar 30 09:17:49 ansible.management.bitsa.organisation systemd[1]: Started Zabbix Agent. [root@ansible run]# systemctl stop zabbix-agent [root@ansible run]# systemctl status zabbix-agent zabbix-agent.service - Zabbix Agent Loaded: loaded (/usr/lib/systemd/system/zabbix-agent.service; disabled) Active: inactive (dead) Mar 30 09:01:52 ansible.management.bitsa.organisation systemd[1]: Failed to start Zabbix Agent. Mar 30 09:01:52 ansible.management.bitsa.organisation systemd[1]: Unit zabbix-agent.service entered failed state. Mar 30 09:02:20 ansible.management.bitsa.organisation systemd[1]: Starting Zabbix Agent... Mar 30 09:02:20 ansible.management.bitsa.organisation systemd[1]: Started Zabbix Agent. Mar 30 09:17:32 ansible.management.bitsa.organisation systemd[1]: Stopping Zabbix Agent... Mar 30 09:17:32 ansible.management.bitsa.organisation systemd[1]: Stopped Zabbix Agent. Mar 30 09:17:49 ansible.management.bitsa.organisation systemd[1]: Starting Zabbix Agent... Mar 30 09:17:49 ansible.management.bitsa.organisation systemd[1]: Started Zabbix Agent. Mar 30 09:18:00 ansible.management.bitsa.organisation systemd[1]: Stopping Zabbix Agent... Mar 30 09:18:00 ansible.management.bitsa.organisation systemd[1]: Stopped Zabbix Agent. [root@ansible run]# |
Comment by Gene Liverman [ 2015 Apr 30 ] |
The workaround above works for the agent but not the server. With both removed from the agent the pid is located at /tmp/zabbix_agentd.pid. Part of the problem seems to be that when a PID location is defined, no pid file is actually created. |
Comment by Gene Liverman [ 2015 Apr 30 ] |
Also, systemctl status zabbix-server shows the following most, if not all, the time: Apr 30 07:58:12 giles systemd[1]: Starting Zabbix Server... Apr 30 07:58:12 giles systemd[1]: zabbix-server.service start request repeated too quickly, refusing to start. Apr 30 07:58:12 giles systemd[1]: Failed to start Zabbix Server. Apr 30 07:58:12 giles systemd[1]: Unit zabbix-server.service entered failed state. In particular, the "repeated too quickly" part stands out. |
Comment by Vasily Kolomeets [ 2015 May 06 ] |
Also affects zabbix-agent v2.4.5 on CentOS 7. |
Comment by Ionut Bochis [ 2015 May 13 ] |
Also affects zabbix_server on CentOS 7 with selinux disabled. -bash-4.2# zabbix_server -V Zabbix server v2.4.5 (revision 53282) (21 April 2015) Compilation time: Apr 23 2015 03:14:46 CentOS Linux release 7.1.1503 (Core) The start service command fails, but the process starts with pid file errors, after that you can't stop/restart the service. Tried to comment out the pid location in config (which defaults to /tmp), without any luck. -bash-4.2# systemctl start zabbix-server.service Job for zabbix-server.service failed. See 'systemctl status zabbix-server.service' and 'journalctl -xn' for details. zabbix_server [1860]: Is this process already running? Could not lock PID file [/tmp/zabbix_server.pid]: [11] Resource temporarily unavailable zabbix_server [1864]: Is this process already running? Could not lock PID file [/tmp/zabbix_server.pid]: [11] Resource temporarily unavailable zabbix_server [1868]: Is this process already running? Could not lock PID file [/tmp/zabbix_server.pid]: [11] Resource temporarily unavailable zabbix_server [1872]: Is this process already running? Could not lock PID file [/tmp/zabbix_server.pid]: [11] Resource temporarily unavailable zabbix_server [1876]: Is this process already running? Could not lock PID file [/tmp/zabbix_server.pid]: [11] Resource temporarily unavailable The pid file is being created, however it does not seem that Zabbix Server tries to use it when a stop/restart is called. -bash-4.2# ls -la /tmp/zabbix_server.pid -rw-rw-r-- 1 zabbix zabbix 3 May 12 22:13 /tmp/zabbix_server.pid
-bash-4.2# cat zabbix-agent.service
[Unit]
Description=Zabbix Agent
After=syslog.target
After=network.target
[Service]
Environment="CONFFILE=/etc/zabbix/zabbix_agentd.conf"
EnvironmentFile=-/etc/sysconfig/zabbix-agent
Type=forking
Restart=always
PIDFile=/run/zabbix/zabbix_agentd.pid
KillMode=process
ExecStart=/usr/sbin/zabbix_agentd -c $CONFFILE
[Install]
WantedBy=multi-user.target
-bash-4.2# cat zabbix-server.service
[Unit]
Description=Zabbix Server
After=syslog.target
After=network.target
[Service]
Environment="CONFFILE=/etc/zabbix/zabbix_server.conf"
EnvironmentFile=-/etc/sysconfig/zabbix-server
Type=forking
Restart=always
PIDFile=/run/zabbix/zabbix_server.pid
KillMode=process
ExecStart=/usr/sbin/zabbix_server -c $CONFFILE
[Install]
WantedBy=multi-user.target
-bash-4.2# stat /usr/sbin/zabbix_agentd File: ‘/usr/sbin/zabbix_agentd’ Size: 307088 Blocks: 600 IO Block: 4096 regular file Device: 3530b6a1h/892384929d Inode: 531358 Links: 1 Access: (0755/-rwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2015-05-12 20:24:38.629667114 +0100 Modify: 2015-04-23 04:20:59.000000000 +0100 Change: 2015-05-12 20:22:30.377230505 +0100 Birth: - -bash-4.2# stat /usr/sbin/zabbix_server File: ‘/usr/sbin/zabbix_server’ -> ‘/etc/alternatives/zabbix-server’ Size: 31 Blocks: 0 IO Block: 4096 symbolic link Device: 3530b6a1h/892384929d Inode: 531356 Links: 1 Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2015-05-12 20:23:30.509372928 +0100 Modify: 2015-05-12 20:22:29.260209282 +0100 Change: 2015-05-12 20:22:29.260209282 +0100 Birth: - -bash-4.2# stat /etc/alternatives/zabbix-server File: ‘/etc/alternatives/zabbix-server’ -> ‘/usr/sbin/zabbix_server_mysql’ Size: 29 Blocks: 0 IO Block: 4096 symbolic link Device: 3530b6a1h/892384929d Inode: 393640 Links: 1 Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2015-05-12 20:23:30.509372928 +0100 Modify: 2015-05-12 20:22:29.260209282 +0100 Change: 2015-05-12 20:22:29.260209282 +0100 Birth: - -bash-4.2# stat /usr/sbin/zabbix_server_mysql File: ‘/usr/sbin/zabbix_server_mysql’ Size: 1392344 Blocks: 2720 IO Block: 4096 regular file Device: 3530b6a1h/892384929d Inode: 531355 Links: 1 Access: (0755/-rwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2015-05-12 20:23:30.509372928 +0100 Modify: 2015-04-23 04:20:59.000000000 +0100 Change: 2015-05-12 20:22:29.102206280 +0100 Birth: - Zabbix agent can start/stop/restart without issues, with pid file line commented in conf file, or with a custom path. -bash-4.2# zabbix_agent -V Zabbix agent v2.4.5 (revision 53282) (21 April 2015) Compilation time: Apr 23 2015 03:19:15 Please let us know what else can we give you to fix this issue. Thanks |
Comment by Ionut Bochis [ 2015 Jun 11 ] |
I've managed to fix this by downgrading the gnutls packages from latest 3.3.8-12.el7 to: gnutls-3.1.18-10.el7_0.x86_64.rpm gnutls-c++-3.1.18-10.el7_0.x86_64.rpm gnutls-dane-3.1.18-10.el7_0.x86_64.rpm gnutls-devel-3.1.18-10.el7_0.x86_64.rpm from vault repo at http://vault.centos.org/7.0.1406/updates/x86_64/Packages/ Hopefully you can find the cause and apply a fix for zabbix-2.4.5 init scripts. |
Comment by Gene Liverman [ 2015 Jun 11 ] |
As mentioned before, this is also a problem on CentOS 6 / Red Hat 6. Based on the comment above I ran some checks on one of each: This box is fully patched: [root@zaakje gliverma]# cat /etc/redhat-release CentOS release 6.6 (Final) [root@zaakje gliverma]# ps -eaf|grep zabbix zabbix 26437 1 0 Feb25 ? 00:00:00 zabbix_agentd -c /etc/zabbix/zabbix_agentd.conf zabbix 26439 26437 0 Feb25 ? 01:19:38 zabbix_agentd: collector [idle 1 sec] zabbix 26440 26437 0 Feb25 ? 04:24:58 zabbix_agentd: listener #1 [waiting for connection] zabbix 26441 26437 0 Feb25 ? 04:25:10 zabbix_agentd: listener #2 [waiting for connection] zabbix 26442 26437 0 Feb25 ? 04:24:34 zabbix_agentd: listener #3 [waiting for connection] zabbix 26443 26437 0 Feb25 ? 00:06:54 zabbix_agentd: active checks #1 [idle 1 sec] root 28267 28255 0 08:01 pts/0 00:00:00 grep zabbix [root@zaakje gliverma]# service zabbix-agent stop Shutting down Zabbix agent: [FAILED] [root@zaakje gliverma]# ps -eaf|grep zabbix zabbix 26437 1 0 Feb25 ? 00:00:00 zabbix_agentd -c /etc/zabbix/zabbix_agentd.conf zabbix 26439 26437 0 Feb25 ? 01:19:38 zabbix_agentd: collector [idle 1 sec] zabbix 26440 26437 0 Feb25 ? 04:24:58 zabbix_agentd: listener #1 [waiting for connection] zabbix 26441 26437 0 Feb25 ? 04:25:10 zabbix_agentd: listener #2 [waiting for connection] zabbix 26442 26437 0 Feb25 ? 04:24:35 zabbix_agentd: listener #3 [waiting for connection] zabbix 26443 26437 0 Feb25 ? 00:06:54 zabbix_agentd: active checks #1 [idle 1 sec] root 28291 28255 0 08:01 pts/0 00:00:00 grep zabbix [root@zaakje gliverma]# kill 26437 [root@zaakje gliverma]# ps -eaf|grep zabbix root 28296 28255 0 08:01 pts/0 00:00:00 grep zabbix [root@zaakje gliverma]# service zabbix-agent start Starting Zabbix agent: [ OK ] [root@zaakje gliverma]# ps -eaf|grep zabbix zabbix 28313 1 0 08:02 ? 00:00:00 zabbix_agentd -c /etc/zabbix/zabbix_agentd.conf zabbix 28315 28313 0 08:02 ? 00:00:00 zabbix_agentd: collector [idle 1 sec] zabbix 28316 28313 0 08:02 ? 00:00:00 zabbix_agentd: listener #1 [waiting for connection] zabbix 28317 28313 0 08:02 ? 00:00:00 zabbix_agentd: listener #2 [waiting for connection] zabbix 28318 28313 0 08:02 ? 00:00:00 zabbix_agentd: listener #3 [waiting for connection] zabbix 28319 28313 0 08:02 ? 00:00:00 zabbix_agentd: active checks #1 [idle 1 sec] root 28325 28255 0 08:02 pts/0 00:00:00 grep zabbix [root@zaakje gliverma]# rpm -qa|grep gnutls gnutls-2.8.5-14.el6_5.x86_64 This box is not patched due vendor requirements: [root@uwg-dw01 gliverma]# cat /etc/redhat-release Red Hat Enterprise Linux Server release 6.1 (Santiago) [root@uwg-dw01 gliverma]# ps -eaf|grep zabbix zabbix 2729 1 0 Mar10 ? 00:00:00 zabbix_agentd -c /etc/zabbix/zabbix_agentd.conf zabbix 2735 2729 0 Mar10 ? 04:47:02 zabbix_agentd: collector [idle 1 sec] zabbix 2736 2729 0 Mar10 ? 04:14:27 zabbix_agentd: listener #1 [waiting for connection] zabbix 2737 2729 0 Mar10 ? 04:14:57 zabbix_agentd: listener #2 [waiting for connection] zabbix 2738 2729 0 Mar10 ? 04:14:28 zabbix_agentd: listener #3 [waiting for connection] zabbix 2740 2729 0 Mar10 ? 00:05:50 zabbix_agentd: active checks #1 [idle 1 sec] root 26006 25993 0 08:07 pts/1 00:00:00 grep zabbix [root@uwg-dw01 gliverma]# service zabbix-agent stop Shutting down Zabbix agent: [FAILED] [root@uwg-dw01 gliverma]# ps -eaf|grep zabbix zabbix 2729 1 0 Mar10 ? 00:00:00 zabbix_agentd -c /etc/zabbix/zabbix_agentd.conf zabbix 2735 2729 0 Mar10 ? 04:47:02 zabbix_agentd: collector [idle 1 sec] zabbix 2736 2729 0 Mar10 ? 04:14:27 zabbix_agentd: listener #1 [waiting for connection] zabbix 2737 2729 0 Mar10 ? 04:14:57 zabbix_agentd: listener #2 [waiting for connection] zabbix 2738 2729 0 Mar10 ? 04:14:28 zabbix_agentd: listener #3 [waiting for connection] zabbix 2740 2729 0 Mar10 ? 00:05:50 zabbix_agentd: active checks #1 [idle 1 sec] root 26024 25993 0 08:07 pts/1 00:00:00 grep zabbix [root@uwg-dw01 gliverma]# kill 2729 [root@uwg-dw01 gliverma]# ps -eaf|grep zabbix root 26030 25993 0 08:07 pts/1 00:00:00 grep zabbix [root@uwg-dw01 gliverma]# service zabbix-agent start Starting Zabbix agent: [ OK ] [root@uwg-dw01 gliverma]# ps -eaf|grep zabbix zabbix 26065 1 0 08:07 ? 00:00:00 zabbix_agentd -c /etc/zabbix/zabbix_agentd.conf zabbix 26067 26065 0 08:07 ? 00:00:00 zabbix_agentd: collector [idle 1 sec] zabbix 26068 26065 0 08:07 ? 00:00:00 zabbix_agentd: listener #1 [waiting for connection] zabbix 26069 26065 0 08:07 ? 00:00:00 zabbix_agentd: listener #2 [waiting for connection] zabbix 26070 26065 0 08:07 ? 00:00:00 zabbix_agentd: listener #3 [waiting for connection] zabbix 26071 26065 0 08:07 ? 00:00:00 zabbix_agentd: active checks #1 [idle 1 sec] root 26073 25993 0 08:08 pts/1 00:00:00 grep zabbix [root@uwg-dw01 gliverma]# rpm -qa|grep gnutls gnutls-2.8.5-4.el6.x86_64 As you can see, the latter box has gnutls 2.8.5-4 which, according to http://vault.centos.org/6.0/os/x86_64/Packages/, is the oldest one available for CentOS 6. |
Comment by Kodai Terashima [ 2015 Jun 11 ] |
Still I cannot reproduce this this problem,, but is CentOS6 problem related to |
Comment by sysops [ 2015 Jun 23 ] |
Hi, vi /etc/systemd/system/multi-user.target.wants/zabbix-agent.service systemctl daemon-reload now... unccoment pidfile path in zabbix agent conf file:
– |
Comment by Hilton Howie [ 2015 Jun 24 ] |
I have the same issue. After reboot zabbix-agent wont start. SELinux disabled. I have tried setting PID to /tmp/zabbix_agentd.pid but no difference. Im on CentOS7 zabbix_agentd [3239]: cannot create PID file [/var/run/zabbix/zabbix_agentd.pid]: [2] No such file or directory PID file /var/run/zabbix/zabbix_agentd.pid not readable (yet?) after start. Installed Packages |
Comment by sysops [ 2015 Jun 25 ] |
I was incorrect in post before: vi /etc/systemd/system/multi-user.target.wants/zabbix-agent.service systemctl daemon-reload now... COMMENT pidfile path in zabbix agent conf file: #PidFile=/tmp/zabbix_agentd.pid shell> setenforce 0 |
Comment by sysops [ 2015 Jun 25 ] |
if you do not want disable selinux you can do this: semodule -l | grep zabbix semodule -e zabbix getsebool -a | grep zabbix setsebool -P zabbix_can_network on yum provides \*/audit2allow yum install policycoreutils-python Now check log tail -8 /var/log/audit/audit.log and for each row with This was my audit.log, so i create 14 file echo type=AVC msg=audit(1435075094.231:41126): avc: denied { create } for pid=21207 comm="zabbix_agentd" scontext=system_u:system_r:zabbix_agent_t:s0 tcontext=system_u:system_r:zabbix_agent_t:s0 tclass=unix_dgram_socket' > file1 echo 'type=AVC msg=audit(1435075379.703:42670): avc: denied { connect } for pid=21207 comm="zabbix_agentd" scontext=system_u:system_r:zabbix_agent_t:s0 tcontext=system_u:system_r:zabbix_agent_t:s0 tclass=unix_dgram_socket' > file2 echo 'type=AVC msg=audit(1435075645.627:44293): avc: denied { write } for pid=21206 comm="zabbix_agentd" name="log" dev="devtmpfs" ino=1453 scontext=system_u:system_r:zabbix_agent_t:s0 tcontext=system_u:object_r:devlog_t:s0 tclass=sock_file' > file3 echo 'type=AVC msg=audit(1435075728.386:44707): avc: denied { sendto } for pid=21207 comm="zabbix_agentd" path="/dev/log" scontext=system_u:system_r:zabbix_agent_t:s0 tcontext=system_u:system_r:kernel_t:s0 tclass=unix_dgram_socket' > file4 echo 'type=AVC msg=audit(1435075811.172:45035): avc: denied { write } for pid=24416 comm="zabbix_agentd" name="tmp" dev="dm-0" ino=201326721 scontext=system_u:system_r:zabbix_agent_t:s0 tcontext=system_u:object_r:tmp_t:s0 tclass=dir' > file5 echo 'type=AVC msg=audit(1435075901.284:45041): avc: denied { add_name } for pid=24436 comm="zabbix_agentd" name="zabbix_agentd.pid" scontext=system_u:system_r:zabbix_agent_t:s0 tcontext=system_u:object_r:tmp_t:s0 tclass=dir' > file6 echo 'type=AVC msg=audit(1435076171.596:45087): avc: denied { create } for pid=24530 comm="zabbix_agentd" name="zabbix_agentd.pid" scontext=system_u:system_r:zabbix_agent_t:s0 tcontext=system_u:object_r:tmp_t:s0 tclass=file' > file7 echo 'type=AVC msg=audit(1435076261.705:45093): avc: denied { write open } for pid=24553 comm="zabbix_agentd" path="/tmp/zabbix_agentd.pid" dev="dm-0" ino=201432265 scontext=system_u:system_r:zabbix_agent_t:s0 tcontext=system_u:object_r:tmp_t:s0 tclass=file' > file8 echo 'type=AVC msg=audit(1435076442.140:45113): avc: denied { remove_name } for pid=24587 comm="zabbix_agentd" name="zabbix_agentd.pid" dev="dm-0" ino=201430045 scontext=system_u:system_r:zabbix_agent_t:s0 tcontext=system_u:object_r:tmp_t:s0 tclass=dir' > file9 echo 'type=AVC msg=audit(1435076532.703:45140): avc: denied { unlink } for pid=24616 comm="zabbix_agentd" name="zabbix_agentd.pid" dev="dm-0" ino=201432588 scontext=system_u:system_r:zabbix_agent_t:s0 tcontext=system_u:object_r:tmp_t:s0 tclass=file' > file10 echo 'type=AVC msg=audit(1435076653.319:45169): avc: denied { write } for pid=24683 comm="mysql" name="mysql.sock" dev="dm-0" ino=135227527 scontext=system_u:system_r:zabbix_agent_t:s0 tcontext=system_u:object_r:mysqld_var_run_t:s0 tclass=sock_file' > file11 echo 'type=AVC msg=audit(1435077249.690:45198): avc: denied { connectto } for pid=24794 comm="mysql" path="/var/lib/mysql/mysql.sock" scontext=system_u:system_r:zabbix_agent_t:s0 tcontext=system_u:system_r:mysqld_t:s0 tclass=unix_stream_socket' > file12 echo 'type=AVC msg=audit(1435077243.460:45183): avc: denied { read } for pid=24766 comm="mysql" name="my.cnf" dev="dm-0" ino=68033735 scontext=system_u:system_r:zabbix_agent_t:s0 tcontext=system_u:object_r:mysqld_etc_t:s0 tclass=file' > file13 echo 'type=AVC msg=audit(1435077808.003:45227): avc: denied { open } for pid=24934 comm="mysql" path="/etc/my.cnf" dev="dm-0" ino=68033735 scontext=system_u:system_r:zabbix_agent_t:s0 tcontext=system_u:object_r:mysqld_etc_t:s0 tclass=file' > file14 now install policy: audit2allow -i file1 -M zabbix_agentd1 semodule -i zabbix_agentd1.pp audit2allow -i file2 -M zabbix_agentd2 semodule -i zabbix_agentd2.pp audit2allow -i file3 -M zabbix_agentd3 semodule -i zabbix_agentd3.pp audit2allow -i file4 -M zabbix_agentd4 semodule -i zabbix_agentd4.pp audit2allow -i file5 -M zabbix_agentd5 semodule -i zabbix_agentd5.pp audit2allow -i file6 -M zabbix_agentd6 semodule -i zabbix_agentd6.pp audit2allow -i file7 -M zabbix_agentd7 semodule -i zabbix_agentd7.pp audit2allow -i file8 -M zabbix_agentd8 semodule -i zabbix_agentd8.pp audit2allow -i file9 -M zabbix_agentd9 semodule -i zabbix_agentd9.pp audit2allow -i file10 -M zabbix_agentd10 semodule -i zabbix_agentd10.pp audit2allow -i file11 -M zabbix_agentd11 semodule -i zabbix_agentd11.pp audit2allow -i file12 -M zabbix_agentd12 semodule -i zabbix_agentd12.pp audit2allow -i file13 -M zabbix_agentd13 semodule -i zabbix_agentd13.pp audit2allow -i file14 -M zabbix_agentd14 semodule -i zabbix_agentd14.pp |
Comment by Artem [ 2015 Jun 26 ] |
Many thanks for sysops! 1. I installed zabbix-agent from repository |
Comment by Gene Liverman [ 2015 Jun 26 ] |
This does serve as a work around for the agent but does not fix the server. With this setup I can get the server's status but I cannot stop it unless I use kill. |
Comment by Gene Liverman [ 2015 Jun 26 ] |
Also, on CentOS 6, the agent responds like the server on RHEL 7... status works with this change but stop does not. |
Comment by David Conn [ 2015 Jul 30 ] |
We are having the same problem with Zabbix 2.4.5 (both server and agent services) on a fully patched RHEL 7.1 installation. Disabling SELinux simply isn't an option for us. systemctl status zabbix-server.service shows: systemctl status zabbix-agent.service shows: We've enabled the following SELinux booleans: I'm really surprised about this issue for two reasons: |
Comment by sysops [ 2015 Jul 30 ] |
I use Oracle Linux (RHEL 7) with zabbix agent (not server) and I had the same problem sudo tail -f /var/log/audit/audit.log | grep zabbix and if you find any rows then |
Comment by Volker Fröhlich [ 2015 Sep 11 ] |
You are mixing up two separate issues here! The fact that a PID file in /run/zabbix/zabbix_server.pid is not allowed by policy, is due to the fact that the policy expects it in /run/zabbixsrv. If you want your PID file in /run/zabbix, you have to run your own policy package. The fact that you can't disable it in the config, is due to the systemd unit file, which is looking for one. My experience is, if you don't define one, systemd apparently doesn't know which one is the master process. Therefore I left it in for Fedora/EPEL. The comment should be adapted, I know. |
Comment by Volker Fröhlich [ 2015 Sep 23 ] |
Related to https://bugzilla.redhat.com/show_bug.cgi?id=1220392 |
Comment by Thilo Bangert [ 2015 Oct 02 ] |
today i had a similar/related issue. basically zabbix-agent did not want to start. The zabbix-agent log said, it could not create the pidfile and this was due to a missing /run/zabbix directory. to fix this i
If I am not missing something, it may be necessary to include the tmpfiles.d file in the distribution, as the /run/zabbix directory has to be recreated on every boot (as /run is on tmpfs and all its contents are lost on reboot). [bangert@franck system]$ cat /etc/redhat-release |
Comment by Jari Liimatainen [ 2015 Nov 09 ] |
Hi Guys, This problem is related to the fact that /run directory is tmpfs. That is, it is created at every boot. That's why e.g. permissions set by RPM packages will be lost when the server is booted. To fix the issue correctly, zabbix service files (under /usr/lib/systemd/system/) have to create the run directories with root authories before the service is executed. This is how it's done: (This is my modified version of zabbix-agentd.service, but same trick has to be applied to zabbix-server.service file) [Unit] Description=Zabbix Monitor Agent After=syslog.target network.target [Service] Type=simple # Create and chown /var/run/zabbix directory with root authorieis PermissionsStartOnly=true ExecStartPre=-/usr/bin/mkdir -p /var/run/zabbix ExecStartPre=/usr/bin/chown -R zabbix:zabbix /var/run/zabbix ExecStart=/usr/sbin/zabbix-agentd -f User=zabbix Group=zabbix PrivateTmp=yes [Install] WantedBy=multi-user.target Please add this fix to both, server and agent! |
Comment by Jari Liimatainen [ 2015 Nov 09 ] |
This fix should be applied to any service that tries to creat PID file under /var/run/zabbix directory. |
Comment by Thilo Bangert [ 2015 Nov 09 ] |
Hi Jari, i agree with your analysis, however the correct way to fix this is to install a tmpfiles config file - that way systemd will create the run dir upon boot. No need to ExecStartPre.. This is what i have tried to explain in the comment above. Revisiting this just now, i realize that the tmpfiles.d file should not be installed in /etc/tmpfiles.d but in /usr/lib/tmpfiles.d/ See also http://www.freedesktop.org/software/systemd/man/tmpfiles.d.html Thanks |
Comment by Jari Liimatainen [ 2015 Nov 09 ] |
That sounds like a better fix! Can you provide an example, what should the config file look like? |
Comment by Thilo Bangert [ 2015 Nov 09 ] |
This is what i would propose for /usr/lib/tmpfiles.d/zabbix.conf: # PID file directory for zabbix D /run/zabbix 0755 zabbix zabbix The above works for me. |
Comment by Jari Liimatainen [ 2015 Nov 09 ] |
I just noticed that in my fresh installation (done today), zabbix-server service is run as "zabbixs" user! If the group is set to zabbixs, I guess that would solve it? Alternatively, if zabbix-server would use different run directory (/run/zabbixs), that would solve it as well, but currently it seems to create its PID file in the same directory as zabbix-agentd. |
Comment by Thilo Bangert [ 2015 Nov 09 ] |
ah - i see. personally i'd prefer if all components had their own dedicated run directories, as you describe as an alternative. The solution with the zabbixs group is not so nice, since it would require the zabbixs group also to be present on machines that install the client only, and that doesnt make any sense... |
Comment by Jari Liimatainen [ 2015 Nov 09 ] |
Yes, I agree, and actually there seems to be in my installation (opensuse 13.2) separate tmpfile for zabbix-agent and zabbix-server, but there were errors in them and also errors in default configuration files and default service files. After fixing them all manually, everything worked like a charm. Should there be separate bugs related to all these files? |
Comment by Thilo Bangert [ 2015 Nov 09 ] |
Dont know. My take is that these issues are so simple, they should just get fixed.... but this bug has been lingering for way to long already, so you might just as well try and open a/some new issue(s) - perhaps that gets the ball moving... |
Comment by Kodai Terashima [ 2016 Feb 14 ] |
Some different things are mixed in comments. I updated systemd service file in 2.2.12rc1, 2.2.8rc1 and 3.0.0rc1. I hope the fix solve original issue. zalex_ua The change supposedly fixed this message of systemd right after daemon is started: "Zabbix-server.service: Supervising process 28719 which is not our child. We'll most likely not notice when it exits." where PID looks like pid of running main daemon process |
Comment by Anton Samets [ 2016 Feb 15 ] |
Guys, also we suggest to change in systemd file next thing: KillMode=process to KillMode=control-group Today we have found the issue, if you try to restart agent, it doesn't start. And in logs a tons of: 2016-02-15T08:29:30.677018+00:00 sg1-up-a85 zabbix_agentd[51975]: using configuration file: /etc/zabbix_agentd.conf 2016-02-15T08:29:30.677386+00:00 sg1-up-a85 zabbix_agentd[51975]: listener failed: bind() for [[-]:10050] failed: [98] Address already in use 2016-02-15T08:29:30.944696+00:00 sg1-up-a85 zabbix_agentd[51985]: Starting Zabbix Agent [sg1-up-a85]. Zabbix 2.2.9 (revision 52686). 2016-02-15T08:29:30.945185+00:00 sg1-up-a85 zabbix_agentd[51985]: using configuration file: /etc/zabbix_agentd.conf 2016-02-15T08:29:30.945512+00:00 sg1-up-a85 zabbix_agentd[51985]: listener failed: bind() for [[-]:10050] failed: [98] Address already in use 2016-02-15T08:29:31.196747+00:00 sg1-up-a85 zabbix_agentd[52003]: Starting Zabbix Agent [sg1-up-a85]. Zabbix 2.2.9 (revision 52686). 2016-02-15T08:29:31.197314+00:00 sg1-up-a85 zabbix_agentd[52003]: using configuration file: /etc/zabbix_agentd.conf 2016-02-15T08:29:31.197866+00:00 sg1-up-a85 zabbix_agentd[52003]: listener failed: bind() for [[-]:10050] failed: [98] Address already in use 2016-02-15T08:29:31.312681+00:00 sg1-up-a85 zabbix_agentd[52012]: Starting Zabbix Agent [sg1-up-a85]. Zabbix 2.2.9 (revision 52686). 2016-02-15T08:29:31.313097+00:00 sg1-up-a85 zabbix_agentd[52012]: using configuration file: /etc/zabbix_agentd.conf 2016-02-15T08:29:31.313502+00:00 sg1-up-a85 zabbix_agentd[52012]: listener failed: bind() for [[-]:10050] failed: [98] Address already in use Please change this behavior to default. |
Comment by Volker Fröhlich [ 2016 Feb 16 ] |
For 3.0, my approach for Fedora and consequently EPEL7 will be, to run the daemons in the foreground, remove the PidFile directive and set Type to simple. |
Comment by Richard Griffiths [ 2016 Mar 04 ] |
Hi, First post I too had this problem. Latest CentOS Linux release 7.2.1511 (Core) and Zabbix REPO from http://repo.zabbix.com/zabbix/3.0/rhel/7/x86_64/zabbix-release-3.0-1.el7.noarch.rpm I worked around this by editing /etc/zabbix/zabbix_server.conf and switching out the PidFile location to /tmp. i.e.
PidFile=/tmp/zabbix_server.pid
#PidFile=/var/run/zabbix/zabbix_server.pid
I then edited /usr/lib/systemd/system/zabbix-server.service [Service] section to point to the new PIDFile location.
[Service]
Environment="CONFFILE=/etc/zabbix/zabbix_server.conf"
EnvironmentFile=-/etc/sysconfig/zabbix-server
Type=forking
Restart=on-failure
PIDFile=/tmp/zabbix_server.pid
#PIDFile=/run/zabbix/zabbix_server.pid
KillMode=control-group
ExecStart=/usr/sbin/zabbix_server -c $CONFFILE
ExecStop=/bin/kill -SIGTERM $MAINPID
RestartSec=10s
The service now starts and restarts and I can get to the setup page. It all survives a reboot too. Hope this helps someone else! |
Comment by Oleksii Zagorskyi [ 2016 Nov 24 ] |
Answering to anton.samets's suggestion above: zabbix-agent-2.4.7-1.el7.x86_64.rpm - KillMode=process zabbix-agent-2.4.8-1.el7.x86_64.rpm - KillMode=control-group and added: ExecStop=/bin/kill -SIGTERM $MAINPID |
Comment by Adam Smith [ 2017 Apr 06 ] |
I was having this issue in Ubuntu 16.04 recently as well, to get around it there is a config file named zabbix-agent.conf in /usr/lib/tmpfiles.d it seems there is a missing - - at the end of the line adding the zabbix pid folder old line
d /var/run/zabbix 0755 zabbix zabbix
new line
d /var/run/zabbix 0755 zabbix zabbix - -
this has solved my issue. |
Comment by Kodai Terashima [ 2017 Apr 10 ] |
Thank you very much for the pointed it out. I will fix it on 3.0.9 and 2.2.18 |
Comment by Marek Krolikowski [ 2017 Jun 22 ] |
Same problem in Zabbix-Agent 3.2.6-1+jessie root@ntp2:~# /etc/init.d/zabbix-agent status cze 22 18:25:43 ntp2 systemd[1]: Starting Zabbix Agent... root@ntp2:~# ls -al /run/zabbix/zabbix_agentd.pid |
Comment by Harry Venables [ 2017 Jul 10 ] |
We are experiencing the same issue with the zabbix agent. We are running Ubuntu 16.04.1 LTS and zabbix_agentd (daemon) (Zabbix) 3.0.9 zabbix-agent.service - Zabbix Agent Loaded: loaded (/lib/systemd/system/zabbix-agent.service; enabled; vendor preset: enabled) Active: active (running) since Mon 2017-07-10 16:28:12 BST; 1s ago Process: 10197 ExecStop=/bin/kill -SIGTERM $MAINPID (code=exited, status=0/SUCCESS) Process: 10280 ExecStart=/usr/sbin/zabbix_agentd -c $CONFFILE (code=exited, status=0/SUCCESS) Main PID: 10284 (zabbix_agentd) Tasks: 6 Memory: 2.5M CPU: 25ms CGroup: /system.slice/zabbix-agent.service ├─10284 /usr/sbin/zabbix_agentd -c /etc/zabbix/zabbix_agentd.conf ├─10285 /usr/sbin/zabbix_agentd: collector [idle 1 sec] ├─10286 /usr/sbin/zabbix_agentd: listener #1 [waiting for connection ├─10287 /usr/sbin/zabbix_agentd: listener #2 [waiting for connection ├─10288 /usr/sbin/zabbix_agentd: listener #3 [waiting for connection └─10289 /usr/sbin/zabbix_agentd: active checks #1 [idle 1 sec] Jul 10 16:28:12 xvm69256.vps.cloud.tagadab.com systemd[1]: Starting Zabbix Agent... Jul 10 16:28:12 xvm69256.vps.cloud.tagadab.com systemd[1]: zabbix-agent.service: PID file /run/zabbix/zabbix_agentd.pid not readable (yet?) after start: No such file or directory Jul 10 16:28:12 xvm69256.vps.cloud.tagadab.com systemd[1]: Started Zabbix Agent. In the zabbix_agentd.conf the pidfile it set to PidFile=/var/run/zabbix/zabbix_agentd.pid
In the zabbix-agent.service [Unit] Description=Zabbix Agent After=syslog.target After=network.target [Service] Environment="CONFFILE=/etc/zabbix/zabbix_agentd.conf" EnvironmentFile=-/etc/default/zabbix-agent Type=forking Restart=on-failure PIDFile=/run/zabbix/zabbix_agentd.pid KillMode=control-group ExecStart=/usr/sbin/zabbix_agentd -c $CONFFILE ExecStop=/bin/kill -SIGTERM $MAINPID RestartSec=10s [Install] WantedBy=multi-user.target Tried most of the solutions above but cant seem to get zabbix agent to restart |
Comment by Ahmed Mami [ 2017 Jul 28 ] |
Making a Domain Permissive for zabbix agent |
Comment by Anton [ 2017 Dec 30 ] |
[root@test-server~]# systemctl status zabbix-agent ● zabbix-agent.service - Zabbix Agent Loaded: loaded (/usr/lib/systemd/system/zabbix-agent.service; enabled; vendor preset: disabled) Active: active (running) since Fri 2017-12-29 08:51:02 MSK; 1 day 2h ago Process: 3527 ExecStop=/bin/kill -SIGTERM $MAINPID (code=exited, status=0/SUCCESS) Process: 3529 ExecStart=/usr/sbin/zabbix_agentd -c $CONFFILE (code=exited, status=0/SUCCESS) Main PID: 3532 (zabbix_agentd) CGroup: /system.slice/zabbix-agent.service ├─3532 /usr/sbin/zabbix_agentd -c /etc/zabbix/zabbix_agentd.conf ├─3533 /usr/sbin/zabbix_agentd: collector [idle 1 sec] ├─3534 /usr/sbin/zabbix_agentd: listener #1 [waiting for connection] ├─3535 /usr/sbin/zabbix_agentd: listener #2 [waiting for connection] ├─3536 /usr/sbin/zabbix_agentd: listener #3 [waiting for connection] └─3537 /usr/sbin/zabbix_agentd: active checks #1 [idle 1 sec] Dec 29 08:51:02 test-server systemd[1]: Starting Zabbix Agent... Dec 29 08:51:02 test-server systemd[1]: PID file /run/zabbix/zabbix_agentd.pid not readable (yet?) after start. Dec 29 08:51:02 test-server systemd[1]: Started Zabbix Agent. [root@test-server~]# sestatus SELinux status: disabled [root@test-server~]# cat /etc/redhat-release CentOS Linux release 7.4.1708 (Core) [root@test-server~]# [root@test-oneclick-yum1 ~]# yum list zabbix-agent Installed Packages zabbix-agent.x86_64 3.4.4-2.el7 [root@test-server ~]# ls /tmp/zabbix_server.pid ls: cannot access /tmp/zabbix_server.pid: No such file or directory |
Comment by Joao Juvenal Vitorino Junior [ 2018 Apr 30 ] |
Same issue with "Oracle Linux Server release 7.4"
srv01 systemd[1]: Failed to start Zabbix Agent. srv01 systemd[1]: Unit zabbix-agent.service entered failed state. srv01 systemd[1]: zabbix-agent.service failed. srv01 systemd[1]: zabbix-agent.service holdoff time over, scheduling restart. srv01 systemd[1]: Starting Zabbix Agent... srv01 systemd[1]: PID file /run/zabbix/zabbix_agentd.pid not readable (yet?) after start. zabbix_agentd (daemon) (Zabbix) 3.2.11 Revision 76339 27 December 2017, compilation time: Dec 27 2017 15:13:43
SELinux status: disabled Agent only works If I manually create /run/zabbix. |
Comment by Francisco [ 2018 May 06 ] |
Greetings,
I had the same issue and by looking at logs with this command "tail -1 /var/log/zabbix/zabbix_agentd.log" i have observed: i had gathered info on this from: and checked the semaphore array max usage: $>ipcs -s | grep -c "^0x" the last value, 128 is the max limit reached for semaphore arrays, so i needed to increase that, by doing: After this, i needed to check the new sempahore array: and restart zabbix: $> systemctl status zabbix-agent Thanks to everyone for their help |
Comment by Vladislavs Sokurenko [ 2018 Oct 26 ] |
Zabbix can be stopped with SIGINT, when SIGINT is received by main process it will send SIGTERM to child processes. The problem is that child processes will handle signal by calling exit() function which is not signal-safe and calls functions registered with atexit(3) and on_exit(3). This means that exit() might hang sometimes. We need to stop zabbix by calling only signal-safe functions from a signal handler or blocking SIGINT signal in places that might call not signal-safe functions. You can have a look at Some info about signal-safety |
Comment by richlv [ 2018 Oct 27 ] |
Is this related to vso proper investigation should be done, however if pkill zabbix_agent -9 helps then it means that old zabbix could not be properly stopped, that’s why it can’t be started again, you can confirm it by checking that only some processes of zabbix have exited. |
Comment by Anton [ 2019 Mar 21 ] |
When install zabbix-agent simple create /usr/lib/tmpfiles.d/zabbix-agent.conf with content
d /var/run/zabbix 0755 zabbix zabbix - -
And error fixed |
Comment by dimir [ 2019 May 08 ] |
genebean, is this still an issue or it has been fixed in the newer (>=4.0) versions? |
Comment by Vladislavs Sokurenko [ 2019 Jun 17 ] |
Fixed in feature branch: |
Comment by David Drum [ 2019 Jun 17 ] |
This issue is affecting our organization. I cannot access the PR from @vso. When will this be made public? Thanks. |
Comment by Vladislavs Sokurenko [ 2019 Jun 17 ] |
Could you please describe the issue davidd ? |
Comment by dimir [ 2019 Jun 17 ] |
The code is available in development branch. If you would like to access it, it's branch named feature/ https://git.zabbix.com/projects/ZBX/repos/zabbix/commits?until=refs%2Fheads%2Ffeature%2FZBX-9867-4.0 If you are interested in the release which will contain this fix, see "Fix Version/s" in ticket description. |
Comment by David Drum [ 2019 Jun 17 ] |
@Vladislavs Sokurenko our issues match the discussion on this page pretty well. We're running CentOS Linux with SELinux enabled, /var/run is a symlink to /run, same inability to restart, etc. @dimir we are running 4.2.0 installed via yum and will look forward to installing 4.2.4 when it is released. Thank you both for your help and quick response. |
Comment by dimir [ 2019 Jun 17 ] |
Glad to be of help. 4.2.4 packages are planned to be available by the end of this month. |
Comment by Vladislavs Sokurenko [ 2019 Jul 01 ] |
Fixed in:
|
Comment by Vladislavs Sokurenko [ 2019 Jul 02 ] |
Do you know in what code does it hang ? Would be nice to connect with gdb and do backtrace |