[ZBX-10542] zabbix-server can not start on rhel 7.1 Created: 2016 Mar 16  Updated: 2018 Jan 29  Resolved: 2017 Oct 26

Status: Closed
Project: ZABBIX BUGS AND ISSUES
Component/s: Server (S)
Affects Version/s: 3.0.1
Fix Version/s: None

Type: Incident report Priority: Blocker
Reporter: patrik uytterhoeven Assignee: Unassigned
Resolution: Won't fix Votes: 9
Labels: rhel, server
Environment:

rhel 7.1 (not tested on centos)


Issue Links:
Duplicate
is duplicated by ZBX-11631 Zabbix should install a SELinux polic... Closed

 Description   

Zabbix server would not start on RHEL 7.1 had to upgrade to 7.2

[root@sec0011li run]# tail /var/log/zabbix/zabbix_server.log
 22074:20160316:092101.095 Jabber notifications:      YES
 22074:20160316:092101.095 Ez Texting notifications:  YES
 22074:20160316:092101.095 ODBC:                      YES
 22074:20160316:092101.095 SSH2 support:              YES
 22074:20160316:092101.095 IPv6 support:              YES
 22074:20160316:092101.095 TLS support:               YES
 22074:20160316:092101.095 ******************************
 22074:20160316:092101.095 using configuration file: /etc/zabbix/zabbix_server.conf
 22074:20160316:092101.095 cannot set resource limit: [13] Permission denied
 22074:20160316:092101.095 cannot disable core dump, exiting...

This should be fixed or in the docs support for rhel 7 should be replace by 7.2 else people will run into problems



 Comments   
Comment by Aleksandrs Saveljevs [ 2016 Mar 16 ]

This is a continuation of the discussion in ZBX-10086.

Comment by Aleksandrs Saveljevs [ 2016 Mar 16 ]

Patrik, do you know what prevents Zabbix from disabling core dump on RHEL 7.1 and why it is suddenly possible in RHEL 7.2?

Comment by patrik uytterhoeven [ 2016 Mar 16 ]

Nope srry
was updating zabbix at a customer so things had to move on ...
did the upgrade to 7.2 from 7.1 and zabbix would start without a problem

Comment by Aleksandrs Saveljevs [ 2016 Mar 16 ]

Well, in that case, what should we fix or document? It could have been some configuration on RHEL 7.1 that prevented disabling the core dump (i.e. calling setrlimit() with certain parameters) and, since the exact reason is not known, it might have been possible to fix it without doing the upgrade.

The code for disabling core dumps is pretty simple:

int	zbx_coredump_disable(void)
{
	struct rlimit	limit;

	limit.rlim_cur = 0;
	limit.rlim_max = 0;

	if (0 != setrlimit(RLIMIT_CORE, &limit))
	{
		zabbix_log(LOG_LEVEL_WARNING, "cannot set resource limit: %s", zbx_strerror(errno));
		return FAIL;
	}

	return SUCCEED;
}
Comment by patrik uytterhoeven [ 2016 Mar 16 ]

I will check it with a clean rhel 7.1 to see if i can replicate it myself
but a warning in the documentation that you disable core dumps should be placed in the installation guide i think. So it is clear when you install Zabbix that it disables the core dumps

Customers with Redhat instead of Centos could rely on coredumps for support from RHEL

Comment by Andris Mednis [ 2016 Mar 16 ]

Documented at https://www.zabbix.com/documentation/3.0/manual/installation/requirements#supported_platforms .

Comment by patrik uytterhoeven [ 2016 Mar 16 ]

thx

i will do a clean install on 7.1 when i have time and report back on this ticket

Comment by patrik uytterhoeven [ 2016 Mar 16 ]

same issue with agents on some hosts with rhel 7.1

[root@sec0006li secadm]# ulimit -a
core file size (blocks, -c) 0

sysctl -a | grep -i fs.suid_dumpable
fs.suid_dumpable = 0

still agent refuses to start

17091:20160316:150928.330 using configuration file: /etc/zabbix/zabbix_agentd.conf
17091:20160316:150928.330 cannot disable core dump, exiting...

cat /etc/redhat-release
Red Hat Enterprise Linux Server release 7.1 (Maipo)

This IMHO is wrong !
if clients also demand to disable core dumps ...

This is not a choice that should be made by Zabbix
a warning that coredumps can have performance impact ok
but disabling this on the client side imho not done ...

Comment by Andris Mednis [ 2016 Mar 16 ]

Disabling of core dump was added as part of encryption support as a recommended practice ( https://www.securecoding.cert.org/confluence/display/c/MEM06-C.+Ensure+that+sensitive+data+is+not+written+out+to+disk ).

Comment by patrik uytterhoeven [ 2016 Mar 16 ]

ok makes sense as a security implementation
still i can imaging this to be a show stopper

anyway i found the issue it's related to the selinux-policy package
after updating the selinux policy from version selinux-policy-3.13.1-60.el7.noarch

to version selinux-policy-3.13.1-60.el7_2.3.noarch the client starts without problems

Comment by Andris Mednis [ 2016 Mar 16 ]

Added SELinux to note on https://www.zabbix.com/documentation/3.0/manual/installation/requirements#supported_platforms .

Comment by patrik uytterhoeven [ 2016 Mar 16 ]

Thx

can we not add this as an option to the agent/server configuration file ?

this way people who enable core dumps and want to keep dumps enabled when running zabbix have a choice

Comment by Andris Mednis [ 2016 Mar 16 ]

Good idea. You are welcome to create a ZBXNEXT for it and see community feedback, votes for it.

Comment by nikit0ss [ 2016 May 18 ]

i have this problem too:

This occurs immediately after: service zabbix-server start

2661:20160518:204908.832 Starting Zabbix Server. Zabbix 3.0.2 (revision 59540).
2661:20160518:204908.832 ****** Enabled features ******
2661:20160518:204908.832 SNMP monitoring: YES
2661:20160518:204908.832 IPMI monitoring: YES
2661:20160518:204908.832 Web monitoring: YES
2661:20160518:204908.832 VMware monitoring: YES
2661:20160518:204908.832 SMTP authentication: NO
2661:20160518:204908.832 Jabber notifications: YES
2661:20160518:204908.833 Ez Texting notifications: YES
2661:20160518:204908.833 ODBC: YES
2661:20160518:204908.833 SSH2 support: YES
2661:20160518:204908.833 IPv6 support: YES
2661:20160518:204908.833 TLS support: YES
2661:20160518:204908.833 ******************************
2661:20160518:204908.833 using configuration file: /etc/zabbix/zabbix_server.conf
2661:20160518:204908.833 cannot set resource limit: [13] Permission denied
2661:20160518:204908.833 cannot disable core dump, exiting...

CentOS release 6.7 (Final)

Comment by demudrol [ 2016 Jul 25 ]

So how i fixed it:
CentOS release 6.7 (Final)
Got the same error as nikit0ss - cannot disable core dump, exiting...

0. install setroubleshoot
1. grep "SELinux is preventing" /var/log/messages (your AVC message could be in another place)
2. Got message like "SELinux is preventing /usr/sbin/zabbix_server_mysql from using the setrlimit access on a process. For complete SELinux messages. run sealert -l d6e0......)
3. Run sealert -l d6e0......
4. Got message like "SELinux is preventing /usr/sbin/zabbix_server_mysql from using the setrlimit access on a process." and suggest to me run "grep zabbix_server /var/log/audit/audit.log | audit2allow -M mypol", but there is no "zabbix_server" in /var/log/audit/audit.log. A message like "6:ERROR 'syntax error' at token '' on line 6" told me about ot.
5. So i copied Raw Audit Messages like "type=AVC msg=audit(1469438001.181:7890719): avc: denied

{ setrlimit }

for pid=11221 comm="zabbix_server" scontext=unconfined_u:system_r:zabbix_t:s0 tcontext=unconfined_u:system_r:zabbix_t:s0 tclass=process" to /var/log/audit/audit.log.
6. Run again "grep zabbix_server /var/log/audit/audit.log | audit2allow -M mypol" and got my new policy.
7. Installed it with "semodule -i mypol.pp".
8. Zabbix-server start and it's running now. Profit!

If there is any issue in my answer - please tell about it.
Hope this answer will help.

Comment by Stefan Radman [ 2016 Dec 03 ]

I just ran into the exactly same with the zabbix agent (3.0.5 rev 62889) on RHEL 7.3 (no issue on CentOS 7.2).

# tail -3 /var/log/zabbix/zabbix_agentd.log
 35962:20161203:161003.664 using configuration file: /etc/zabbix/zabbix_agentd.conf
 35962:20161203:161003.664 cannot set resource limit: \[13] Permission denied
 35962:20161203:161003.664 cannot disable core dump, exiting...
 # cat /var/log/audit/audit.log | grep zabbix_agentd | grep denied | tail -1
type=AVC msg=audit(1480777894.701:2350): avc:  denied  { setrlimit } for  pid=36120 comm="zabbix_agentd" scontext=system_u:system_r:zabbix_agent_t:s0 tcontext=system_u:system_r:zabbix_agent_t:s0 tclass=process

Solution provided by demudrol for the server also worked for the agent:

 # cat /var/log/audit/audit.log | grep zabbix_agentd | grep denied | audit2allow -M zabbix_agent_setrlimit
******************** IMPORTANT ***********************
To make this policy package active, execute:

semodule -i zabbix_agent_setrlimit.pp

# cat zabbix_agent_setrlimit.te 

module zabbix_agent_setrlimit 1.0;

require {
	type zabbix_agent_t;
	class process setrlimit;
}

#============= zabbix_agent_t ==============
allow zabbix_agent_t self:process setrlimit;
# semodule -i zabbix_agent_setrlimit.pp
# systemctl start zabbix-agent
Comment by Daniel [ 2016 Dec 13 ]

I've just tested new ZBX installation on fresh CentOS 7.3 1611 and got into this same problem.
demudrol's solution is working fine here as well. THX.

Comment by Rob Pickerill [ 2016 Dec 14 ]

I also hit the same issue, and created a custom policy similar to demudrol to fix this to work around the setrlimit sys calls being denied by SELinux. I am using PSK encryption to communicate with a zabbix server so that explains the setrlimit calls.

versions:
CentOS Linux release 7.3.1611 (Core)
zabbix-proxy-mysql-3.2.2-1.el7.x86_64

sealert summary:
SELinux is preventing /usr/sbin/zabbix_proxy_mysql from using the setrlimit access on a process.

zabbix logs:
5705:20161214:163203.022 using configuration file: /etc/zabbix/zabbix_proxy.conf
5705:20161214:163203.022 cannot set resource limit: [13] Permission denied
5705:20161214:163203.022 cannot disable core dump, exiting...

Can I be of assistance to anyone move this forward so that its not a problem for others?

Comment by Norvik Banka [ 2017 Mar 10 ]

Why zabbix packages not include selinux policy?

Comment by Anton Zolotarjov [ 2017 Mar 14 ]

Can you please post the contents of the custom policy you created for the zabbix server?

Comment by Rob Pickerill [ 2017 Mar 14 ]

Hey, yes of course, just change the type to suit zabbix_agent_t or zabbix_t (this is the same as posted by others in thread).

module zabbix_setrlimit 1.0;

require {
	type zabbix_t;
	class process setrlimit;
}

allow zabbix_t self:process setrlimit;

Which provides (includes default policies):

sesearch --allow --source zabbix_t  --class process --target zabbix_t
Found 1 semantic av rules:
   allow zabbix_t zabbix_t : process { fork sigchld sigkill sigstop signull signal getsched setsched setpgid getcap setrlimit } ; 
sesearch --allow --source zabbix_agent_t  --class process --target zabbix_agent_t
Found 1 semantic av rules:
   allow zabbix_agent_t zabbix_agent_t : process { fork sigchld sigkill sigstop signull signal getsched setsched setpgid getcap setrlimit } ; 

This bug has been also posted to Red Hat who maintain the SELinux policies, and looks like its landed in fedora 25 so maybe we will see it sometime soon in EL and Zabbix agent/server can be started with SELinux with default policies
https://bugzilla.redhat.com/show_bug.cgi?id=1323518
https://bugzilla.redhat.com/show_bug.cgi?id=1393332

Comment by Thomas Mueller [ 2017 May 25 ]

Proposed patch to upstream refpolicy: http://oss.tresys.com/pipermail/refpolicy/2017-May/009635.html

Comment by Thomas Mueller [ 2017 May 26 ]

refpolicy setrlimit patch was merged: https://github.com/TresysTechnology/refpolicy-contrib/commit/9fbf1b94fa4e9f6936ea7100f606ac572ed7af95

Comment by dimir [ 2017 Oct 26 ]

Just checked, it's available in RHEL 7.4 selinux-policy-3.13.1-166.el7 . Please try this update and in case the problem is still there re-open the issue.

Closing as "Won't Fix" but actually this is fixed in upstream.

Comment by richlv [ 2018 Jan 22 ]

rhel 7.4, selinux-policy-3.13.1-166.el7_4.7 and zabbix 4.0.0alpha2 packages. still fails the same way.
should this be reopened ?

Comment by dimir [ 2018 Jan 24 ]

Looks like they have fixed it in the agent, but not server/proxy. With default SELinux rules (enforcing), up-todate CentOS 7.4:

$ sesearch --allow --source zabbix_agent_t --target zabbix_agent_t | grep setrlimit
   allow zabbix_agent_t zabbix_agent_t : process { fork sigchld sigkill sigstop signull signal getsched setsched setpgid getcap setrlimit } ; 
$ sesearch --allow --source zabbix_t --target zabbix_t | grep setrlimit
$ 

Not reopening because it's a SELinux issue.

Comment by richlv [ 2018 Jan 24 ]

thank you for checking, appreciated. given that the packages still don't work out of the box, are we aware of any upstream reports to get this finally fixed ?

Comment by dimir [ 2018 Jan 26 ]

What helps is adding setrlimit to zabbix_t domain:

--- policy-rhel-7.4-contrib.patch   2018-01-24 17:01:02.583965693 +0200
+++ policy-rhel-7.4-contrib.patch.new   2018-01-24 17:00:48.332264747 +0200
@@ -119301,7 +119301,7 @@
 +#
 +
 +allow zabbix_domain self:capability { setuid setgid };
-+allow zabbix_domain self:process { setpgid setsched getsched signal_perms };
++allow zabbix_domain self:process { setpgid setsched getsched signal_perms setrlimit };
 +allow zabbix_domain self:fifo_file rw_fifo_file_perms;
 +allow zabbix_domain self:sem create_sem_perms;
 +allow zabbix_domain self:shm create_shm_perms;

Thomas Mueller, could you propose another patch to RH?

Comment by dimir [ 2018 Jan 26 ]

A workaround - download selinux sources, patch, recompile and install the fixed version:

  • $ wget http://vault.centos.org/7.4.1708/updates/Source/SPackages/selinux-policy-3.13.1-166.el7_4.7.src.rpm
  • $ rpm -ivh selinux-policy-3.13.1-166.el7_4.7.src.rpm
  • $ cd ~/rpmbuild/SOURCES
  • apply the patch above
  • $ rpmbuild -ba selinux-policy.spec
  • install dependencies if required
  • wait till the packages are built (it takes looooong time)
  • $ cd ../RPMS/noarch
  • $ sudo rpm -Uvh --force selinux-policy-3.13.1-166.el7.centos.7.noarch.rpm selinux-policy-targeted-3.13.1-166.el7.centos.7.noarch.rpm selinux-policy-devel-3.13.1-166.el7.centos.7.noarch.rpm
  • $ sudo init 6
Generated at Mon Dec 17 06:48:49 EET 2018 using JIRA 7.3.9#73021-sha1:26adfd3d76803e1a04219c5cd05b58dbafff0dbf.