[ZBX-12567] Server 3.4.0 doesn't start on CentOS 7 after upgrade from 3.2.7 - Permission denied on /tmp/zabbix_server_preprocessing.sock Created: 2017 Aug 23  Updated: 2019 Feb 03  Resolved: 2017 Sep 07

Status: Closed
Project: ZABBIX BUGS AND ISSUES
Component/s: Documentation (D)
Affects Version/s: 3.4.0
Fix Version/s: None

Type: Problem report Priority: Critical
Reporter: Glenn Matthys Assignee: Unassigned
Resolution: Fixed Votes: 0
Labels: selinux, socket
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

CentOS Linux release 7.3.1611 (Core)


Attachments: File zabbix_server_add.te    
Issue Links:
Duplicate
is duplicated by ZBX-12592 zabbix agent doesn't start with selin... Closed
Team: Team C
Sprint: Sprint 15, Sprint 16
Story Points: 0.5

 Description   

After

service zabbix-server stop
yum localinstall http://repo.zabbix.com/zabbix/3.4/rhel/7/x86_64/zabbix-release-3.4-1.el7.centos.noarch.rpm
yum clean all; yum upgrade
service zabbix-server start

The database is upgraded, but the server fails to continue:

 2661:20170823:081331.612 server #57 started [alert manager #1]
  2661:20170823:081331.612 cannot start alert manager service: Cannot bind socket to "/tmp/zabbix_server_alerter.sock": [13] Permission denied.
  2659:20170823:081331.613 server #56 started [icmp pinger #5]
  2658:20170823:081331.619 server #55 started [icmp pinger #4]
  2662:20170823:081331.622 server #58 started [preprocessing manager #1]
  2662:20170823:081331.622 cannot start preprocessing service: Cannot bind socket to "/tmp/zabbix_server_preprocessing.sock": [13] Permission denied.
  2657:20170823:081331.626 server #54 started [icmp pinger #3]
  1059:20170823:081331.632 One child process died (PID:2661,exitcode/signal:1). Exiting ...
  1059:20170823:081333.671 syncing history data...
  1059:20170823:081333.671 syncing history data done
  1059:20170823:081333.671 syncing trend data...
  1059:20170823:081333.671 syncing trend data done
  1059:20170823:081333.672 Zabbix Server stopped. Zabbix 3.4.0 (revision 71467).
zabbix_server [2673]: [file:'log.c',line:245] lock failed: [22] Invalid argument


 Comments   
Comment by Vladislavs Sokurenko [ 2017 Aug 23 ]

A new configuration parameter SocketDir has been added to Zabbix server and proxy configurations. This parameter points to the directory where internal Zabbix socket files are stored (/tmp by default). Server and proxy use different socket file names so it's safe to use the same SocketDir for server and proxy running on the same system. However, running multiple servers or proxies in the same system will require different SocketDir configurations.

Comment by Glenn Matthys [ 2017 Aug 23 ]

Looks like selinux is blocking the operation:
type=AVC msg=audit(1503468811.620:72): avc: denied

{ create }

for pid=2662 comm="zabbix_server" name="zabbix_server_preprocessing.sock" scontext=system_u:system_r:zabbix_t:s0 tcontext=system_u:object_r:tmp_t:s0 tclass=sock_file

Result from audit2allow:
#!!!! WARNING 'zabbix_t' is not allowed to write or create to tmp_t. Change the label to zabbix_tmp_t.
allow zabbix_t tmp_t:sock_file create;

Comment by Glenn Matthys [ 2017 Aug 23 ]

When the zabbix-server process is in this state, I have to kill it with "kill -9". Running "systemctl zabbix-server stop" hangs forever.

vso yes, this is due to semaphores being stuck, probably failures like that should be handled better.

Comment by Glenn Matthys [ 2017 Aug 23 ]

"Fixed it" by using:

cat /var/log/audit/audit.log | grep zabbix | audit2allow -M zabbix_server
semodule -i zabbix_server.pp

However, after that, the zabbix server still fails to start successfully, even after cleaning out /tmp:
3908:20170823:083810.401 cannot start alert manager service: The file "/tmp/zabbix_server_alerter.sock" is used by another process.
3906:20170823:083810.406 server #55 started icmp pinger #4
3837:20170823:083810.407 One child process died (PID:3908,exitcode/signal:1). Exiting ...

vso please do pgrep zabbix_server to make sure that server is completely stopped before starting it again, also do you run only one server instance on this machine ?

Comment by Glenn Matthys [ 2017 Aug 23 ]

It is completely stopped (checked ps xfau multiple times before starting), yes I am running only one instance.

Comment by Glenn Matthys [ 2017 Aug 23 ]

Did a reboot of the server, same problem. It's selinux blocking the necessary socket operations. Working my way through that now.

Comment by Glenn Matthys [ 2017 Aug 23 ]

Looks like I had a successful start now. I had to allow the following selinux AVC's:

allow zabbix_t self:unix_stream_socket connectto;
allow zabbix_t tmp_t:sock_file unlink;
allow zabbix_t tmp_t:sock_file create;
allow zabbix_t tmp_t:sock_file write;

Makes sense.

vso thanks for the report, we will see what can be done from our side, it could be at least documented.

zalex_ua I can confirm that these rules are enough. I've prepared zabbix_server_add.te file, which contains policies for 2 directories: /tmp and /var/run/zabbix (to be user friendly for older and newest .conf files). Also, a rule for "process setrlimit" included here, which should be included to CentOS 7.4 (as I've understood).
To apply it, download and execute:

yum install policycoreutils-python
checkmodule -M -m -o zabbix_server_add.mod zabbix_server_add.te
semodule_package  -m zabbix_server_add.mod -o zabbix_server_add.pp
semodule -i zabbix_server_add.pp 
Comment by dimir [ 2017 Aug 23 ]

Yes, 3.4.0 introduces IPC using sockets so additional SELinux rules may need to be applied. I guess we need to add this information to upgrade notes.

Comment by dimir [ 2017 Aug 23 ]

Added upgrade notes, wiper could you please check?

Possibly we need to add it also to 3.4 documentation. What would be a good candidate, Appendix->SELinux? I think we have no information regarding SELinux issues currently at all.

wiper updated slightly

Comment by dimir [ 2017 Aug 25 ]

(2) [D] martins-v, please confirm:

https://www.zabbix.com/documentation/3.4/manual/installation/upgrade_notes_340#possible_issues_with_selinux

martins-v Reviewed, some word order/formatting fixes.

martins-v Note that I've also copied this to a new section devoted to issues with SELinux.

<dimir> Great idea! CLOSED

Comment by dimir [ 2017 Sep 06 ]

Fixed by documenting possible issues with SELinux.

Generated at Fri Mar 29 12:33:52 EET 2024 using Jira 9.12.4#9120004-sha1:625303b708afdb767e17cb2838290c41888e9ff0.