[ZBXNEXT-195] Proposal: Privilege separation for ZABBIX agent Created: 2010 Jan 06  Updated: 2018 Jan 31

Status: Open
Project: ZABBIX FEATURE REQUESTS
Component/s: Agent (G)
Affects Version/s: None
Fix Version/s: None

Type: Change Request Priority: Major
Reporter: J. Fischer Assignee: Unassigned
Resolution: Unresolved Votes: 7
Labels: logmonitoring, patch, privilege, security
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

UNIX-like operating systems


Attachments: File zabbix_agentd-1.8.2-mx.patch     File zabbix_agentd-privsep.patch    
Issue Links:
Causes
caused by ZBXNEXT-4677 Network errors on one host interface ... Closed
Duplicate

 Description   

The following is open for discussion. I would be glad to implement this and send patches
for ZABBIX 1.8 and/or trunk and/or head, because I need the functionality provided by
the implementation of this proposal and don't want to maintain this "out of tree". I also
think that this might be useful for all users of ZABBIX who are monitoring log files.

1. Purpose

ZABBIX agent usually runs as an unprivileged user, which is a good thing to do. Alas,
on UNIX systems, often log files are owned by a dedicated user (most likely, root) and
have strict file permissions on them to prevent other users from reading these files.
This makes it virtually impossible to practically monitor log files with ZABBIX without
a rather large configuration change on all monitored hosts (i.e. touch file ownerships
and permissions for all log files monitored).

Privilege separation for ZABBIX agent aims to provide a simple, effective and proven
mechanism to make log file monitoring within ZABBIX more usable.

2. How it works

Before ZABBIX agent drops it privileges to "zabbix" user, a special process will be
forked and steadily run in the background, much like other agent pollers do. This
process will not drop privileges and continue to run with super-user privileges.

This process will not communicate with the network or other processes than the ZABBIX
agent, and even this communication will be very restricted.

Basically, the privileged process works in the following way:

  • Receive a request from ZABBIX agent to open a given file, i.e. /var/log/messages
  • Verify that the file is in a white list (new configuration parameter necessary)
  • Open the file with O_RDONLY flag to get a file descriptor
  • Pass this file descriptor back to ZABBIX agent
  • Close file descriptor (will stay open in the unprivileged parent process)

It is then the responsibility of the unprivileged ZABBIX agent to work with and close this
file descriptor accordingly, so it will not leak. Reading and processing the data from the
log file will also be handled in the unprivileged process.

The code running as UID 0 is kept as minimal as possible and consists only of a very few
lines.

3. The interface

The communication between the unprivileged Agent and the privileged process will be
performed via a socketpair(2) of type AF_LOCAL.

The ZABBIX agent can use new API functions for file operations, e.g.:

int zbx_priv_open(int priv_fd, const char *path, int mode)
int zbx_priv_stat(int priv_fd, const char *path, struct stat *buf)
int zbx_priv_lstat(int priv_fd, const char *path, struct stat *buf)

The new functions try to mimic the behaviour of the original functions, returning error
codes and setting errno just like the originals. The design goal is to make it possible
to use the new functions as drop-in replacement - the only thing that needs to be taken
care of ist the new first argument, priv_fd, which specifies the file descriptor to use
for communicating with the privileged process.

The changes in existing agent code will be as unintrusive as possible.

4. Whitelist

For security purposes, there should be a new configuration parameter which defines the
file patterns allowed to be openened by the privileged process. This is to prevent a
malicious GUI user to have ZABBIX agent return lines from "secret" files, e.g. /etc/shadow
or private SSL keys.

This pattern could be defined, for example, with:

LogfileWhitelist = /var/log/*

The privileged process will then match the filename given with each request against this
whitelist, using shell globbing patterns (above example would match /var/log/messages and
/var/log/mysql/mysqld.log but not /var/spool/cron/...). Multiple patterns can be given,
delimeted by comma (",").

Maybe a blacklist (LogfileBlacklist) configuration option would be a complimentary option,
to disallow access to certain locations instead of using a white list, at the users
choice.
It is yet to be decided whether it is a good idea to use realpath(3) inside the privileged
code to sanitize the given paths. On one hand, we must prevent that a malicious user can
circumvent the white- or blacklist by using e.g. /var/log/../../etc/shadow as path or using
symbolic links (e.g. /tmp/foo -> /etc/shadow). realpath(3) could be a remedy, but it has
shown security flaws in the path in certain implementations.

We could:

  • Check whether the path is a symbolic link and deny access if it is
  • Check whether the path contains /.. somewhere and deny access if it is

This would be the most simplistic approach, maybe not as good as a check with realpath().

5. Extendibility

The privsep protocol can easily be extended to provide more privileged functions to the
ZABBIX agent in a sane and secure way.

For example, it could be used for data collection that requires root privileges (e.g. some
stuff in /proc etc).

6. Compatibility

The code has been tested on Linux and BSD (Open-, Net- and FreeBSD). It will not work on
non-UNIX-like OS (like Windows), and should work on most UNIX derivates that more or less
comply to the POSIX standard (providing socketpair(2), select(2) and sendmsg(2)).

Alas, I have no access to "big-iron" UNIX-derivates such as HP-UX, AIX or Solaris that
have a compiler environment so I cannot test it on those platforms.

7. Making it optional

7.1 At compile-time

A new option to configure should be introduced, namely --enable-privsep. When not specified,
the privilege separation code will not be compiled and the zbx_priv_*() functions will act
simply as wrappers around zbx_open() etc.

7.2 At run-time

Similar to OpenSSH's sshd, a new configuration item "UsePrivilegeSeperation" should be
introduced which takes a boolean argument to define whether the agent should make use of
the privsep feature (only available if compiled with --enable-privsep, of course).

8. Open issues

8.1 logrt[] items

It will not be possible to use this with items of type logrt[] due to the use of opendir(3).
I currently see no way to privsep opendir(), but one could emulate it using getdirentries(2).
This would imply a rather intrusive change to process_logrt() in logfiles.c of ZABBIX agent.

8.2 Testing

As written above, I only have limited access to more "exotic" platforms (from the OSS point
of view). As such, I can only test the code on Linux (RHEL/Centos, Ubuntu/Debian) and Open-
BSD (probably Free- and NetBSD as well).

Looking forward to any input and constructive criticism.



 Comments   
Comment by J. Fischer [ 2010 Jan 21 ]

I have worked out a prototype for the implementation of privilege separation in ZABBIX agent. The patch for it is attached.
NOTE: The current patch is against the 1.8 version of ZABBIX agent as can be downloaded from ZABBIX website.

To try it out:

  • apply the patch
  • run "reconfigure" (part of GNU autotools), since rebuild of a Makefile.in and configure.in is required
  • run ./configure --enable-agent --enable-privsep
  • build the sources using "make"
  • edit zabbix_agentd.conf and add a PrivWhitelist configuration (see below)
  • configure a log[] item for a file which the "zabbix"-user usually has no read permissions to
  • run the agent

The patch introduces three new options for zabbix_agentd.conf:

PrivEnable=(0|1) - Runtime configuration of privilege separation functionality (Default: 1)

PrivEnforce=(0|1) - Whether to enforce that any call to zbx_priv_open() will be performed via the privileged process. If set to 0, the agent will check whether the user "zabbix" has read permissions to a given file, and if it has, use default zbx_open() on that file. If set to 1, any call to zbx_priv_open() will be send through the privileged process and all whitelist constrainsts (see below) will apply. The default is 0.

PrivWhitelist=<list of patterns> - Patterns to paths that the privileged process is allowed to open. Multiple patterns can be given and are separated using comma. Pattern matching will NOT be performed recursively. That is, a pattern "/var/log/*" would allow access for "/var/log/messages" but not "/var/log/audit/audit.log". This is to prevent path escape through using "/var/log/../../etc/shadow" or similar. The default is no pattern, which disallows access to any file.

Open issues:

  • Only been tested on x86_64 Linux (RHEL 5.4, Ubuntu 9.10) yet
  • Logging in the privileged process is somewhat complicated. For now, the privileged process uses syslog as it cannot use zabbix_open_log() because of shared memory mutexes. The privileged process is forked right after configuration and before the agent daemonizes, opens log file and drops privileges. This could be changed, but as long as I maintain this patch out-of-tree, the changes would be far too intrusive
  • The current place for most of the functionality is "zbxnix" library. Maybe this should be moved over to the agentd's code as it only applies for the agent daemon, anyway (otoh could be used in server's as well as in inetd-style agent too).

Things to note:

  • The privileged process will exit upon communication problems with the rest of the agent (e.g. protocol error)
Comment by J. Fischer [ 2010 Apr 29 ]

Attached is an updated patch which will apply cleanly to 1.8.2 version of ZABBIX agent.

Also, support for HP-UX has been added. Might also compile on other commercial and/or older UNIX systems now.
Tested on RHEL 4 and 5, Ubuntu, HP-UX 11.23.

Comment by richlv [ 2010 May 26 ]

as noted in http://www.zabbix.com/forum/showthread.php?p=63528#post63528, supporting command execution with privsep would also be beneficial

Comment by richlv [ 2014 Dec 12 ]

ZBXNEXT-2638 is similar

Comment by Oleksii Zagorskyi [ 2015 Mar 07 ]

Something related (but to server) mentioned in ZBXNEXT-2735

Comment by Glebs Ivanovskis (Inactive) [ 2018 Jan 31 ]

Just wondering, what is the difference between this proposal (or similar ZBXNEXT-2638) and running two or more agents with different access privileges, firewall rules, etc. and with different configurations (e.g. user parameters, remote commands enabled/disabled, etc.)?

P.S. Speaking of

The code running as UID 0 is kept as minimal as possible and consists only of a very few
lines.

You can't imagine how little code is enough to do terrible security bugs.

Comment by Marc [ 2018 Jan 31 ]

glebs.ivanovskis, using multiple agents is actually no option due to ZBX-8623.

Generated at Thu Jun 19 07:41:36 EEST 2025 using Jira 9.12.4#9120004-sha1:625303b708afdb767e17cb2838290c41888e9ff0.