[ZBXNEXT-1109] support multiple server for active proxy Created: 2012 Feb 07  Updated: 2024 Feb 15

Status: Open
Project: ZABBIX FEATURE REQUESTS
Component/s: Proxy (P)
Affects Version/s: None
Fix Version/s: None

Type: Change Request Priority: Major
Reporter: Kodai Terashima Assignee: Unassigned
Resolution: Unresolved Votes: 44
Labels: highavailability, redundancy
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Duplicate
duplicates ZBXNEXT-584 Support multiple server in active che... Closed

 Description   

Active proxy support only one server at this moment (first one in Server setting in zabbix_proxy.conf).
Multiple server support for active mode proxy is good solution for redundancy of Zabbix server.



 Comments   
Comment by Filipe Paternot [ 2014 Apr 25 ]

I would prefer more for creating multiple zabbix environments (test/production).

This way the proxy could behave like the agent (http://blog.zabbix.com/multiple-servers-for-active-agent-sure/858/) and answer to many zabbix servers at once.

I think we can do something remotely close to it when we set the variable ProxyLocalBuffer on zabbix_proxy.conf, but it would be more like a Macgyver..

Comment by Dmitry Verkhoturov [ 2016 May 26 ]

I'll be able to make a patch here maybe — we're using two Zabbix servers, I already have patch for zabbix_sender.

Comment by zhang [ 2016 Jul 16 ]

@Dmitry, could you please share your patch?

Comment by Dmitry Verkhoturov [ 2016 Jul 18 ]

zhangjun, I did not make it because I still don't use Zabbix proxy and 2.4 is lack of support now — I'll probably make it for 3.0 someday, but can't give you any ETA on that.

Comment by Dmitry Verkhoturov [ 2016 Sep 16 ]

Patches for 2.4 and 3.0 are published in zabbix-patches github repo, feel free to use them: https://github.com/zabbix/zabbix-patches

zhangjun, please take a look.

Comment by Dmitry Verkhoturov [ 2017 Jan 24 ]

Patch updated for 3.2 and available in https://github.com/zabbix/zabbix-patches

Comment by Chris Gleason [ 2017 May 02 ]

Does anyone know if this was ever fixed in trunk? I'm looking through release notes and don't see.

I'm using repo packages to install and can't use patches so I'd like to see if this is do able via an upgrade.

Comment by Dmitry Verkhoturov [ 2017 May 02 ]

Right now it's not in trunk even for 3.4 as far as I know. Maybe it's really in "Won't fix" status for Zabbix SIA.

Comment by Marc [ 2017 May 02 ]

Status = Open and Resolution = Unresolved should actually speak for itself, doesn't it?

Comment by Glebs Ivanovskis (Inactive) [ 2017 Jul 21 ]

Guys, could you clarify what's expected from poor Zabbix proxy?

Agent can be polled in passive mode by multiple servers/proxies without any problems because each check is independent and no state needs to be stored on agent's side. Agent can also work with multiple servers/proxies in active mode, but it's a bit more complicated because there is a list of checks to do from every server/proxy. Current solution is that active checks for every server/proxy run within a single process/thread. Independently. List of checks is stored in memory of "active" process/thread. I don't think that the same is expected from proxy.

Proxy has a much more complicated configuration to manage, that's why it has a database, configuration cache, etc. Making it work with multiple servers independently while storing all configurations in one database and one configuration cache would be really complicated. Creating multiple databases and multiple configuration caches for one proxy instance would have no benefits compared to multiple proxy instances running in parallel (this configuration can be achieved without code modification). And I'm not sure it's required.

What's probably expected is an ability for an active proxy to switch between multiple servers as a configuration source and gathered data sink, so that there is only one server proxy works with at a time. Currently such switch requires editing configuration file and restarting a proxy with inevitable downtime and a risk of data loss. The request is about making this procedure easier, right?

Comment by Glebs Ivanovskis (Inactive) [ 2017 Jul 21 ]

If my guess is correct then (partial) implementation of ZBXNEXT-3162 can probably solve the problem.

Comment by Dmitry Verkhoturov [ 2017 Jul 28 ]

glebs.ivanovskis, running few instances of proxy, all of which will be "ServerActive" for client but each have their own servers is nice. However, could this be documented as official solution for situation where you need active items from more than one server and need to use proxy at the same time? Seems far from obvious for me, however it might be the solution for that ZBXNEXT.

I'll try to write doc myself, sounds like a plan.

Comment by Glebs Ivanovskis (Inactive) [ 2017 Jul 28 ]

It seems that high availability setup is not covered in our documentation. There is unofficial guide, but it is quite old. I'd suggest to create a documentation issue for this. Paskal, will you be so kind to do that? I think you will describe real life needs better than me.

Comment by Dmitry Verkhoturov [ 2017 Jul 31 ]

That's a draft, where should I publish it? Zabbix.org wiki?

Case you have multiply Zabbix servers (each of which gathers data from agents) and need to use proxy with this setup, you need multiply proxies, because each could work only with one server. Instruction below is about setting 2+ proxies on same host.

1. Install zabbix-proxy (zabbix-proxy-sqlite3 package, for example)

2. Adjust /etc/zabbix/zabbix_proxy.conf for Zabbix Server 1, then copy it to /etc/zabbix/zabbix_proxy_2.conf and adjust it for second server.
Points of interest:
Field | value for first server 1 | value for first server 2
Server= | IP_of_server_1 | IP_of_server_2
ListenPort= | 10051 | 10061 # ports must be different for first and second server if you want to use proxy in Active mode (default)
PidFile= | /var/run/zabbix/zabbix_proxy.pid | /var/run/zabbix/zabbix_proxy_2.pid
DBName= | SQLite3_path_1 | SQLite3_path_2
DBSocket= | /tmp/mysql.sock | /tmp/mysql_2.sock

3. Copy /etc/init.d/zabbix-proxy to /etc/init.d/zabbix-proxy-2 and adjust following line:
'NAME=zabbix_proxy' ==> 'NAME=zabbix_proxy_2'
'--exec $DAEMON >/dev/null 2>&1' ==> '--exec "$DAEMON -c $CONFFILE" >/dev/null 2>&1'
And add another variable below NAME:
CONFFILE=/etc/zabbix/zabbix_proxy.conf

4. In case your system use systemd (CentOS, Debian 8+, Ubuntu 15.04+), copy /etc/systemd/system/multi-user.target.wants/zabbix-proxy.service to /etc/systemd/system/multi-user.target.wants/zabbix-proxy-2.service and adjust following lines:

Environment="CONFFILE=/etc/zabbix/zabbix_proxy_2.conf"
PIDFile=/run/zabbix/zabbix_proxy_2.pid

Now you're ready to go, do "service zabbix-proxy start" and "service zabbix-proxy-2 start" to get two proxies up and running, and enable these services to run automatically.

It's not tested so far, I'll make real-life test a bit later, and will create a ZBXNEXT about including it into documentation.
(It seems so taped with blue electrical tape, that it doesn't fit official doc in my opinion, but whatever you say). Also, many things are left outside - for example, zabbix_sender patch (https://github.com/zabbix/zabbix-patches/blob/master/zabbix-3.2/ZBXNEXT-1109/ZBXNEXT-1109.patch) didn't make it to source but is required to send trappers to two servers.
I've rechecked zabbix_sender patch, it works with different ports for two servers - so the only problem is it's unofficial and AFAIK we have no separate ticket for zabbix_sender work with 2+ servers.

Comment by Glebs Ivanovskis (Inactive) [ 2017 Jul 31 ]

I think it is a great start! Good job!

In my opinion it is good enough to go into main Zabbix documentation. Create a ZBX(NEXT), put your suggestions there and martins-v will find a place for it.

Comment by Chris Gleason [ 2017 Jul 31 ]

Just to clarify what Glebs is asking in this comment what I was looking for was the ability to specify two servers in the Server= field in the zabbix_proxy.conf file so I could send requests to a prod and staging server. Currently it just sends to the first server in the field.

I can accomplish this by specifying multiple proxies at the agent level, but would like to keep my zabbix_agent.conf file clean and use the proxy to send to multiple servers if possible.

Comment by Dmitry Verkhoturov [ 2017 Aug 24 ]

Patch for two Server support for zabbix-agent was updated to 3.4 release version and still available by link https://github.com/zabbix/zabbix-patches.

Comment by Dmitry Verkhoturov [ 2017 Oct 30 ]

I am not using proxy and not trying to use proxy for 2 servers, it seems I won't have time to properly test my predictions on how it should work and write a reproducible documentation on that. If someone need proxy to work with 2 servers, please write and test documentation on how to achieve that using my and Gleb's answers.

Comment by Chris Gleason [ 2017 Oct 30 ]

I can test this shortly. I need to upgrade our major zabbix version soon anyways, so I'll give it a whirl.

Comment by Chris Gleason [ 2017 Nov 23 ]

Have not had time to get around to this. I install from packages so I can't use that patch.

I'm testing the 3.4 upgrade now so when I'm done I'll use it to test glebs multiple zabbix_server process.

Comment by Plamen Nenov [ 2017 Nov 23 ]

Dmitry Verkhoturov,
the solution you suggest is great , but what about the download of configurations from these 2 different zabbix servers,
basically if these 2 proxy instances use same DB and if you have different checks on these 2 zabbix servers (which is quite possible if we are talking for PROD and DEV zabbix server),
your 2 proxy instances will end up with 2 different configurations from PROD and DEV zabbix servers
how that is going to work ?

Comment by Dmitry Verkhoturov [ 2017 Dec 09 ]

You should use 2 separate DBs of course. Agent it OK with having different active checks from different servers.

Comment by Alex [ 2017 Dec 09 ]

Hello everyone, what happen when our proxy will have 2 or more different db? For now we try use multi-proxy configuration for collect to master server in datacenter in europe region and reserve local server in US region. Every proxy instance collect from agents same items. If we start update major version without downtime, with updating db scheme, we start run 4 instance proxy and agents will send 4 times more identical data instead of once. The issue of deduplication must be considered. In my opinion simple diff between some fields tables with hosts and items should solve this issue and not cause great difficulties. For solve question different db name, db name must be same as server active name.

Comment by Dmitry Verkhoturov [ 2017 Dec 11 ]

Two zabbix-servers with same configuration in their DB will query same things from zabbix-agent and will get the same answer (in case they do the query at the same time).
But if you need to send something from hosts using zabbix-sender, you need to know full list of servers to run zabbix-sender against one by one.
If zabbix-sender support multiply ServerActive out of the box, there is be no such problem - I already have 2 servers there (ServerActive=a,b in /etc/zabbix/zabbix_agentd.conf) and zabbix-sender sends information to both of them.
This patch support two ServerActive instead of current 1, to do it right it should be reworked to support any amount.
Biggest question is do we need to return bad exit code in case of second (5th, 10th) server send failure? In my opinion, no - second server is failover or developement, so it's OK not to fail on sending data to it.

I propose to add zabbix-sender multi-server sending functionality in the upstream 4.0 version of Zabbix.
There is already a patch for v3.4 for 2 servers setup, if it could be improved by Zabbix team to properly support any amount of servers in "ServerActive=" variable - it will be a very neat feature in 4.0.

Comment by Jarrett Heale [ 2018 Jul 30 ]

This would be extremely useful for when you have multiple sites with local staff as well as an HQ site who also need to monitor all servers. HQ could make sure services at each site were up, and more importantly the local staff could maintain and montior their own servers and Zabbix (and most importantly receive alerts for failed servers/services) even if the link back to HQ was down.

I've asked on the forum to find out if there's another way to do something similar in the meantime, but this enhancement would be the ideal way to configure it, at least in terms of firewall rules.

Comment by Ron Van Dompseler [ 2018 Oct 05 ]

We currently use this solution in our 2.4.8 environment where we have zabbix proxy installed on multiple SMARTS IP Gateways. Each server has 4 or 5 "virrtual zabbix proxies" configured for different interfaces on the servers and it works great.

We are now in the process of doing a major reworking of our infrastructure to 4.0 and running it in parallel with the current environment in order to rework our design of 20+ separate instances down to one master instance with HA and DR. For this particular instance that utilizes the SMARTS GATEWAY for SNMP datacollection we are stuck with certain rules where clients are configure only to accept from specific IPs.

Because we need to use the same IP's in our proxy.conf files (ports can be different) I believe I need to install the 4.0 zabbix-proxy-sqlite3 and create the virtual 4.0 proxies in the same way we did with the old ones. We installed the original from repo so I can't install the new one in the same location. I am not expert in having to compile from source to use a different location... not sure if that is even possible. I could probably just copy the binaries and conf files to a different location  but I am not sure on what should be copied where and what would need to be changed in the service files etc.

 

Any ideas would be greatly appreciated.............

Comment by Vitaly Zhuravlev [ 2019 Jan 18 ]

Paskal,

I propose to add zabbix-sender multi-server sending functionality in the upstream 4.0 version of Zabbix.
There is already a patch for v3.4 for 2 servers setup, if it could be improved by Zabbix team to properly support any amount of servers in "ServerActive=" variable - it will be a very neat feature in 4.0.

There is a ticket for that. Please check ZBX-9794

Comment by G. V. Tjong A Hung [ 2020 Oct 26 ]

For those who care for an update, it seems that zabbix_sender still supports only 1 host.

https://www.zabbix.com/documentation/current/manpages/zabbix_sender

So for now multiple zabbix_proxy servers or just 1 zabbix_server

Comment by richlv [ 2023 Nov 10 ]

How does this relate to ZBXNEXT-6923?

Generated at Tue Mar 19 11:01:49 EET 2024 using Jira 9.12.4#9120004-sha1:625303b708afdb767e17cb2838290c41888e9ff0.