Fixed in development branch: https://git.zabbix.com/projects/ZBX/repos/zabbix/browse?at=refs%2Fheads%2Ffeature%2FZBXNEXT-8724-6.5

PR:

• master: https://git.zabbix.com/projects/ZBX/repos/zabbix/pull-requests/6815/overview

Overview

This introduced 2 new warning messages and 1 error message that Zabbix proxy produces when starting up.

• Passive proxy and ServerPort is specified in zabbix_proxy.conf (warning):

  NOTE: ServerPort parameter is ignored for passive proxy and is also deprecated 

• Active proxy and ServerPort is specified in zabbix_proxy.conf (warning):

  NOTE: ServerPort parameter is deprecated, please specify port in Server parameter (e.g. 127.0.0.1:10052)

• Passive proxy and invalid Server entry in zabbix_proxy.conf (error):

  unexpected Server parameter <VALUE>; for passive proxy, please specify address or list of comma delimited addresses 

Fixed in

• 7.0.0beta1 716fab5812a

Implemented in 

• pre-6.0.18rc1 dc9d5faee7a, b079b7e2fa1
• pre-6.4.3rc1 813180f225f, 0f33adf9096
• pre-7.0.0alpha1 (master) 54dbce78b45, 6da43f9652a, e164f80595b

Documentation updated:

• Introduction > What's new in Zabbix (6.0.18, 6.4.3)
• Installation > Upgrade notes (6.0.18, 6.4.3)
• Appendixes > Process configuration > Zabbix proxy (6.0, 6.4, 7.0)

Zabbix Server; Proxy

MikroTik RouterOS 7.6

The WISP industry would love Zabbix-Proxy to be able to run in Mikrotik Container.  I get the same error when running the container - 

"zabbix_proxy [2]: cannot create locks: cannot allocate shared memory for locks"

I have great news!

Running zabbix proxy 6.0.1 without encryption on MikroTik hAP ac²:

I have problem with new images, so I used 6.0.1. I have problem with encryption too.

/disk/format-drive 0 file-system=ext4 partition-table=yes label=flashdrive
/disk/print
Flags: B - BLOCK-DEVICE; M - MOUNTED; p - PARTITION
Columns: SLOT, MODEL, SERIAL, INTERFACE, SIZE, FREE, FS
#     SLOT        MODEL                SERIAL          INTERFACE                   SIZE            FREE  FS  
0 B   usb1        USB SanDisk 3.2Gen1  05019xxxxx7bd7  USB 2.10 480Mbps  15 376 318 464                 
1 BMp usb1-part1  @512-15'376'318'464                                    15 376 317 952  15 187 513 344  ext4

/interface/veth/add name=veth1 address=10.70.3.2/24 gateway=10.70.3.1
/interface/bridge/add name=dockers
/ip/address/add address=10.70.3.1/24 interface=dockers
/interface/bridge/port add bridge=dockers interface=veth1
/ip/firewall/nat/add chain=srcnat action=masquerade src-address=10.70.3.0/24

/container/envs/add name=zabbixproxy_envs key=ZBX_HOSTNAME value="mikrotik.test.cz"
/container/envs/add name=zabbixproxy_envs key=ZBX_SERVER_HOST value="zabbix.test.net"
/container/envs/add name=zabbixproxy_envs key=ZBX_TIMEOUT value="15"
/container/envs/add name=zabbixproxy_envs key=ZBX_STARTPOLLERS value="2"
/container/envs/add name=zabbixproxy_envs key=ZBX_STARTTRAPPERS value="1"
/container/envs/add name=zabbixproxy_envs key=ZBX_CONFIGFREQUENCY value="30"
/container/envs/add name=zabbixproxy_envs key=ZBX_CACHESIZE value="2M"
/container/envs/add name=zabbixproxy_envs key=ZBX_HISTORYCACHESIZE value="4M"
/container/envs/add name=zabbixproxy_envs key=ZBX_HISTORYINDEXCACHESIZE value="2M"
/container/envs/add name=zabbixproxy_envs key=ZBX_VMWARECACHESIZE value="1M"

/container/config/set registry-url=https://registry-1.docker.io tmpdir=disk3/pull
/container/config/set ram-high=32M

/container/add remote-image="zabbix/zabbix-proxy-sqlite3:alpine-6.0.1" interface=veth1 root-dir="disk3/zabbixproxy" envlist=zabbixproxy_envs hostname="mikrotik.test.cz" logging=yes status=stopped
/container/set 0 logging=yes
/container/start 0

I tried it yesterday and it works on RouterOS 7.13.4.

fuzzytime is exactly how I addressed this, but I second the idea of this being visible in frontend as a built-in check.

Debian 10 Buster amd64
Zabbix 5.0.3

Since there are no recoveries possible on the sqlite file when there is a problem, I do think it should be deleted and re-created when any problem with it occurs. It is always better to have a running proxy then one that is complaining about a corrupt or out of version proxy database.
We are not always in control of the machines running the proxies so sometimes those boxes/vm's get terminated a bit database unfriendly. An Oracle/postgres database can handle that, sqlite not.

So thanks for creating this zbxnext, I really hope that a re-create of the database will be done on any problem.

<ssimonenko>: Hi, Ronald! We have implemented the functionality to drop outdated SQLite3 database. You might test latest master.

Available in versions:

• 6.4.0alpha1 (master): commits 1bd3b03a803

Thanks!

Thank you so much ssimonenko for implementing this feature request!

Documentation updated:

• What's new in 6.4.0
• Upgrade notes for 6.4.0
• Upgrade procedure in 6.4.0

Thank you for kind words, starko.

It is not related to housekeeping but receiving of new configuration where lots of items got deleted:

delete from items where (itemid in (4445395,4445396,4445397,4445398,4445399,4445401,4445402,4445403,4445404,4445405,4445406,4445407,4445408,4445409,4445410,4445411,4

Could you please provide more information on what is happening, are you trying to remove lots of items from Zabbix proxy ?

Perhaps bulk deletion should be implemented to avoid huge queries.

Could be caused by ZBX-13266, see recommended limits

What kind of queries are slow ?

Implemented in:

• 7.0.0alpha3 (master) 55d41231c96
• pre-7.0.0beta2 22783668d31, 9f0d95d1f12

Updated documentation:

• What's new
• Proxy memory buffer
• Proxy configuration parameters (ProxyBufferMode, ProxyMemoryBufferSize, ProxyMemoryBufferAge added)
• Internal items ("zabbix[proxy_buffer,*]" items added)

Please, backport this to 6.0.x versions.

This is good for SNMP polling and agent passive checks. What about Zabbix agent active checks and agent-autoregistration?

Sounds a bit like ZBXNEXT-2963. Zabbix proxy is often advertised as a way to do "distributed monitoring", but it is not really a distributed system if you need a human in the loop to do "distributing". I would replace "proxy" in the Description with "Zabbix server node".

I wish a proxy could also signal the server "I am going down, please assign my work to somebody else" so there is no gap in my data.
Then I can shut proxies down for maintenance without reconfiguring Zabbix first and that will be help even on smaller installations.

PS: I suggest changing

to

Actually, it is already possible to achieve automatic failover from one proxy to another, but you need something like an haproxy between agent and proxies (I use that method since 6/7 years).

The drawback is that you no longer have any control on which proxy you want for your host, it's haproxy that chooses it for you – you can change it from zabbix UI but it will change back after a short while.

And, it only works with hosts with zabbix-agent (and active checks)

Anyway, I look forward to the future native failover function

6. Weight of proxies to choose the best one, for example, with minimal load.

One thing to keep in mind is that proxies might be in different (further away) location, but still be part of the same pool. It should be possible to add our own weight as well, so we can state which proxies out of the pool to prefer. This is to keep things like timeout settings to a minimum per proxy.

This will be important for performance and maintenance/upkeep cost.

Hi, I would like to add some of my inputs/design ideas for this proxy ha feature:

1) Unlike the server ha behaviour (server processes started upon timeout/failover), the proxy processes should always run in all proxy servers of the pool.

2) The agents should always be in active mode and send data to all proxies.

3) The primary proxy must be determined by zabbix server (do whatever logic needed here for electing proxy from pool). Only primary proxy should send collected data to zabbix server, standby proxies should discard received data. LLDs and passive checks should be run by the primary proxy server elected by zabbix server. Active check refreshes will also needs to be handled by primary proxy elected by zabbix server.

4) Zabbix server (frontend) should have option to define timeout for electing new proxy from the pool (example: 10 seconds). All the proxy servers in the pool should store defined timeout old data (example: only discard data older than 10 seconds) and forward this data to zabbix server when it becomes primary proxy. This is to ensure no data loss.

5) For multi DC environments, the zabbix server (frontend) should have an option for defining proxy pools/groups for each DC environment.

6) The frontend needs relevant changes like support for multi DC environments. There should be a facility to segregate and display hosts based on DC environments (do not club all hosts in all environments into one single view). The problems view should be equipped with relevant DC filters as well.

I believe this makes a true proxy ha. Please improvise/add more detail if you feel I missed something.

We are very close to the final design of Proxy HA and load balancing. There are just a few minor technical issues that need to be resolved before v1.0 of the document. We need approximately a week to finalise the design. I will share it here once ready so we all can get a better understanding how it will work. And of course any feedback is highly appreciated!

@Alexei Great news! I will check it as soon as you will publish the first draft
Thanks os much

@Alexei by when this feature is planned to be made live in alpha release

[email protected] , it will be ready when it is ready, then we will release it in one of the alpha releases. The design is about to be finalized now and I think that the functionality will be ready for test use in mid-February.

Looking forward to the launch of this feature

May I ask which specific version implements it?

songxwn, It is coming in 7.0.0. First preview of this functionality will likely be available in 7.0.0beta3.

Looks like it is not mentioned in 7.0.0beta3 release notes. What's the current status, please ?

jvrbicky , this functionality is coming in 7.0.0RC1, most likely at the very beginning of May. 

Great news, Thanks Alexei!

I suppose if turned off on proxy the pre-processing should happen on server.

This could work like this. The values are sent with a new "preprocessed" flag with possible values yes/no. Yes, this means a change in communication protocol.

In the frontend, proxy configuration there will be a "checkbox" showing where should the preprocessing be taking place: server or proxy.

Normal operation

When a proxy or server needs to decide if it should preprocess the collected value or not it follows the logic:

Proxy:

• if "checkbox" set to "proxy"
  • preprocess and set "preprocessed=yes"
• otherwise
  • set "preprocessed=no"

Server:

• if item value has "preprocessed=no"
  • preprocess
• otherwise
  • do not preprocess

As you can see server does not even care about the "checkbox" state.

When the "switch" is toggled

This is a tricky case and there might be a situation when a party (server or proxy) needs to preprocess but hasn't the preprocessing rules in cache yet.

Proxy:

• if "checkbox" set to "proxy" and item has preprocessing rules (cached)
  • preprocess
• otherwise
  • skip preprocessing

Server:

• if item value has "preprocessed=no" and item has preprocessing rules (cached)
  • preprocess
• otherwise
  • skip preprocessing

For server, optionally it could pick up the missing preprocessing rules from the database on demand. This way, even if proxy "forgot" to do it (because of the recent switch) server will perform the job for it. The downside is that this is too much work with potential problems (quoting wiper). And to me, with little benefit.

I believe there is a very rare case when someone starts toggling the switch in a production environment and if still this is the case it is expected to have a couple of unpreprocessed values. We can document this "side effect" of "switching" and user can survive with 1 or 2 unpreprocessed values (rather possible 1, thinking of "instant" config sync in 7.0).

To be decided for server: pick up not cached yet tasks on demand or leave values unpreprocessed

Centos7

Thank you for your report, this will be fixed under ZBX-9084

That seems to be a link to a customer issue, not accessible for the general public.

Zabbix triggers aren't stateless, you cannot process item values out of order without side effects.

1. There is {TRIGGER.VALUE} macro and Recovery expression, you can't calculate trigger value at time t without knowing its value at time t-1.
2. There are trigger dependencies. Dependent trigger shouldn't be calculated at all if a more important trigger is in PROBLEM state.
3. There are trigger tags and correlation.

It's similar to ZBXNEXT-2168
And closed as won't fix ZBXNEXT-409

@Gleb Ivanovskis

I agree that there will be errors in the processing logic of the triggers.
If we set priorities between errors in logic and the actual status of the enterprise system that we are monitoring, the choice is obvious.

I think every user of the ENTERPRISE MONITORING SYSTEM ZABBIX wants to see immediately the current status of the monitoring object, and the metrics that are calculated from the historical data always wait.

We have a big installation of Zabbix and we lose a lot of time to stop the Zabbix server, start and get relevant information when setting up the system or updating. On average, this is 2-3 hours, which is unacceptable for the ENTERPRISE SYSTEM.

It certainly cannot be an universal solution, because it depends on what's being monitored and how. Log monitoring can be really messed up if the data are not sent in historical order.

For example user wants to detect service state by monitoring service up/down records in a log file. If service went down for some time and the values are sent in reverse order, then Zabbix would first receive 'service up' log record, do nothing (as the service was already up), then receive 'service down' log record and generate a problem, which would not go away until the service goes down/up again.

What Glebs and Andris said is a very big hurdle to change this. Event generation (even without the recovery expression) is the first and probably the biggest problem.
If somebody does not have triggers, things become much easier, of course. Perhaps Zabbix could stuff the most recent value somewhere and recalculate delayed events once proxy data is up to date, but that seems way too overcomplicated.

A variation of it could be taking the last n minutes of data from proxies that are behind, and then ingesting the historical values. For historical values, the only events that would be processed would be problem events that would still be active based on more recent data. Still, complicated, and historical data in the end might not match events.

Timur, do you have concrete/specific suggestions on how to resolve this? If your proxies are hours behind, do you care about historical events - or even data?

For log-items there is a maxdelay parameter, which can be used to skip over old data (ignore them) to get sooner to the current data.

Maybe it could be generalized to other situations to give a user ability to choose between impatiently waiting while server catches up and skipping the old data in favour of actual data? E.g. "take only last 5 min of data". Obviously, disabled by default, but available if user decides so.

I agree that there will be errors in the processing logic of the triggers.
If we set priorities between errors in logic and the actual status of the enterprise system that we are monitoring, the choice is obvious.

Potentially wrong results now or correct results a bit later? Hm, I'd probably wait.

We have a big installation of Zabbix and we lose a lot of time to stop the Zabbix server, start and get relevant information when setting up the system or updating.

It is difficult to give you advice without knowing your setup and your priorities. Is it so slow because of poor DB performance? Is it slow because of complex triggers? Do you care about loosing some outdated data?

Of course you need to consider different types of data separately.
How do I present this decision:

Numeric.
Actual data from the proxy server comes as a priority. If we did not have an hour of availability, we form first of all the actual data sending. In based, triggers based on numeric metrics are the state status of the component and are tied to the last function, so there should be no logic errors. Triggers and aitems that use historical data will be in the "unsupported" state until all data is available for calculation. Analyzing the situation on the example of the agent status, we see that the availability trigger will immediately become active and disappear before the actual data arrives. This message misleads administrators until the actual data is received and processed by the trigger. These triggers, which show the status of industrial software, also show incorrect status, which destroys the credibility of the monitoring system.

Text.
Text data can be processed in order, because basically information from log files is additional to the status of the service. Although it is also possible to come up with mechanisms for obtaining relevant data first.

There is another idea that requires additional thought.
Process triggers on Proxy servers, thus we will unload Zabbix-Server, Zabbix-Server will synchronize information between Proxy servers and perform global correlation. If we follow this path, then at the start of Zabbix-Server Proxy servers will immediately throw off the list of current triggers, and load buffered data in the background.

Oh no.

Totally disagree that "numeric" data can be processed out-of-order.

Most of my triggers for "numeric" data use "avg", last(1)-last(N), or "forecast" functions.

Knowing the latest values without knowing the preceding ones will break the trigger evaluation.

If we calculate the average value for 3 minutes, Zabbix will wait for these three values and calculate. Also with other triggers

In this case, we will get close to the actual status. What are we going to mislead their users old data

I am one of the users. Proxy making a TCP connection to the server every second is very undesirable for me.

I am trying to use Zabbix to monitor a battery-powered Raspberry-pi device in a remote location. The plan was to have a proxy and an agent running on the device, so the proxy would gather data over some time and then connect to the Zabbix server to upload that.  Uploading once every 10 minutes or even once an hour would be enough. Sometimes the connection could be unavailable for days, still ok, as long as the proxy can buffer the data.

Unfortunately currently Zabbix proxy would make a new TCP connection to the server every second, keeping my expensive LTE connection always-on. Not even sending or receiving any useful data!

We also have this problem. We have metered connections over satellite (Inmarsat) to all of our customer sites.

Sending every second a keep alive is really unnecessary and causing unwanted traffic. For LAN connections it does not matter, but over costly satellite links it does.

We need the possibility to adjust this value again. Please add it back.

Best regards

Thomas

It could be better if active proxy would keep persistent connection to Zabbix server and Zabbix server would notify Zabbix proxy only when it actually needs to check task.

wiper: It would need to be optional. There might be thousands of proxies.

It could be better if active proxy would keep persistent connection to Zabbix server and Zabbix server would notify Zabbix proxy only when it actually needs to check task.

Not really better for the scenarios of remote monitoring via unreliable/expensive link, that we are talking about here. Zabbix proxy is great for this purpose as it can cache gathered data for long time and send it only in configured (long) intervals and when the link is available. Making connections every half or every few hours for sending batch of data will be cheaper and more reliable than trying to keep a long-running connection alive.

I have done further tests now and got some statistics via netflow.

We are a shipping company and monitor the IT on container ships over satellite link. The links are very expensive.

Traffic from Server to the proxy is about 20 MB in 24 hours. Traffic from Proxy is about 30 MB in 24 hours. This will be around 1.5GB traffic per month.

We really need to lower the interval in order to go forward with the rollout of Zabbix.

wiper, can you point me in a direction in the source code to increase the second interval?

I have read already trough datasender.c but it is not clear for me where the heartbeat is created.

Maybe it is possible to insert a sleep somewhere as a workaround?

Is there any chance that this feature gets implemented again officially?

Dear ThomasHH!

Can you point me in a direction in the source code to increase the second interval?

Simply change the value here if you don't mind recompiling server and proxy every time you need to change interval. Exposing this setting in config file is not too difficult as well.

include/zbxtasks.h

#define ZBX_TASK_UPDATE_FREQUENCY	1

Many thanks,

it took a while to get all the dependencies, but finally i compiled only the proxy with the changes (60 sec) and it works as expected.

I will now look into how to expose it to config file.

i need some hint again to make this value available in the configfile.

i used TaskUpdateFrequency=30 in the config file and made these changes :

i added in proxy.c

int ZBX_TASK_UPDATE_FREQUENCY = 60; 

in function zbx_load_config

{"TaskUpdateFrequency",       &ZBX_TASK_UPDATE_FREQUENCY,     TYPE_INT,            PARM_OPT,   1,          3600},           

in function zbx_set_defaults

if (NULL == ZBX_TASK_UPDATE_FREQUENCY)        ZBX_TASK_UPDATE_FREQUENCY = 60; 

However, the values of ZBX_TASK_UPDATE_FREQUENCY seems not to be set or recognized properly either from config file or from set_default function, as the interval is now faster than a second

Is there anything else, i need to change

Have you removed (or commented out) this?

#define ZBX_TASK_UPDATE_FREQUENCY	1

If code does not compile after this change, fix compilation errors by adding

extern int ZBX_TASK_UPDATE_FREQUENCY;

Overall I think you are on the right track.

Great, Many Thanks!!

it is working now also from config file.

Do you see a possibility to get this change in the official version?

best regards

Thomas

Dear ThomasHH, I am happy to hear that you got it working!

As to including this feature into "official version", please take my further words with a grain of salt, because I am not affiliated with Zabbix SIA. As of today, the only way Zabbix accepts code contributions mentioned on their website is via Agent 2 plugins. De jure Zabbix has never admitted that patches are not welcome (i.e. have very little chance of being accepted), but de facto it is what it is. The usual explanation is that "patches are low quality and need rework anyway", but realistically even I as a former Zabbix employee struggle to get tiniest changes through (see ZBX-15677 for example). Getting everyone you know to register in Zabbix JIRA and vote for this ticket can help, but this is not a guarantee.

If you have a working proof-of-concept, I would still encourage you to attach a patch file to this ticket and/or push it to community maintained patches repo on GitHub. This way other users can discover and use your results.

OK, I see. Then i need to compile the proxy part after each update.

I am not an developer, but i will see if i manage to create an patch file to upload for others.

Anyhow many thanks for your support!

This patch was tested on Zabbix version 6.2.4.

This brings in a new config parameter "TaskUpdateFrequency"

You can use it in the zabbix proxy config file like

TaskUpdateFrequency=60

this will change the heartbeat to every minute instead of every second.

TaskUpdateFrequency.patch

I saw this ticket got created, not sure if it's relevant

ZBXNEXT-7931 - Active proxy heartbeats removed in 6.4 (ignored by server since).

In a similar situation as the poster. Have bandwidth constrained pipes and the non-configurable heartbeat is simply saturating them. I would be more than happy to simply turn off the heartbeat and have the only time my servers check in be when the active check timer goes off. At least that way I have control over the amount of traffic being sent. Even worse is attempting to use TLS, which causes a full TLS session to be built every second. Not sure who possibly thought this was a good idea, but it's a very bad idea in any moderate sized network.

It appears that Zabbix agent 3.0 supports the argument, but that's very old. I'd love to use zabbix but I do not want to be in the game of custom patching my environment for any release. Here's to hoping someone on the zabbix team decides to take it seriously. Until then I guess it's back to nagios for me.

Hi:

   anyone know if the problem is resolved at 6.4/7.0? I am currently use 6.0 with passive proxy. I need to recompile every time to reduce 90% useless network traffic.

Don't know, but for our case, we setup a limit via firewall - not ideal, but so far works reasonably good.

ip6tables -t filter -A output_tun_xyz -p tcp --dport 10051 --tcp-flags SYN,ACK,FIN,RST SYN -m limit --limit 5/hour -j ACCEPT
ip6tables -t filter -A output_tun_xyz -p tcp --dport 10051 --tcp-flags SYN,ACK,FIN,RST SYN -j DROP

New GO zabbix agent should have persistent storage I think, so we shouldn't need satelite proxies anymore, but it would be useful in cases where we do.

First of all, this not being configurable - is there for a reason. We have things that depend on this being sent frequently:

• remote command execution with active proxy
• the Execute now functionality
• determine Agent interface status

The impact on making this configurable should definitely be measured first.

There is too much traffic generated and not used by the server nor the proxy.

I shall be able to configure delay of the remote execution with active proxies and determine Agent interface status.

Is there any possibility in Proxy or Server version 6.4 (or 7.0) to change ZBX_TASK_UPDATE_FREQUENCY without recompiling server/proxy?

Eventually, do you know, how to do that in the Docker containers?

Quoting from here: https://www.zabbix.com/documentation/current/en/manual/appendix/protocols/server_proxy#data-request-1

Note that active proxy will still poll Zabbix server every second for remote command tasks (with an empty proxy data request).

So, no, currently it is not configurable. I think it won't hurt if we allow this parameter to be configured with not too big higher limit:

### Option: TaskUpdateFrequency
#       How often proxy requests/sends task information from/to Zabbix Server in seconds.
#       For a proxy in the passive mode this parameter will be ignored.
#
# Mandatory: no
# Range: 1-15
# Default:
# TaskUpdateFrequency=1

4 requests a minute shouldn't be an issue?

4 request a minute may definitely be an issue for a system that otherwise could get online only once an hour. There is no need for the proxy to connect to the server more often when it is explicitly configured to upload data only once an hour or so.

Agree with longer periods between contact to the server.

Again it should be configurable, if we like to check new configurations from the server or send keepalive.

There are systems, which never be reconfigurable and are not real-time monitored. And what is real time exactly for different systems. It could be 1 sec, but also 15 minutes or 1 hour.

And I still don't understand, why there is no option to configure ZBX_TASK_UPDATE_FREQUENCY.

IMO, dimir has a point, that changing the update frequency does have follow on consequences and they should be understood. Especially if one subsequently asks for support from zabbix.

OTOH, I believe it would be beneficial, if the interval was freely configurable, with a warning note, that it has impact on responsiveness and checks for (1st thing, second ..). Many people do not in fact use or require any of those things and have a non-negotiable requirement for reduced data usage. There will always be innumerable ways to misconfigure a server.

I think it makes sense to allow users to set the option to the value they need, with warning that setting it to too high value might result in delays in task execution and active agent availability in UI.

Yes. Please let uses set the option to the value they need. "1-15" is not enough. please check ZBX-18958.  my environment only use passive proxy. value "60" reduce 90% network traffic. and now I am using value "150"  which is working fine.

Why is this not already implemented. It should not be that different from a normal host added?

I build a proof-of-concept version of zabbix_sender that uses the proxy "history data" call and the TLS part works. I could share a patch file if needed, however the PoC version didn't solve my use-case.

A proxy can only add data to specific hosts and must execute all items. One a hosts is marked as monitored by a proxy the server will not execute checks like icmpping.

So the feature request would require a server change to allow a 'trapper proxy' to be able to write to all hosts without requiring all proxy features.

In summary: I would like to be able to use Zabbix Sender as I currently do but with TLS without hosts specific configuration.

Undefined

Hello again.

I would like to know if it's in the planing for new features.

cloud
cent os 7

The queue is actually not really a queue but just a list of items that should have been updated by now, but haven't for some reason.
Usually this is related to performance issues.

Please refer to https://zabbix.org/wiki/Getting_help for ways of getting help from Zabbix company or the Zabbix community.
I would suggest raising your question on the Zabbix IRC channel first, as there are usually community members available to help you.

Do you agree? wiper

I would seem so.

Require confirmation from sasha

Distantly related. Users are wondering why and what server sends something to passive proxy every second.

some exponential backoff from passive proxies might have been useful... but that's partially offtopic here as well

related task ZBX-12936

Postponed because of low value for end user.

This was eventually implemented as ZBXNEXT-7931. Closing.

Could you please link to the documentation page that seems suspicious for you?

For instance, http://www.zabbix.com/documentation/2.0/manual/appendix/config/zabbix_proxy says the following about "Server" configuration parameter, which looks alright to me:

"IP address (or hostname) of Zabbix server. Active Proxy will get configuration data from the server. For a proxy in the passive mode this parameter will be ignored."

as i understood this, issue is about passive proxy working same as other daemons and only accepting connections from addresses, specified in the "Server" parameter.
i suppose documentation was only mentioned because it seemed strange to ignore that parameter.

Correct,
As stated in the description, I think the proxy should use the Server parameter in passive mode the same way the agent does , by filtering by IP the servers allowed to connect.
If this is already the case, the documentation should be fixed...

Thanks! Currently, passive proxy does not check server IP address. Reopening.

ZBXNEXT-866 asks to print out ip of the server we got the config from

ZBXNEXT-1486 is related.

Fixed in:

• 4.0.0alpha1 (trunk) r73528

I think this is the issue that CVE is about: https://talosintelligence.com/vulnerability_reports/TALOS-2017-0327

Do you consider backporting this?

vso please consider using encryption, there is no vulnerability in that case. As far as I know it is not planned to backport because it requires database changes.

interestingly enough, neither this issue, nor the spec actually talk about limiting what active proxies can request from the server, but the changelog has :

A.F....PS. ZBXNEXT-4181 fixed Zabbix server to accept active Zabbix proxy requests only from allowed address if specified (Sasha, vso)

(1) server limiting active proxies is not mentioned in the upgrade notes.

this is important, as it will break things for people who use scripts on the server to send data for hosts, monitored by a proxy.

...but it looks like this is optional and documented in whatsnew instead.

the development is confusing and does not match the specification.

vso this was old specification, removed comment to avoid confussion, Won't Fix

<richlv> There's a specification link in the issue description. Is that the specification that was used?

What about the new active proxy limitation missing from the upgrade notes?

vso new active proxy limitation is optional, so it's missing from upgrade notes as nothing to worry about. No this specification was not used.

<richlv> Thank you for the answer. Previously the Server parameter was ignored, thus for somebody who had it specified an upgrade would change the behaviour. Is that correct?
Also, could it please be possible to modify the description to say that the linked specification was not used in the end?

vso I yes, please see following link:
https://www.zabbix.com/documentation/4.0/manual/introduction/whatsnew400#more_secure_connections_for_proxies
I have also deleted obsolete spec from issue description.

<richlv> Thank you for the description update.
I was also wrong the on the update topic, the field was not there in the older versions of Zabbix.

CLOSED

I like this idea.

As per Oleg presentation, LLD rules are processed directly into DB (attached, also here: https://assets.zabbix.com/files/zabsummit2018/Oleg_Ivanivskyi-Zabbix_Internals.pdf).

It would be even better than initial request if LLD rules were processed into `configuration cache`, and then `configuration syncer` would write changes only it into the database. This way we can easy off some of the DB pressure and also speed up the LLD update process, since proxies would (depending on sync times) have items sooner.

The MVP is indeed what the ticket asks: we need is to sync LLD data in a asynchronous way (without blocking the whole proxy<=>server communication until it ends).

I am sorry but it might be implemented already under ZBXNEXT-4087 is the feature still actual ?

https://www.zabbix.com/documentation/4.2/manual/introduction/whatsnew420#preprocessing_and_custom_macros_in_low-level_discovery

Separate processing for low-level discovery

Processing low-level discovery rules has been split from data gathering processes into its own processing, consisting of a configurable number of low-level discovery workers and a low-level discovery manager. This change greatly reduces low-level discovery impact on data gathering (poller, trapper processes) and proxy communications (trapper, proxypoller processes). Before data gathering could be delayed because the corresponding data gathering processes were busy with low-level discovery.

LLD worker and LLD manager process was introduced and it will handle LLD in background, to simplify, for example Zabbix proxy sends data to server and disconnects, then LLD worker on Zabbix server performs discovery.

This feature request was implemented as part of ZBXNEXT-4087 in pre-4.2.0alpha4, closing as duplicate

Any

Looks like a duplicate of ZBXNEXT-3497.

More on this https://www.zabbix.com/forum/zabbix-suggestions-and-feedback/12275-feature-request-howto-trigger-on-last-seen-interval-of-proxies

So with changes that hosts behind downed proxies won't alert, we need this now more than ever.   We have no way of knowing when a proxy is no longer functioning.   

bunkzilla I agree with you, that this should be part of the official Zabbix proxy template, but what hinders you from adding that item/trigger yourself? Since the addition of nodata being sensitive to proxies, we add this to our proxy template, which tells us reliably if a Zabbix proxy is down.

@starko could you describe with more details how you handle this problem (which item/trigger did you add to the official template)?

Faust Yes, no problem, this is our config:

Item

Trigger

@starko Very much appreciated, thanks!

Great idea! On our side, we use this trigger on the "internal" proxy monitoring (the remote contains {HOST.HOST}) :

So, i might be a bit ignorant here, but.... https://git.zabbix.com/projects/ZBX/repos/zabbix/browse/templates/app/zabbix_server it's there, in this template?
What are you guys exactly missing at this point?

Hey! I missed this way. Maybe the best way (you cannot forget a proxy)

Brian, great find, but your approach is since 6.4 only, and by looking at the template it has a completely different approach in proxy monitoring. Proxies are not created as hosts but as items on Zabbix server host. It's not the same if you have a proxy where both physical (e.g. Linux) metrics are monitored together with Zabbix internal stats.

ingus.vilnis I agree the approach is different, yet sufficient.
With the default template you get a heads-up if there is a problem with the proxy, regarding it's availability:

and:

So that makes this ZBXNEXT obsolete i think.

I do see another issue though, as this will not work:

(new host is not assigned to the proxy itself) so that remains manual work, or some custom scripting.

Thanks Brian! Indeed this seems to be a perfectly valid approach but as Ingus said, it's only in 6.4 (since commit 3b13bad9ef547293a69697c99547d5c2b4797116 if I am correct). So, I am not sure if this could be backported to 5.0 or 6.0 templates? But we need a solution for those LTS version too.

Faustin, for all other versions set it up as Marco has brilliantly displayed. It works perfectly. 

Yep that's already in prod on my zabbix instances

Initially it was discussed in ZBXNEXT-3055.

Zabbix 3.0
Zabbix Proxy 3.0
Centos 6
Mysql 5.1

I can confirm it for latest rev 55626

Interesting point is that possible to remove a host but not possible to add.

seems to be a duplicate of ZBX-9883, closing

this sounds like monitoring some items on proxies, and some on the server.

are those items needed on all proxied hosts ? if only on few, maybe a separate host can be created only for that purpose ?

Reminds me distantly of ZBXNEXT-1285.

@richlv
A separated host is a Workaround that provides me successfull checks, but I cannot use Triggers/Graphs that work with items from one host and another within templates. I have to manually created everything or build a Framework behind zabbix with the API. As we are talking about 30 of our 90 Hosts and our important and fast increasing type of devices this will become a huge mess for us.

@Marc
Yes saw this, but did not completely think this will solve our issue. Also found ZBXNEXT-1411 but in our Environment we ware working with 100% of templated items and so I cannot differ between two SNMP-Interfaces. If ZBXNEXT-1411 is resolved I could add an Agent-Interface facing the Server to each device and run These scripts as remote-commands. Unfortunately without the Option to disable the Proxy for a specific Interface or item I did not find any Approach to solve my issue.

well, the only option that i can think of (and hold on, it's very hackish still) is to have templated trapper or active agent items, then run the script on the server and send in the values faking the proxy protocol. yes, i have done that for customers when there was no other, simpler option

Hi Rich,
yes I'm doing it in this way. I have written a script that collects all Trapper items from database and use this in a lld-discovery on the Server. I'm adding a [<interval>] to the key of the element. Then I create an external check with a small script, that reads and explodes the command line from the original item->key_ and then executing this script and sends the data to the Proxy. It's working fine by now, but still would be nice to have the Feature implemented in the future.

it might be a bit more complicated with active proxies - we might not want to send empty config to them, as the next refresh rate could be in an hour or more. would be great to have them keep the config but stop collecting/sending in the values, and check that status every heartbeat cycle...

Could you please provide a use case where that would be useful?

This would allow for monitoring from multiple locations/datacenters for a particular host within the same Zabbix interface, without having to utilise multiple hosts/Zabbix instances. Additionally, this would also allow for the flexibility to perform passive checks from particular proxies for particular items, such as grouping SNMP checks. This would allow for us to implement SNMP monitoring for thousands of servers without the requirement for updating the SNMP service on each host with an additional monitoring IP address.

I'm thinking about new item keys for internal monitoring like zabbix[version] similarly a we have currently agent.version for agent.
For proxies we could extend existing key with additional value for 3rd param - zabbix[proxy,<name>,version].
But the same key for zabbix and proxy - looks better desition for me.

Then compare received values with regexp() trigger function for the same/several global regexp.

Note that we support already such a key for java proxies - zabbix[java,,version]

https://www.zabbix.com/documentation/2.4/manual/config/items/itemtypes/internal

cross reference - ZBXNEXT-2805

Could be (also) nice to have a "Version" column in Administration -> Proxies.

I've found workaround and my URL is now:

https://{HOST.HOST}{$LONG_URL_PART1}{$LONG_URL_PART2}{$LONG_URL_PART3}

As we discussed on ZBX-15816 we suggest to extend the User Macro characters to something like 1024 or more because is very useful for "HTTP Agent" Check when we need to use Token.

I have tested the latest 4.4alpha3 but the size of MACRO input box is still 256, please extend to 1024 characters.
Thanks

Any updates in scope of this issue ? 

I have same problem with official template for windows i need more space for all services. Vote +1

I have a problem with the official Windows template: it is not possible to specify all the necessary exceptions for autodiscovering services...

I do that since 3.0 with a Global Regular Expressions and never had any problems. I have over 50 exceptions or so.

Hi Marco,

but this is not nice solution all in one. You need import regular and template each.

Tom

Hi Marco,

I can easily use a workaround, but I want to have a solution to the problem that will remove the stupid limitation and allow me to simply use the standard templates without changes.

Quote from the Template Guideline (https://www.zabbix.com/documentation/guidelines/thosts):

1.1.4 Avoid global macros

If user macros are used, define them in the template itself instead of using global macros - that way users get either the default values or an example of what the macro names are. If global macros are used, they are not exported along with the template.

1.1.5 Avoid global regexes

Avoid using global regexes in templates if possible, as they are not exported with the template. If global regex is used, document in the README what global regex with what values must be used with the template.

Available in versions:

• pre-5.2.0alpha2 1eff2fdfbb

Documentation updated:

• What's new in 5.2
• User macros

will it applied to 5.0.xx version?

We are not changing DB schema for existing version.

Large scale enviroment where there needs to be seperation between core infurstucture and services monitored via proxy.

I can echo these concerns (the confusion of knowing assignment is not obvious) and I believe there are some fundamental items that cannot be reported via the zabbix-proxy itself to maintain proxy monitoring integrity.

Example: (ref: ZBX-6036) The proxy sqlite db is corrupt and there is a need for an internal metric (perhaps a PRAGMA integrity_check) - by the nature of this type of problem the proxy cannot forward metrics to the server. Another symptom detection would be to use the NVPS from the proxy; however, the metrics again flow through the proxy itself, which means the server never recieves updated metrics.

I believe some of this was forseen in the original spec: ZBXNEXT-1098 - https://www.zabbix.org/wiki/Docs/specs/ZBXNEXT-1098

Possibility of sending internal metrics as part of heart beat messages

Additionally, the only access to internal metrics (proxy or server) are through internal items. It would make sense to expose these in a log or a process interface such that they could be queried via multiple means. xref: ZBXNEXT-401

ZBXNEXT-2321 introduces support for low-level discovery through ODBC SQL queries and https://www.zabbix.com/documentation/3.0/manual/discovery/low_level_discovery#discovery_using_odbc_sql_queries documents a way to use that feature to solve this problem.

UnixODBC

A slightly related issue might be ZBXNEXT-2200, which talks about having a configuration parameter which specifies how many pollers may access a given host at any given time.

I can also think of pooling SQL queries to be issued over a single TCP connection.

Btw, for now, when it comes to serious database monitoring, which means doing significantly more than a couple of queries, then DBforBIX could be your (and probably your DBAs) friend.
You loose some flexibility and comfort since it works autonomous and makes use of Zabbix trapper protocol. On the other hand it allows to avoid putting credentials into Zabbix or on Zabbix server/proxy host which then could, under certain circumstances, be exposed pretty easy (ZBX-8464, ZBX-4916, ...)
Depending on implementation ZBXNEXT-1660 might help but is not expected to happen sooner or later

Marc, thanks but DBforBIX is terrible, yes as you mention for its trapper type, and the main thing - it is difficult to do LLD, we have hundreds of tables and jobs, and I don't want to generate such config for DBforBIX.
ODBC persistence should be realized right in zabbix, I think it is good way. What about securing connection - it is not problem you may do this in different way: vpn, ssh and so on.

I see. My concerns regarding security are not about the connectivity, since there are as already mentioned by you ways to work around that.
Anyhow, I totally agree that improving Zabbix its database monitoring capabilities is the way to go The recently added ODBC discovery function is a first step.

Since I don't believe there will be session pooling for any connection type in the future (everything is about trapping or polling but not about persistent connections), Zabbix can't reach efficiency of applications like DBforBIX in that domain.

Although that might possibly not be necessary, even when planning to monitor thousands of database (instances).
Pooling of queries and querying them in a bulk should do. At least I hope so, otherwise we're still bound to 3rd party products like DBforBIX or Oracle Enterprise Manager, what is not my preference

Let's wait for developers thoughts

Any comment from developers? Can be this feature planned to 2.5 version?

ZBXNEXT-2902 asks for persistent sessions in context of loadable modules support of alerter process.

@nucleusv
It seams not to be on the road map yet but there is always the fast lane over (co-)sponsoring development.

ZBXNEXT-3188 is similar but more general minded.

@Marc, ZBX's which you've provided I think they it is another story...

Related request: ZBXNEXT-3239.

Dear nucleusv, can you suggest some more reading on this statement?

For example Oracle ODBC driver for Linux doesn't support shared connections, ODBC Pooling = yes and CPTimeout, zabbix crashes with these settings.

Glebs, what do you mean some more reading ?

I mean... Is there a related piece of Oracle documentation or some expert's blog/article/paper/presentation covering this topic (this one I've already seen )? Please point me to it. Or is it, perhaps, a well known fact among Oracle administrators?

glebs.ivanovskis, maybe I can help on this matter.

As of release 2.0.0 of unixODBC the driver supports connection pooling.
It works on per-process basis by leaving connection open so application process may reuse the connection on the next try.
Details can be found at http://www.unixodbc.org/doc/conn_pool.html

There are couple of problems on using ODBC Pooling with Zabbix:
1. As it has already been said, it works on per-process basis. So if Zabbix server once asked ODBC for connection to DB #1 via poller #1 and the next time it asks ODBC for connection to DB #1 via poller #2, unixODBC will open new connection, since pollers #1 and #2 are different processes. So at the worst case we will have (count of zabbix_server pollers) connections open on every monitored DB. It makes (count of monitored DB) x (count of zabbix_server pollers) open connections, which are a lot!
2. As Vadim Nesterov has said, Oracle ODBC driver doesn't support pooling.

akint, thank you for response! Unfortunately, very little new information for me. What I can't understand is if pooling is done on UnixODBC side why Oracle driver needs to support pooling too? Or do you mean that combination "Zabbix + Oracle driver + pooling" leads to crashes?

@Anton Alekseev,

> So at the worst case we will have (count of zabbix_server pollers) connections open on every monitored DB. It makes (count of monitored DB) x (count of zabbix_server pollers) open connections, which are a lot!

I don't suggest to go this way. I suggest to make a special process type -> DB poller (number of such processes are set up in config), like snmp pollers and etc.

> Or do you mean that combination "Zabbix + Oracle driver + pooling" leads to crashes?
When I worked with this this issue, I had Oracle 11 drivers, they didn't support pooling and it was zabbix crashing.

glebs.ivanovskis, when you enable ODBC Pooling for Oracle driver by setting non-zero CPTimeout, any application, which uses this driver, crashes with segmentation fault.
I have tested it with the every combination of:

• drivers: Oracle 11 and 12
• unixODBC: 2.2.14 (default in EL6) and 2.3.4 (most recent, compiled from source).
• applications: zabbix_server and simple Python script using pyodbc.

nucleusv I'm agree on your proposal, I have just tried to point out that using unixODBC Pooling would be inefficient even if it worked. =)

I would love something like connection pooling in the zabbix-server. I've written a sort of proxy (between zabbix-server/proxy and zabbix-agent) to catch DB requests and provide connection pooling. It forwards every other request to the normal zabbix-agent.
(if you're interested: https://github.com/Fabian1976/mios-agent)
I haven't updated or used it in a while, so I don't know if it still functions in Zabbix 3.2 or higher.

Since having made good experience by installing PgBouncer alongside the Zabbix frontend to pool its database connections, I'm seriously thinking of using PgBouncer alongside Zabbix proxies for pooling ODBC connections too.

Possibly, using a 3rd party pooling application could be a option too, if such kind of pooling software exists for all supported RDBMS platforms, of course. This may come at the expense of having another configuration to maintain. On the other hand, the odbci.ini has to be configured anyway. Last but not least, tailored pooling software promises to provide the best implementation for the respective RDBMS.

Available in versions:
  - pre-6.0.0beta2 - 56f98dcb04, 023f240bae3

Documentation updated:

• ODBC monitoring
• Upgrade notes for 6.0.0
• What's new in Zabbix 6.0.0

This functionality is coming in Zabbix 6.0 LTS in a few weeks. Meanwhile you may preview it in the latest beta or RC releases.

"next gen dm" would be a better fit for this functionality, but i can't find an issue about it right now - if it's found, this should be closed as a duplicate

I don't recall already existing issues about "next gen dm".

Probably it's time to create and use new label
I think "dm-nextgen" should be ok.

Actually, this would be an absolute killer feature! I often stumble over desired setups where it would be very benefitial to have, say a Zabbix server in a natted network, a proxy having a public fixed IP address and then other proxies, again behind NAT devices. Also, larger organization structures with networks spread over the world would benefit from this feature quite a lot.

I'm a bit concerned about the increased delay by DataSenderFrequency resp. ProxyDataFrequency of each proxy in the chain.

When it's just about breaking the TCP connection, then this can likely justified by a reverse proxy like HAProxy:

Zabbix server --> HAProxy --> Zabbix proxy <--> Zabbix agent
Zabbix server <-- HAProxy <-- Zabbix proxy <--> Zabbix agent

When it's on the other hand about a connection direction change, then there's indeed something new required. Something that may act like a configuration cache and data queue or data transceiver, depending on the proxy mode:

Zabbix server --> Queue       <-- Zabbix proxy <--> Zabbix agent
Zabbix server <-- Transceiver --> Zabbix proxy <--> Zabbix agent

Where I'd actually limit it to the latter, the Zabbix transceiver. But I think now I'm slightly off-topic

Edit: Transceiver does not appear appropriate. Mediator is possibly a better term.

Yes, it's usually the second setup that I see a need for, and the "Queue" and "Tansceiver" is actually exactly what the Proxy does, but unfortunately not in such a chain. Sure you would have to set your DataSenderFrequency accordingly, but it should be OK to just take a little longer until agent configuration drops through the chain of proxies.
Btw for the first of the scenarios (with HAProxy), do you have an example config on how this could look like on the HAProxy side?

I have some hosts that are in a remote DC and have some nodes connected via a VPN to those hosts. I monitor the remote hosts via a proxy and want to start monitoring the far nodes but don't have a direct connection to them (so would like to connect

zabbix server -> DC proxy -> system (with zabbix-proxy) ->
remote vpn nodes with no direct route (zabbix-agent)

Hi all,

I stumbled upon that, and as a matter of fact, I always thought that chaining proxies in any way was not supported. Currently running 3.2.7.

However, I have a setup that's like this:

zabbix agent <-- zabbix proxy in DMZ (passive) <-- zabbix proxy internal network (passive) <-- zabbix master

It seems I am getting values for all, no error. Everything looks fine.

Am I just getting lucky? Or that configuration is actually supported?

Thank you.

@Fred, as far i understand Zabbix Master sends configuration to zabbix proxy internal, so you setup it on zabbix gui,  but in what way zabbix proxy in DMZ receive configuration from proxy Internal ? How do you setup it ? can you share some config ?

My main take on this one is regarding security. Zabbix server often has lots of key to many kingdoms (e.g. ssh keys for remote commands) and creeps me out letting its server port opened to the world. One single vulnerability in a publicly available 10051 could be a catastrophe.

A passive proxy in a DMZ consolidating all the data from agents and other proxies could avoid exposing the server to the world entirely.

                ,-------------,
ZBX server ---> | ZBX passive | <--- ZBX remote active proxy <---> ZBX active/passive agents
                |   proxy     | <--- ZBX active agents
                `-------------`

My approach so far is to restrict access to the zabbix server based on source IPs of the proxies, but this is cumbersome, specially in environments where active proxy resides behind failovers and can arrive from many source IPs (some backup links are dynamic)

this will most likely be solved in ZBXNEXT-1848

Available in pre-2.3.3 (trunk) r47422.

Documented with ZBXNEXT-1848

sasha CLOSED

I think that new zabbix should provide special key which will generate json string which can be used on add monitoring of zabbix[proxy,<proxy_name>,lastaccess] over LLD .. if something like this is not implemented yet.

Without this probably will be quite hard to maintain proxies availability monitoring in environments where set of proxies is not static.

sasha Moved to ZBXNEXT-2548.

It worth to mention that the proxy name (2nd item key param) is still required.
So Gabriele's request to do it optional - was ignored.

But ZBX-8898 suggests a workaround.

as discussed today, this would be a great feature (especially for the proxy internal monitoring items).

at least one edge case, though... if proxy has some config and then triggers are added/changed on the server, proxy will not know about that for a while, thus data sequence - triggers/events would be messed up. while not a frequent situation, edge cases are the most annoying to debug

per-item prioritisation would be a bit too much of a micro-management, imho

ZBXNEXT-409 appears distantly related.

See also ZBXNEXT-5075

ZBX-8202 could be related

ZBX-8035 could be related too?

It was probably done that way to workaround the case where server and proxy clock are not synchronized.

In the latest Zabbix 2.2, where delayed item queue is calculated by the server (rather than the frontend, as in 2.0), this problem is not present (excepting the soon-to-be-fixed ZBX-8476).

So we decided not to fix it in Zabbix 2.0, which would be hard to fix in the frontend alone. For this reason, closing as "Won't fix".

Sounds reasonable to me.
Thanks for the explanation.

i believe the main goal is the same as ZBXNEXT-1285, marking as dupe

nope, it doesn't. But it could

also, we need a clear description of distributed monitoring in HA: in the classic distributed zabbix monitoring every point is a SPOF because if one proxy goes down every server looks at this proxy fails to send data (or metrics are lost). If the Master fails every proxy loss data without any alert (can be mitigated using a failover cluster configuration)

In 6.0 things are very different now, so this is not actual anymore -> Closing.

It was implemented under ZBXNEXT-1580 in Zabbix 6.2.