1. PROBLEM

 

Hosts are monitored by a proxy group (two proxies, <PROXY_A> and <PROXY_B>). We send results of many hosts to

 

trapper items with one call: zabbix_sender -z "<SERVER_NODE_1>;<SERVER_NODE_2>" -p 10051 -T -i <file>.

 

About half of the values are rejected: "processed: 102; failed: 148; total: 250", exit code 2.

 

The hosts and trapper items are correct (type, key, status), the same values sent for a single host are accepted.

 

The proxy log shows, for the rejected values:

 

  cannot retrieve key "<KEY>" on host "<HOST>" from configuration cache

 

Nothing in the sender output or in the documentation points to the cause.

 

 

2. HOW WE FOUND THE CAUSE

 

• Sending one host's values (-vv): the server answers {"redirect": {"address":"<PROXY_B>:10051"}

  ,"response":"failed"},

 

  the sender follows the redirect -> "processed: 3; failed: 0". The same host sent to <PROXY_A> directly is redirected to <PROXY_B>.

 

• Sending values of <HOST_1> (assigned to <PROXY_B>) and <HOST_2> (assigned to <PROXY_A>) in one request:

 

  one redirect to <PROXY_B>, the whole request goes there -> "processed: 3; failed: 3; total: 6", exit code 2.

 

  The proxy log of <PROXY_B> shows the "cannot retrieve key" messages for <HOST_2>.

 

• Source code (7.0.29, 7.4, 8.0, master): src/libs/zbxdbwrap/proxy.c, zbx_process_sender_history_data() calls

 

  peek_hostkey_host() ("peek first host name in host:key history data array") and decides on the redirect

 

  (zbx_dc_config_get_hostid_by_name()) only for that first host. The redirect response applies to the whole request.

 

 

3. SOLUTION

 

zabbix_sender has the option g (-group): "Group values by hosts and send to each host in a separate batch".

 

Each batch then contains one host, so each gets its own redirect. The option exists since 7.0.0

 

(src/zabbix_sender/zabbix_sender.c, send_buffer.c, ZBX_SEND_GROUP_HOST) and is present in 7.0.29, 8.0 and master.

 

Result with the same input: zabbix_sender -z "<SERVER_NODE_1>;<SERVER_NODE_2>" -p 10051 -T -g -i <file>

 

  -> 225 batches, "sent: 676; skipped: 0; total: 676", failed: 0 in every response, answers from both proxies.

 

 

4. DOCUMENTATION GAP

 

a) The zabbix_sender man page does not describe -g. man/zabbix_sender.man (dated 2022-01-06) in release/7.0, 7.4, 8.0

 

   and master has no entry, and neither

 

   https://www.zabbix.com/documentation/7.0/en/manpages/zabbix_sender nor

 

   https://www.zabbix.com/documentation/current/en/manpages/zabbix_sender (8.0)

 

   lists it in the synopsis or the options.

 

b) The proxy load balancing / high availability page says that Zabbix sender data requests are redirected to the correct

 

   proxy of the host based on the current proxy-host assignment. [Page: .../manual/distributed_monitoring/proxies/ha

 

   in 7.0; location in 8.0: .../manual/concepts/proxy/ha] This holds only for requests containing a single host.

 

   The limitation (the first host of the request decides) and the need for -g are not mentioned.

 

c) The Zabbix sender protocol page does not describe the "redirect" response of a sender data request.

 

d) The failure is hard to diagnose: hosts and items are correct, the sender only reports "failed: N" and exit code 2.

 

 

5. SUGGESTED CHANGES

 

• zabbix_sender man page (all supported versions), add -g to the synopsis and the options:

 

  "-g, --group Group values by hosts and send to each host in a separate batch. Use this option when the input

 

   contains hosts that are monitored by different proxies of a proxy group. Without it, the redirect to a proxy is

 

   determined by the first host of the request, and the values of hosts monitored by other proxies are rejected."

 

• Proxy load balancing and high availability page, extend the Zabbix sender sentence:

 

  "When Zabbix sender sends data to the Zabbix server, the redirect to a proxy is determined by the first host in the

 

   request. If a request contains hosts monitored by different proxies, use the zabbix_sender option g (-group)

 

   or send one request per host."

 

• Zabbix sender protocol page: add a note that the redirect in the response to a sender data request relates to the

 

  first host of the data array.

 

• Optionally mention the proxy log message "cannot retrieve key ... from configuration cache" as a symptom.