In our environment, we have 10 proxies serving approximately 50,000 hosts. These proxies are grouped into a proxy group, where each proxy handles around 5,000 hosts and over 1 million items.

We intentionally avoid frequent polling to reduce infrastructure load, so the average Required vps remains around 1600. The environment is generally stable — VM resource utilization and internal Zabbix proxy process utilization are typically around 20–30%.

Most importantly, we have thoroughly verified the entire data path, from the Zabbix agent through the proxy to Zabbix-server and the database. We also carefully analyzed the proxy load. Everything operates as expected and there are no performance bottlenecks.

All agents use the following configuration relevant to this report:

 
RefreshActiveChecks=120
HeartbeatFrequency=60
BufferSend=90
 
The proxies use the following configuration:
ProxyConfigFrequency=60
DataSenderFrequency=5
StartTrappers=100
ListenBacklog=8192
 
Some notes:

• StartTrappers=100 was chosen after extensive testing. Lower values caused TCP queues on the proxy to fill up under load, while this value keeps them mostly empty.
• ListenBacklog was increased to reduce TCP backlog saturation.

Problem

The issue primarily affects active Zabbix agent items.

The Queue view reports significantly more delayed items than are actually delayed.

For a long time, our environment has shown a queue of approximately 800,000 items. After analyzing the queue and removing obvious issues (for example, unavailable agents), around 500,000–600,000 queued items remain.

This remaining queue consists almost entirely of healthy hosts. These items appear in the queue only because we intentionally configured larger values for BufferSend and DataSenderFrequency.

A detailed analysis of these queue entries shows that:

• most of them are delayed by no more than 95 seconds, which exactly matches BufferSend (90s) + DataSenderFrequency (5s);
• the entries appear periodically according to the item polling interval;
• this behavior is fully expected.

We understand that the agent may send buffered data earlier. We are referring to the worst-case scenario, where data is transmitted only after the full BufferSend interval has elapsed.

Expected behavior

Currently, the Queue appears to be calculated based only on:
nextcheck
 
However, for active agents it should also take into account delays that are explicitly introduced by the configured buffering mechanism.

For active agent items the effective threshold should be approximately:
nextcheck + BufferSend
 
and for hosts monitored through a proxy:
nextcheck + BufferSend + DataSenderFrequency
 
Otherwise, the Queue reports items as delayed even though they are behaving exactly as configured.

Why this matters

We believe this could benefit many other users as well.

Even with the default BufferSend=5, users may frequently see delayed items in the first column of the Queue overview, despite the fact that everything is working correctly. The apparent delay is simply caused by the configured buffering behavior of the active agent.

Additional notes

Reducing BufferSend, reducing DataSenderFrequency, or adding more proxies is not a valid solution for large environments.

These changes increase infrastructure resource consumption and result in significantly more TCP connections. In environments of our size, this leads to increased pressure on the kernel TCP queues without solving the underlying issue.

From our perspective, the Queue should distinguish between actual collection delays and expected buffering delays introduced by the configured agent/proxy behavior. Otherwise, the Queue becomes difficult to use as an operational indicator because it reports a very large number of false-positive delayed items.