-
Type:
Problem report
-
Resolution: Unresolved
-
Priority:
Trivial
-
None
-
Affects Version/s: 7.0.30
-
Component/s: Server (S)
-
None
-
Environment:Zabbix server 7.0.30, official image zabbix/zabbix-server-pgsql:alpine-7.0-latest. PostgreSQL 16. Docker on Ubuntu 24.04, kernel 6.8. Template "Proxmox VE by HTTP" (official), item types 19 (HTTP agent) and 21 (Script). Targets: two Proxmox VE 9.2.x nodes, pveproxy HTTPS :8006, API token auth, ~250 discovered items per node, master items polled every 5m.Zabbix server 7.0.30, official image zabbix/zabbix-server-pgsql:alpine-7.0-latest. PostgreSQL 16. Docker on Ubuntu 24.04, kernel 6.8. Template "Proxmox VE by HTTP" (official), item types 19 (HTTP agent) and 21 (Script). Targets: two Proxmox VE 9.2.x nodes, pveproxy HTTPS :8006, API token auth, ~250 discovered items per node, master items polled every 5m.
-
Support backlog
The Zabbix server's HTTP agent poller opens a TCP connection per poll to an HTTPS endpoint and never calls close() on it. Sockets accumulate indefinitely in CLOSE_WAIT. Each one pins an ephemeral source port, so the process eventually cannot allocate a source port: connect() fails with EADDRNOTAVAIL and every HTTP agent item for that destination goes unsupported until the server is restarted. Observed continuously for over a week with no deceleration.
Steps to reproduce
- Link "Proxmox VE by HTTP" to a Proxmox VE 9.x node using an API token.
- Let LLD discover the guests (~20 VMs, ~250 items).
- Inside the server's network namespace, watch the CLOSE_WAIT count:
ss -tan | grep -c 'CLOSE-WAIT.*<pve-ip>:8006' - The count rises monotonically and never falls.
Expected result
The socket is closed once the HTTP agent request completes; the CLOSE_WAIT count stays near zero.
Actual result
One socket per poll is left in CLOSE_WAIT permanently. The peer (pveproxy) has sent FIN; Zabbix never closes its end.
Measurements
- Leak rate 8.0-9.3 sockets/min across two endpoints (~4.5 and ~3.5 each), consistent with ~1 socket per HTTP agent poll.
- The rate is linear over 4+ days with no deceleration. This rules out the "connection pool ramping up to steady-state concurrency" explanation.
- Observed 28,231 CLOSE_WAIT sockets to a single destination occupying 28,231 distinct local ports, against a range of 32768-60999 (28,232 ports) - one short of total exhaustion. It later reached exactly 28,232 of 28,232.
- Exhaustion is per destination, because port reuse is keyed on the full 4-tuple. Two monitored nodes therefore fail about a day apart, which disguises this as a per-target fault rather than a single Zabbix bug.
- Only a server restart reclaims the ports (50,726 -> 2 in one instance).
Two things that made this hard to diagnose
It is not file-descriptor exhaustion. With nofile raised to 262144 the poller sat at ~9.5% and never approached the limit. Raising nofile does not help; the binding ceiling is the ephemeral port range.
Host-side checks are misleading. From the host the target answered normally (ping OK, port 8006 open, HTTP 401 auth challenge). Only from inside the container's network namespace did the failure appear:
wget: can't connect to remote host: Address not available
The container has its own network namespace and its own port range.
StartHTTPAgentPollers does not mitigate this. With 3 pollers configured, poller #1 performed all the work (762 fds) while #2 and #3 stayed idle at 23 fds logging "got 0 values". Adding pollers does not distribute the leak.
Possible cause
Zabbix 7.0 introduced persistent connections (cURL handle reuse) for HTTP agent checks. The symptom is consistent with a cURL easy/multi handle not being cleaned up when the peer closes the connection, leaving the socket in CLOSE_WAIT while the handle is dropped.
Impact
Complete, silent loss of HTTP agent monitoring for the affected destination roughly every 4 days at the default port range. Because the items go unsupported rather than triggering, nothing alerts unless internal-event actions are enabled.
Workaround in use
- Widen the container ephemeral port range (net.ipv4.ip_local_port_range = 11000 60999, 50,000 ports) - this extends time to exhaustion from ~4 to ~7.3 days.
- Scheduled server restart every 3-4 days.
Neither addresses the leak itself.
As 7.0 is the current LTS and in full support, a fix on the 7.0.x line would be much appreciated - moving to 7.4 is unattractive given its EOL of 2026-12-31.