ExportXMLWordPrintable

    • Type: Problem report
    • Resolution: Unresolved
    • Priority: Trivial
    • None
    • Affects Version/s: 7.0.30
    • Component/s: Server (S)
    • None
    • Environment:
    • Support backlog
    • 1

      The Zabbix server's HTTP agent poller opens a TCP connection per poll to an HTTPS endpoint and never calls close() on it. Sockets accumulate indefinitely in CLOSE_WAIT. Each one pins an ephemeral source port, so the process eventually cannot allocate a source port: connect() fails with EADDRNOTAVAIL and every HTTP agent item for that destination goes unsupported until the server is restarted. Observed continuously for over a week with no deceleration.

      Steps to reproduce

      1. Link "Proxmox VE by HTTP" to a Proxmox VE 9.x node using an API token.
      2. Let LLD discover the guests (~20 VMs, ~250 items).
      3. Inside the server's network namespace, watch the CLOSE_WAIT count:
        ss -tan | grep -c 'CLOSE-WAIT.*<pve-ip>:8006'
      4. The count rises monotonically and never falls.

      Expected result

      The socket is closed once the HTTP agent request completes; the CLOSE_WAIT count stays near zero.

      Actual result

      One socket per poll is left in CLOSE_WAIT permanently. The peer (pveproxy) has sent FIN; Zabbix never closes its end.

      Measurements

      • Leak rate 8.0-9.3 sockets/min across two endpoints (~4.5 and ~3.5 each), consistent with ~1 socket per HTTP agent poll.
      • The rate is linear over 4+ days with no deceleration. This rules out the "connection pool ramping up to steady-state concurrency" explanation.
      • Observed 28,231 CLOSE_WAIT sockets to a single destination occupying 28,231 distinct local ports, against a range of 32768-60999 (28,232 ports) - one short of total exhaustion. It later reached exactly 28,232 of 28,232.
      • Exhaustion is per destination, because port reuse is keyed on the full 4-tuple. Two monitored nodes therefore fail about a day apart, which disguises this as a per-target fault rather than a single Zabbix bug.
      • Only a server restart reclaims the ports (50,726 -> 2 in one instance).

      Two things that made this hard to diagnose

      It is not file-descriptor exhaustion. With nofile raised to 262144 the poller sat at ~9.5% and never approached the limit. Raising nofile does not help; the binding ceiling is the ephemeral port range.

      Host-side checks are misleading. From the host the target answered normally (ping OK, port 8006 open, HTTP 401 auth challenge). Only from inside the container's network namespace did the failure appear:

      wget: can't connect to remote host: Address not available

      The container has its own network namespace and its own port range.

      StartHTTPAgentPollers does not mitigate this. With 3 pollers configured, poller #1 performed all the work (762 fds) while #2 and #3 stayed idle at 23 fds logging "got 0 values". Adding pollers does not distribute the leak.

      Possible cause

      Zabbix 7.0 introduced persistent connections (cURL handle reuse) for HTTP agent checks. The symptom is consistent with a cURL easy/multi handle not being cleaned up when the peer closes the connection, leaving the socket in CLOSE_WAIT while the handle is dropped.

      Impact

      Complete, silent loss of HTTP agent monitoring for the affected destination roughly every 4 days at the default port range. Because the items go unsupported rather than triggering, nothing alerts unless internal-event actions are enabled.

      Workaround in use

      1. Widen the container ephemeral port range (net.ipv4.ip_local_port_range = 11000 60999, 50,000 ports) - this extends time to exhaustion from ~4 to ~7.3 days.
      2. Scheduled server restart every 3-4 days.

      Neither addresses the leak itself.

      As 7.0 is the current LTS and in full support, a fix on the 7.0.x line would be much appreciated - moving to 7.4 is unattractive given its EOL of 2026-12-31.

            Assignee:
            Zabbix Development Team
            Reporter:
            Rian AlAreqi
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated: