Async SNMP poller responses cross-contaminate between hosts when StartSNMPPollers >= 5

XMLWordPrintable

    • Type: Problem report
    • Resolution: Unresolved
    • Priority: Trivial
    • None
    • Affects Version/s: 7.0.25
    • Component/s: Server (S)
    • Environment:

        1. Steps to Reproduce

      1. Configure a Zabbix 7.0.25 server with *5 or more SNMP pollers* (`StartSNMPPollers=5+`).
      2. Add ≥ 20 SNMP-monitored hosts in the same subnet, each with the `system.name[sysName.0]` item from `Generic by SNMP` (or any `cpqHe*` / vendor-specific SNMP item from vendor templates).
      3. Tested with both `bulk=1` (default) and `bulk=0` on the SNMP interfaces — no difference.
      4. Tested with both `useip=1` (fixed IP) and `useip=0` (DNS resolution) — no difference.
      5. Allow the pollers to run for 5+ minutes.

        1. Expected Behaviour

      Each host's SNMP item is populated with the SNMP response from *that host's* IP/DNS endpoint.

        1. Actual Behaviour

      SNMP responses are intermittently routed to the *wrong host's items*. Concrete observations:

      Polled host (technical) Item `system.name[sysName.0]` value Expected
      `AP05` `AP15` `AP05`
      `AP04` `AP16` `AP04`
      `AP02` `AP14` `AP02`
      `AP01` `AP12` `AP01`
      `PLL01` `PWEA2` `PLL01`
      `SWBZ09` `SWSW01` `SWBZ09`
      `SWBZ05` `SWBZ06` `SWBZ05`

      The "incorrect" sysName values are always those of *other hosts in the same poll batch*, never random / garbage strings. This rules out network-level corruption and points to a response-to-host mapping race in the async poller layer.

      The triggering effect: stock template trigger `"System name has changed"` fires repeatedly (and incorrectly).

      Manual `snmpget -v2c -c <community> <ip> .1.3.6.1.2.1.1.5.0` from the Zabbix server host always returns the *correct* sysName for the queried IP — the underlying SNMP agents and the network are fine.

        1. Workaround

      Setting `StartSNMPPollers=1` in `zabbix_server.conf` and restarting the server *completely eliminates* the cross-contamination. All SNMP items immediately start showing correct values from their own hosts. Trade-off: a single poller is slower at clearing the queue but works correctly.

      We tried (without success):

      • `bulk=0` on all SNMP interfaces
      • Switching `useip=0` (DNS mode) on all interfaces
      • `config_cache_reload`, `snmp_cache_reload`, full server restart
      • Reducing item update intervals

      Only `StartSNMPPollers=1` resolved the issue.

        1. Hypothesis

      The async SNMP poller infrastructure dispatches multiple in-flight SNMP requests in parallel (one per poller worker), and the response-to-item mapping appears to confuse responses when:

      • multiple in-flight requests are outstanding simultaneously, AND
      • response packets arrive in close timing windows (likely on the same UDP socket or a shared completion queue).
        1. Evidence / Logs

      (Collected on 2026-05-07. Excerpts redacted — IPs / hostnames available on request.)

      • Item history shows the exact moment when responses got mismapped (timestamps correlate with poller process activity).
      • After `StartSNMPPollers=1` + server restart, *all subsequent polls* (verified for AP01-08, SWBZ09 etc.) show correct sysName values.
      • No firewall / network changes occurred at the boundaries of the affected time window.
      • Reproducer is robust: pre-restart, contamination resumes within ~5 minutes; post-restart with 1 poller, never reoccurs.
        1. Severity (Suggested)

      *Major* — silent data corruption (items receive values from the wrong source), causing false-positive alerts on stock-template triggers (e.g. `"System name has changed"`, `"Disk media type changed"`, `"Sensor name changed"`). Affects any monitoring setup with mid-to-large SNMP fleets.

        1. Submitter Notes
      • The issue did NOT exist on Zabbix 6.x with traditional synchronous SNMP pollers.
      • The issue ONLY appeared after enabling 5 parallel async SNMP pollers in 7.0.25.
      • We have not tested 7.0.26+ — the bug may already be fixed upstream. Filing for visibility.

        1. image-2026-06-09-15-06-54-836.png
          image-2026-06-09-15-06-54-836.png
          70 kB
        2. image-2026-06-09-15-07-00-258.png
          image-2026-06-09-15-07-00-258.png
          70 kB
        3. image-2026-06-09-15-07-04-323.png
          image-2026-06-09-15-07-04-323.png
          78 kB
        4. image-2026-06-09-15-07-33-194.png
          image-2026-06-09-15-07-33-194.png
          129 kB
        5. image-2026-06-09-15-14-01-256.png
          image-2026-06-09-15-14-01-256.png
          63 kB
        6. image-2026-06-09-15-18-41-944.png
          image-2026-06-09-15-18-41-944.png
          38 kB
        7. image-2026-06-09-15-22-44-456.png
          image-2026-06-09-15-22-44-456.png
          115 kB
        8. image-2026-06-09-15-22-54-301.png
          image-2026-06-09-15-22-54-301.png
          61 kB
        9. image-2026-06-09-15-23-37-487.png
          image-2026-06-09-15-23-37-487.png
          105 kB
        10. image-2026-06-09-15-23-45-996.png
          image-2026-06-09-15-23-45-996.png
          56 kB
        11. image-2026-06-09-15-23-52-886.png
          image-2026-06-09-15-23-52-886.png
          77 kB
        12. image-2026-06-09-15-24-06-146.png
          image-2026-06-09-15-24-06-146.png
          85 kB
        13. image-2026-06-09-15-24-24-329.png
          image-2026-06-09-15-24-24-329.png
          66 kB
        14. image-2026-06-09-15-24-37-707.png
          image-2026-06-09-15-24-37-707.png
          106 kB
        15. image-2026-06-09-15-25-27-518.png
          image-2026-06-09-15-25-27-518.png
          111 kB
        16. image-2026-06-09-15-25-46-961.png
          image-2026-06-09-15-25-46-961.png
          105 kB
        17. image-2026-06-09-15-25-56-791.png
          image-2026-06-09-15-25-56-791.png
          58 kB
        18. image-2026-06-09-15-26-04-255.png
          image-2026-06-09-15-26-04-255.png
          93 kB
        19. image-2026-06-09-15-26-21-879.png
          image-2026-06-09-15-26-21-879.png
          76 kB
        20. image-2026-06-09-15-26-27-333.png
          image-2026-06-09-15-26-27-333.png
          73 kB
        21. image-2026-06-09-15-26-33-789.png
          image-2026-06-09-15-26-33-789.png
          93 kB
        22. image-2026-06-09-15-26-46-035.png
          image-2026-06-09-15-26-46-035.png
          92 kB
        23. image-2026-06-09-15-26-53-560.png
          image-2026-06-09-15-26-53-560.png
          58 kB
        24. image-2026-06-09-15-26-58-280.png
          image-2026-06-09-15-26-58-280.png
          81 kB
        25. image-2026-06-09-15-27-33-464.png
          image-2026-06-09-15-27-33-464.png
          122 kB
        26. image-2026-06-09-15-27-52-609.png
          image-2026-06-09-15-27-52-609.png
          105 kB
        27. image-2026-06-09-15-27-59-180.png
          image-2026-06-09-15-27-59-180.png
          57 kB
        28. image-2026-06-09-15-28-11-414.png
          image-2026-06-09-15-28-11-414.png
          94 kB
        29. image-2026-06-09-15-28-15-823.png
          image-2026-06-09-15-28-15-823.png
          69 kB
        30. image-2026-06-09-15-28-26-276.png
          image-2026-06-09-15-28-26-276.png
          104 kB
        31. image-2026-06-09-15-28-31-278.png
          image-2026-06-09-15-28-31-278.png
          55 kB
        32. image-2026-06-09-15-29-11-446.png
          image-2026-06-09-15-29-11-446.png
          74 kB
        33. image-2026-06-09-15-29-17-002.png
          image-2026-06-09-15-29-17-002.png
          49 kB
        34. image-2026-06-09-15-29-25-894.png
          image-2026-06-09-15-29-25-894.png
          94 kB
        35. image-2026-06-09-15-29-35-489.png
          image-2026-06-09-15-29-35-489.png
          95 kB
        36. image-2026-06-09-15-29-39-779.png
          image-2026-06-09-15-29-39-779.png
          66 kB
        37. image-2026-06-09-15-29-50-656.png
          image-2026-06-09-15-29-50-656.png
          76 kB
        38. image-2026-06-09-15-29-55-491.png
          image-2026-06-09-15-29-55-491.png
          87 kB

            Assignee:
            Andrejs Poddubnaks
            Reporter:
            Sebastian Geisler
            Votes:
            1 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated: