ExportXMLWordPrintable

    • Type: Problem report
    • Resolution: Unresolved
    • Priority: Trivial
    • None
    • Affects Version/s: 8.0.0
    • Component/s: Server (S)
    • Environment:
      Server onpremise

      After upgrading Zabbix server from 7.4.16 to 8.0.0, SNMPv3 hosts started
      flapping: items fail with "first network error, wait for N seconds" and a few
      seconds or minutes later the host recovers with "resuming SNMP agent checks:
      connection restored", then fails again. "No SNMP data" problem events
      increased from ~50/day before the upgrade to ~6,000/day after it.

      Our setup: most monitored SNMPv3 devices are behind NAT. Many devices share
      one public IP address and are distinguished only by the port (port
      forwarding), e.g. 30+ switches behind a single IP.

      Analysis:

      • SNMPv1/v2c hosts: 0 of 30 affected
      • SNMPv3 hosts with a unique IP address: 0 of 25 affected
      • SNMPv3 hosts sharing an IP address with other hosts (different ports):
          all 77 affected hosts belong to this group (77 of 320)
      • Manual snmpget from the Zabbix server to every affected device works
          every time (same credentials) and each device returns its own, unique
          snmpEngineID
      • Zabbix server pollers are not busy (snmp pollers mostly idle), no proxies
      • "zabbix_server -R snmp_cache_reload" does not help, failures resume
          within minutes

      Zabbix 8.0 introduced caching of SNMPv3 EngineID->IP mappings (ZBXNEXT-2352).
      It appears the cache key does not include the port, so devices behind the
      same IP address reuse/overwrite each other's EngineID. Requests then fail,
      the cache entry is evicted and re-probed, and the cycle repeats, which
      causes permanent flapping.

      Steps to reproduce:
      1. Have two or more SNMPv3 devices with different EngineIDs reachable via
         the same IP address on different ports (e.g. NAT port forwarding
         IP:16101 -> device A, IP:16102 -> device B)
      2. Monitor them as separate hosts in Zabbix 8.0.0 with SNMPv3 interfaces
         (same IP, different ports)
      3. Observe intermittent "first network error" / "connection restored"
         for these hosts

      Expected result:
      Devices behind the same IP address but different ports are polled
      correctly, as in Zabbix 7.4. The EngineID cache should be keyed by IP:port
      (or per interface), or there should be a configuration option to disable
      EngineID caching.

      Actual result:
      Hosts sharing an IP address flap between available and unavailable.

      Log excerpt (host names anonymized). All hosts below are SNMPv3 devices
      behind NAT, sharing a public IP address and differing only by port. Note
      the flapping cycle: host2, host6 and host8 recover ("connection restored")
      and fail again within ~10 seconds.

        4584:20261112:071636.331 SNMP agent item "system.cpu.walk" on host "host1" failed: first network error, wait for 90 seconds
        4584:20261112:071636.411 resuming SNMP agent checks on host "host2": connection restored
        4599:20261112:071640.066 SNMP agent item "system.net.uptime[sysUpTime.0]" on host "host3" failed: first network error, wait for 90 seconds
        4599:20261112:071640.077 SNMP agent item "sensor.fans.walk" on host "host2" failed: first network error, wait for 90 seconds
        4584:20261112:071641.216 resuming SNMP agent checks on host "host4": connection restored
        4584:20261112:071642.778 SNMP agent item "net.if.duplex.walk" on host "host5" failed: first network error, wait for 90 seconds
        4584:20261112:071647.648 resuming SNMP agent checks on host "host6": connection restored
        4584:20261112:071648.331 SNMP agent item "system.net.uptime[sysUpTime.0]" on host "host7" failed: first network error, wait for 90 seconds
        4584:20261112:071648.334 resuming SNMP agent checks on host "host8": connection restored
        4584:20261112:071649.329 resuming SNMP agent checks on host "host9": connection restored
        4584:20261112:071653.168 SNMP agent item "system.net.uptime[sysUpTime.0]" on host "host10" failed: first network error, wait for 90 seconds
        4584:20261112:071655.082 SNMP agent item "vm.memory.walk" on host "host11" failed: first network error, wait for 90 seconds
        4584:20261112:071655.082 SNMP agent item "net.if.duplex.walk" on host "host12" failed: first network error, wait for 90 seconds
        4584:20261112:071655.082 SNMP agent item "system.hw.serialnumber.walk" on host "host13" failed: first network error, wait for 90 seconds
        4599:20261112:071657.791 SNMP agent item "sensor.psu.walk" on host "host6" failed: first network error, wait for 90 seconds
        4599:20261112:071658.728 SNMP agent item "system.net.uptime[sysUpTime.0]" on host "host8" failed: first network error, wait for 90 seconds
        4599:20261112:071658.750 resuming SNMP agent checks on host "host14": connection restored
        4599:20261112:071659.541 SNMP agent item "system.net.uptime[sysUpTime.0]" on host "host15" failed: first network error, wait for 90 seconds
        4599:20261112:071659.541 SNMP agent item "system.hw.uptime[hrSystemUptime.0]" on host "host16" failed: first network error, wait for 90 seconds
        4599:20261112:071659.764 resuming SNMP agent checks on host "host17": connection restored

      Related: ZBXNEXT-2352

            Assignee:
            Aigars Kadikis
            Reporter:
            TojeRoman
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Created:
              Updated: