-
Type:
Problem report
-
Resolution: Unresolved
-
Priority:
Trivial
-
None
-
Affects Version/s: 8.0.0
-
Component/s: Server (S)
-
Environment:Server onpremise
After upgrading Zabbix server from 7.4.16 to 8.0.0, SNMPv3 hosts started
flapping: items fail with "first network error, wait for N seconds" and a few
seconds or minutes later the host recovers with "resuming SNMP agent checks:
connection restored", then fails again. "No SNMP data" problem events
increased from ~50/day before the upgrade to ~6,000/day after it.
Our setup: most monitored SNMPv3 devices are behind NAT. Many devices share
one public IP address and are distinguished only by the port (port
forwarding), e.g. 30+ switches behind a single IP.
Analysis:
- SNMPv1/v2c hosts: 0 of 30 affected
- SNMPv3 hosts with a unique IP address: 0 of 25 affected
- SNMPv3 hosts sharing an IP address with other hosts (different ports):
all 77 affected hosts belong to this group (77 of 320) - Manual snmpget from the Zabbix server to every affected device works
every time (same credentials) and each device returns its own, unique
snmpEngineID - Zabbix server pollers are not busy (snmp pollers mostly idle), no proxies
- "zabbix_server -R snmp_cache_reload" does not help, failures resume
within minutes
Zabbix 8.0 introduced caching of SNMPv3 EngineID->IP mappings (ZBXNEXT-2352).
It appears the cache key does not include the port, so devices behind the
same IP address reuse/overwrite each other's EngineID. Requests then fail,
the cache entry is evicted and re-probed, and the cycle repeats, which
causes permanent flapping.
Steps to reproduce:
1. Have two or more SNMPv3 devices with different EngineIDs reachable via
the same IP address on different ports (e.g. NAT port forwarding
IP:16101 -> device A, IP:16102 -> device B)
2. Monitor them as separate hosts in Zabbix 8.0.0 with SNMPv3 interfaces
(same IP, different ports)
3. Observe intermittent "first network error" / "connection restored"
for these hosts
Expected result:
Devices behind the same IP address but different ports are polled
correctly, as in Zabbix 7.4. The EngineID cache should be keyed by IP:port
(or per interface), or there should be a configuration option to disable
EngineID caching.
Actual result:
Hosts sharing an IP address flap between available and unavailable.
Log excerpt (host names anonymized). All hosts below are SNMPv3 devices
behind NAT, sharing a public IP address and differing only by port. Note
the flapping cycle: host2, host6 and host8 recover ("connection restored")
and fail again within ~10 seconds.
4584:20261112:071636.331 SNMP agent item "system.cpu.walk" on host "host1" failed: first network error, wait for 90 seconds
4584:20261112:071636.411 resuming SNMP agent checks on host "host2": connection restored
4599:20261112:071640.066 SNMP agent item "system.net.uptime[sysUpTime.0]" on host "host3" failed: first network error, wait for 90 seconds
4599:20261112:071640.077 SNMP agent item "sensor.fans.walk" on host "host2" failed: first network error, wait for 90 seconds
4584:20261112:071641.216 resuming SNMP agent checks on host "host4": connection restored
4584:20261112:071642.778 SNMP agent item "net.if.duplex.walk" on host "host5" failed: first network error, wait for 90 seconds
4584:20261112:071647.648 resuming SNMP agent checks on host "host6": connection restored
4584:20261112:071648.331 SNMP agent item "system.net.uptime[sysUpTime.0]" on host "host7" failed: first network error, wait for 90 seconds
4584:20261112:071648.334 resuming SNMP agent checks on host "host8": connection restored
4584:20261112:071649.329 resuming SNMP agent checks on host "host9": connection restored
4584:20261112:071653.168 SNMP agent item "system.net.uptime[sysUpTime.0]" on host "host10" failed: first network error, wait for 90 seconds
4584:20261112:071655.082 SNMP agent item "vm.memory.walk" on host "host11" failed: first network error, wait for 90 seconds
4584:20261112:071655.082 SNMP agent item "net.if.duplex.walk" on host "host12" failed: first network error, wait for 90 seconds
4584:20261112:071655.082 SNMP agent item "system.hw.serialnumber.walk" on host "host13" failed: first network error, wait for 90 seconds
4599:20261112:071657.791 SNMP agent item "sensor.psu.walk" on host "host6" failed: first network error, wait for 90 seconds
4599:20261112:071658.728 SNMP agent item "system.net.uptime[sysUpTime.0]" on host "host8" failed: first network error, wait for 90 seconds
4599:20261112:071658.750 resuming SNMP agent checks on host "host14": connection restored
4599:20261112:071659.541 SNMP agent item "system.net.uptime[sysUpTime.0]" on host "host15" failed: first network error, wait for 90 seconds
4599:20261112:071659.541 SNMP agent item "system.hw.uptime[hrSystemUptime.0]" on host "host16" failed: first network error, wait for 90 seconds
4599:20261112:071659.764 resuming SNMP agent checks on host "host17": connection restored
Related: ZBXNEXT-2352