XMLWordPrintable

Details

    • Incident report
    • Status: Closed
    • Critical
    • Resolution: Incomplete
    • 3.4.5, 3.4.6
    • None
    • Server (S)
    • None
    • Dockerised deployment and Standalone on CentOS Virtual machines, using a mariadb as DB backend.

    Description

      Unable to poll SNMP v3 devices with more than 1 poller being enabled on either server or proxy. With a single poller we are able to get some results back but the poller process is being overworked. (Reported in Zabbix)

      We also get gappy data from our SNMP v3 devices, but we do not experience this on Agent based host data

      Reducing the pollers to 1 has been suggested throughout the community but is not sustainable for the size of our estate.

      Scaling the proxies up to 2 causes Authentication failures on the server and massive loss of incoming data streams.

      I have TCPdumped the outgoing traffic from the server when 1 and 5 pollers and with a single poller the SNMP information is sent out correctly, but with 5 pollers the SNMP information is sent incorrectly (I have attached both traces).

      The switches are working fine I can SNMPwalk from the service/proxy directly and retrieve data without fail.

      We use Authpriv - AES/SHA

      Logfile below @ debug level 1

      20769:20180130:085653.978 SNMP agent item "ifAdminStatus[Ten-GigabitEthernet1/0/1]" on host "" failed: first network error, wait for 15 seconds
      20772:20180130:085708.628 resuming SNMP agent checks on host "": connection restored
      20764:20180130:085738.655 item ":ifAdminStatus[Ten-GigabitEthernet1/0/1]" became not supported: Cannot connect to "10.108.1.32:161": Authentication failure (incorrect password, community or key).
      20764:20180130:085738.656 item ":ifAdminStatus[Ten-GigabitEthernet2/0/9]" became not supported: Cannot connect to "10.108.1.32:161": Authentication failure (incorrect password, community or key).
      20769:20180130:085812.051 SNMP agent item "ifOutOctets[Ten-GigabitEthernet2/0/29]" on host "" failed: first network error, wait for 15 seconds
      20773:20180130:085827.659 resuming SNMP agent checks on host "": connection restored
      20762:20180130:085857.889 item ":ifAdminStatus[Bridge-Aggregation1]" became not supported: Cannot connect to "10.108.1.32:161": Authentication failure (incorrect password, community or key).
      20770:20180130:085927.970 SNMP agent item "ifOperStatus[LoopBack0]" on host " -PP" failed: first network error, wait for 15 seconds
      20771:20180130:085942.765 resuming SNMP agent checks on host " -PP": connection restored
      20764:20180130:085942.979 item " -PP:ifNumber" became not supported: Cannot connect to "10.108.1.32:161": Authentication failure (incorrect password, community or key).
      20763:20180130:090012.982 item " -PP:ifOperStatus[LoopBack0]" became not supported: Cannot connect to "10.108.1.32:161": Authentication failure (incorrect password, community or key).
      20770:20180130:090043.500 SNMP agent item "ifOutOctets[Ten-GigabitEthernet2/0/22]" on host " -PP" failed: first network error, wait for 15 seconds
      20773:20180130:090058.763 resuming SNMP agent checks on host " -PP": connection restored
      20764:20180130:090129.297 item " -PP:ifOperStatus[FortyGigE2/0/50]" became not supported: Cannot connect to "10.108.1.32:161": Authentication failure (incorrect password, community or key).
      20764:20180130:090129.297 item " -PP:ifOutOctets[Ten-GigabitEthernet2/0/22]" became not supported: Cannot connect to "10.108.1.32:161": Authentication failure (incorrect password, community or key).
      20770:20180130:090159.233 SNMP agent item "ifOutOctets[Ten-GigabitEthernet1/0/21]" on host " -PP" failed: first network error, wait for 15 seconds
      20772:20180130:090214.830 resuming SNMP agent checks on host " -PP": connection restored
      20762:20180130:090215.442 item " -PP:ifNumber" became supported
      20762:20180130:090245.464 item " -PP:ifOperStatus[Bridge-Aggregation12]" became not supported: Cannot connect to "10.108.1.32:161": Authentication failure (incorrect password, community or key).
      20769:20180130:090319.641 SNMP agent item "ifOutErrors[Ten-GigabitEthernet2/0/45]" on host " -PP" failed: first network error, wait for 15 seconds
      20774:20180130:090334.903 resuming SNMP agent checks on host " -PP": connection restored
      20762:20180130:090335.544 item " -PP:ifNumber" became not supported: Cannot connect to "10.108.1.32:161": Authentication failure (incorrect password, community or key).
      20761:20180130:090405.550 item " -PP:ifInOctets[Ten-GigabitEthernet2/0/13]" became not supported: Cannot connect to "10.108.1.32:161": Authentication failure (incorrect password, community or key).
      20770:20180130:090435.158 SNMP agent item "ifOperStatus[Ten-GigabitEthernet2/0/42]" on host " -PP" failed: first network error, wait for 15 seconds
      20773:20180130:090450.963 resuming SNMP agent checks on host " -PP": connection restored
      20763:20180130:090451.672 item " -PP:ifNumber" became supported
      20761:20180130:090521.670 item " -PP:ifOperStatus[Ten-GigabitEthernet2/0/42]" became not supported: Cannot connect to "10.108.1.32:161": Authentication failure (incorrect password, community or key).
      20761:20180130:090521.670 item " -PP:ifOutOctets[Ten-GigabitEthernet2/0/14]" became not supported: Cannot connect to "10.108.1.32:161": Authentication failure (incorrect password, community or key).
      20770:20180130:090551.567 SNMP agent item "ifOperStatus[InLoopBack0]" on host " -PP" failed: first network error, wait for 15 seconds

      Debug Level 4 log is attached below

      Problems exist in either the dockerised version or the standalone install.

      Attachments

        1. GappySNMPv3.png
          GappySNMPv3.png
          65 kB
        2. zabbix_server.log
          9.08 MB
        3. zabbixvm - 1 Poller.trace
          17 kB
        4. zabbixvm - 5 Pollers.trace
          13 kB

        Issue Links

          Activity

            People

              Unassigned Unassigned
              steveroebuck Steve Roebuck
              Votes:
              10 Vote for this issue
              Watchers:
              19 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: