Network discovery creates blank hosts for failed SNMP checks after upgrading from Zabbix 7.0.27 to 7.0.28

XMLWordPrintable

    • Type: Problem report
    • Resolution: Unresolved
    • Priority: Trivial
    • None
    • Affects Version/s: 7.0.28
    • Component/s: Server (S)
    • Environment:
      Production server running Debian 12 & Zabbix 7.0.28

      Existing production server running Debian 12 & Zabbix 7.0.27.  After upgrading to Zabbix 7.0.28 via apt the Discovery Rules started creating new hosts for IP addresses that are not assigned.  ie: they are not provisioned on a device.

      7.0.28’s multi-IP discovery rewrite allows a non-SNMP service to make the discovery host up, a broad discovery action then runs on the host-level event, and the revised host-creation loop seeds the monitored host from its first service row without first verifying that row is up.

      I found two issues:

      1) On large Zabbix systems using multiple ICMP pinger workers, concurrent fping processes can overlap on the same 16-bit ICMP identifier. This can cause replies from one batch to be associated with another process, producing false reachability or incorrect packet-loss results.

      The issue is more likely on busy systems with large PID ranges and thousands of monitored hosts.

      Zabbix should invoke supported versions of fping with:

      --check-source

      This option validates that each ICMP reply comes from the expected target and prevents mismatched replies from being accepted.

       

      2)

      Affected version: Zabbix Server 7.0.28
      Previously working version: Zabbix Server 7.0.27
      Component: Network discovery / discovery actions / automatic host creation

      Description

      After upgrading from Zabbix 7.0.27 to 7.0.28, network discovery began creating monitored hosts for IP addresses that do not respond to the configured SNMP discovery check.

      The discovery rule contains SNMP checks, and the discovery action is intended to add a host only when the SNMP check succeeds. In 7.0.27, IP addresses that did not respond to SNMP were not added to the monitored host list. Starting with 7.0.28, some of these IP addresses are added as blank or incomplete hosts even though they provide no SNMP response.

      The resulting hosts generally have an IP-based name and no useful SNMP identity or discovered data.

      Expected behavior

      A monitored host should be created only when the discovery service that satisfies the discovery action conditions is in the UP state. An IP address that does not respond to the configured SNMP query should not create a monitored host.

      Actual behavior

      A monitored host can be created even when the SNMP discovery service for the selected IP address is DOWN or has not returned a value. Another discovery service, another IP associated with the same discovery host, or a previously stored discovery service appears to be sufficient to make the overall discovery host UP and trigger host creation.

      Steps to reproduce

      1. Configure a network discovery rule containing an SNMP check.
      1. Configure a discovery action that automatically adds the discovered host, assigns a host group, or links a template.
      1. Scan a range containing:
        • an IP that responds to the configured SNMP OID;
        • an IP that responds to another service, such as ICMP, but not SNMP;
        • an IP that does not respond to SNMP.
      1. Run the test on Zabbix 7.0.27 and confirm that only SNMP-responsive devices are added.
      1. Upgrade to Zabbix 7.0.28 and run the same discovery.
      1. Observe that non-SNMP-responsive IP addresses may be created as blank monitored hosts.

      Suspected regression

      The behavior appears related to the network-discovery changes introduced for ZBX-27200.

      In Zabbix 7.0.28, add_discovered_host() in:

      src/zabbix_server/operations/operations.c

      selects all discovery services associated with the discovered host, including ds.status.

      However, when no existing monitored host has been found, the first service row can enter the host-creation branch using:

      if (0 == hostid)

      The service status is checked only in the later branch:

      else if (DOBJECT_STATUS_UP == atoi(row[17]))

      This means the first discovery-service row can seed creation of a new monitored host and interface without first verifying that the service is UP.

      Because the rows are ordered by IP and port, a DOWN SNMP service or another nonqualifying service can become the first row used to create the host, while a different service or IP causes the overall discovery host event to be UP.

      Proposed correction

      Require the discovery-service row to be UP before allowing it to create a new monitored host:

      {{- if (0 == hostid)
      + if (0 == hostid && DOBJECT_STATUS_UP == atoi(row[17]))}}

      This preserves the lookup of existing monitored hosts for DOWN or LOST events, while preventing a DOWN discovery service from creating a new host.

      Temporary workaround

      We have applied the above source change locally to Zabbix 7.0.28.

      We are also restricting the discovery action with AND conditions similar to:

      {{Discovery object = Service
      Discovery status = Up
      Discovery check = exact SNMP discovery check
      Service type = configured SNMP version}}

      This reduces the likelihood of a host-level or non-SNMP discovery event triggering host creation, but the source-level first-row status omission still appears to be a regression.

      Additional notes

      A completely new IP where every configured discovery service is DOWN does not appear to create a new dhost record by itself. The issue appears when at least one related service or IP is UP, stale discovery-service data exists, or a host-level discovery event triggers the action.

      Please confirm whether the missing status check in add_discovered_host() is intentional and whether the proposed condition is an acceptable fix for the 7.0 branch.

       

      After applying the sysctl & fping tweaks and applying the patch to Zabbix and compiling Zabbix, the "ghost" hosts stopped being created.

            Assignee:
            Michael Veksler
            Reporter:
            Jim Lucas
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Created:
              Updated:

                Estimated:
                Original Estimate - Not Specified
                Not Specified
                Remaining:
                Remaining Estimate - Not Specified
                Not Specified
                Logged:
                Time Spent - 0.5h
                0.5h