ExportXMLWordPrintable

    • Type: Problem report
    • Resolution: Unresolved
    • Priority: Trivial
    • None
    • Affects Version/s: 7.4.15
    • Component/s: Server (S)
    • None

      Environment
      -----------

      • Zabbix server 7.4.15, native HA cluster (2 nodes, 1 active / 1 standby), Ubuntu
      • Database: MySQL 8 (Azure Database for MySQL Flexible Server)
      • Zabbix frontend 7.4.15
      • Agents: zabbix_agent2 7.4.16 (Linux: Ubuntu, RHEL 9, SLES 15 SP4/SP6)
      • Agent config: ServerActive=<node1-fqdn>;<node2-fqdn> (HA cluster syntax), HeartbeatFrequency default (60s)
      • ~3500 enabled hosts, part monitored directly by the server, part through 7.4 proxies

      Problem
      -------
      On hosts monitored directly by the server (no proxy), the active checks availability
      (host_rtdata.active_available) does not follow the real agent state:

      1) Hosts stuck in "Not available" (active_available=2) while the agent is running, sending
         heartbeats and active check data normally. ~57 hosts affected.
      2) Hosts stuck in "Unknown" (active_available=0) although they run agent >= 6.4 (7.4.16)
         and send heartbeats. ~33 hosts affected, including 15 upgraded today from agent < 6.4
         to 7.4.16 that never left "Unknown".

      Active check data from the same hosts IS received and stored (e.g. hourly system.run item
      values are present in history), so only the availability state is wrong.

      Hosts monitored through proxies are not affected. Some hosts monitored directly by the
      server do transition correctly (0 -> 1 and 2 -> 1), so the issue is not systematic.

      Steps to reproduce (observed on one host)
      -----------------------------------------
      1. Host monitored directly by the server, agent2 7.4.x, active items present, state = available (1).
      2. Stop the agent: after the heartbeat timeout the server sets active_available=2 (correct).
      3. Start the agent again.
      4. Agent log shows successful active check configuration refresh and data sending.
      5. Server trapper (DebugLevel 4 on trapper) logs: trapper got request 'active check heartbeat'
         for the host every 60s.
      6. active_available stays 2 indefinitely; availability manager (DebugLevel 4) never writes
         the update back to 1 for this hostid.

      Note: host_rtdata already had active_available=2 before the stop (11:42), yet the availability
      manager wrote "active_available=2" again at 11:48 when the agent was stopped. Its in-memory state
      was therefore out of sync with host_rtdata, and it never writes 1 once heartbeats resume.
      See attached log excerpt.

      Already tried (no effect)
      -------------------------

      • zabbix_server -R config_cache_reload
      • unlinking / relinking the template containing the active items
      • verified Hostname / ServerActive, no "host not found" errors, no duplicate host names

      Expected
      --------
      active_available returns to 1 (available) when heartbeats resume, and goes from 0 to 1
      when a heartbeat-capable agent (>= 6.4) starts sending heartbeats.

      Actual
      ------
      State remains 2 (or 0) until (assumed) server restart. Effect: false "Active checks not
      available" problems, and missing detection of real active-agent outages for hosts in "Unknown".

      Additional info
      ---------------
      Server and availability manager debug logs can be provided on request.
      Planned: upgrade to 7.4.16 (server restart) to check whether the state realigns.

            Assignee:
            Piotr Wegrzyn
            Reporter:
            Andrea Abbate
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

              Created:
              Updated: