We are investigating an intermittent issue affecting Windows Performance Counter collection with Zabbix Agent 2.

The issue was initially observed with Agent 2 7.0.17 and 7.0.18 and continued after upgrading the affected production hosts to 7.0.28.

The main symptom is:

Timeout occurred while gathering data

During an incident, a large number of perf_counter items may stop receiving data or become unsupported at the same time.

The issue is not limited to a single counter or counter path. We have observed it with different groups of counters, including:

threads and processes
IIS / Web Service
HTTP Service Request Queues
.NET
TCP
processor/system counters
custom application counters

Environment

Zabbix Server: 7.0.28
Zabbix Proxy: 7.0.28
Affected agent: Zabbix Agent 2 7.0.28
Comparison agent: Zabbix Agent classic 7.0.28
Main OS: Windows Server 2019 Datacenter, build 10.0.17763
Additional OS: Windows Server 2022 Datacenter, build 10.0.20348

The production Agent 2 fleet was upgraded to 7.0.28 on 2026-07-13. The issue continued to occur after the upgrade.

Relevant findings

One of the affected hosts has approximately 2,795 perf_counter items. A significant portion use highly dynamic instances, including process/thread, PID, IIS and .NET instances.

Agent 2 debug logs showed WindowsPerfMon collection cycles of approximately 1.9 seconds, while the relevant collection interval was approximately 1 second.

We also observed genuine PDH errors during the incidents, including:

Cannot add counter: The specified object was not found on the computer (0xC0000BB8)

Some dynamic instances are expected to disappear during normal operation, for example when an IIS Application Pool is recycled or a process/thread terminates.

However, the main concern is that the impact observed with Agent 2 is broader than an individual dynamic counter failure.

Agent 2 vs. classic Agent

The same Windows hosts were tested with:

Zabbix Agent 2 7.0.28
Zabbix Agent classic 7.0.28

On host spcdsrvv6468, approximately 291 items from the affected group were unsupported while Agent 2 was in the problematic state.

The corresponding items continued to be supported and updated when the classic Agent was used.

The classic Agent also reports failures for dynamic instances, especially PID-based IIS/.NET counters. However, those failures appear to remain mostly limited to the affected counter/instance.

With Agent 2, we observed cases where multiple unrelated perf_counter items entered a timeout condition simultaneously.

This difference is one of the main reasons why we suspect a possible issue in the Agent 2 WindowsPerfMon collector or in the way it handles PDH failures under load.

Recovery after Agent restart

In several occurrences, restarting Agent 2 temporarily restored collection.

For example, during the incident on 2026-07-13, the logs show:

Zabbix Agent 2 stopped. (7.0.28)

After the restart, the mass timeout condition disappeared for the following observation window and later returned.

The 7.0.28 logs from this occurrence did not contain a panic or stack trace.

Previous Agent 2 versions

With Agent 2 7.0.17 and 7.0.18, we also observed:

large numbers of unsupported perf_counter items
timeouts
periods without collection
increased CPU and memory consumption
Agent 2 panics

One recorded panic was:

Critical failure: runtime error: index out of range [14] with length 0

The stack trace included scheduler-related functions such as:

agent/scheduler.pluginHeap.Less
agent/scheduler.(*Manager).processFinishRequest

These symptoms are similar to previously reported issues such as ZBX-24918, ZBX-26726 and ZBX-26757.

However, the current issue is still reproducible on 7.0.28 and does not show the same panic condition.

Controlled tests

Following the investigation with Zabbix Support, controlled tests were performed using:

perf_counter["\System\Processor Queue Length"]

and:

perf_counter["\W3SVC_W3WP(*)% 404 HTTP Response Sent"]

The stable \System\Processor Queue Length counter remained accessible during the controlled test window.

The IIS counter intermittently returned valid values and errors related to negative values/denominators.

The specific Timeout occurred while gathering data condition was not captured during that controlled window.

The next reproduction is therefore focused on the stable \System\Processor Queue Length counter during an active timeout condition, comparing:

direct Windows access using typeperf
access through the running Agent 2 using zabbix_get
the Agent 2 debug log at the same time

The purpose is to determine whether the counter remains available through Windows while the already-running Agent 2 process is unable to return it.

During the latest occurrence on 2026-08-24, Agent 2 7.0.28 continued to report PDH-related errors from the WindowsPerfMon component.

Within the same time window, repeated errors were observed for different performance counter objects, including:

\Thread(_Total/_Total)% Processor Time
\W3SVC_W3WP(_Total)% 404 HTTP Response Sent

Both counters returned:

A counter with a negative value was detected.

The errors were repeated over several minutes, for example between 19:42 and 19:46.

This is relevant because the issue is not limited to a single IIS or application-specific counter. The same failure pattern was observed across different Performance Counter objects during the same occurrence.

These errors should not be considered, by themselves, proof of the root cause. However, together with the previously observed mass timeouts and collection degradation, they support further investigation of the interaction between PDH failures and the Agent 2 WindowsPerfMon collector