Problem description

In Zabbix 8.0.0rc1, the APM trace details view does not display all spans belonging to a trace when the spans are distributed over a longer period of time.

The trace list correctly reports the total number of spans, and all spans are present in ClickHouse. However, opening the trace details view displays only spans whose start timestamps fall within a narrow time window around the beginning of the trace.

This makes the APM trace visualization incomplete and potentially misleading, particularly for long-running operations.

Environment

• Zabbix version: 8.0.0rc1

• Affected component: Frontend (APM trace details)

• APM ingestion: Zabbix Proxy with OpenTelemetry OTLP/gRPC

• Storage: ClickHouse

• Instrumented application: Python data collection service

• Instrumentation: OpenTelemetry Python SDK 1.33.1

• Trace duration: Approximately 40 seconds

• Spans per trace: 10 (one root span and nine child spans)

Steps to reproduce

1. Configure Zabbix APM with a working OpenTelemetry collector/exporter and ClickHouse storage.

1. Instrument an application to generate a trace containing multiple child spans that start at different times over approximately 40 seconds.

1. Export the trace using OTLP/gRPC.

1. Open APM → Traces in the Zabbix frontend.

1. Locate the trace and verify that the trace list reports 10 spans.

1. Open the trace details panel.

Actual result

Only the first 3 spans are displayed, even though the trace contains 10 spans.

The following spans are visible:

• datacollector.cycle (root span)

• solaredge.read

• orno.read

The remaining seven spans are missing from the visualization.

The browser's Network developer tools confirm that the backend response (trace_view_data.spans) contains only three span objects. Therefore, the issue occurs before frontend visualization and is not caused by the JavaScript renderer.

A direct ClickHouse query confirms that all 10 spans exist, have unique span IDs, and have the correct parent-child relationships.

Example trace ID:

88329be27fb314214bf9d31e0ccd0a77

Example span start offsets relative to the root span:

Expected result

The trace details view should display all 10 spans associated with the requested trace ID, including spans that start several seconds after the root span.

The displayed span tree should match the complete trace stored in ClickHouse.

Root cause analysis

The issue appears to originate in:

ui/app/controllers/CControllerApmTraceListSplitView.php

In the doAction() method, the time range is calculated as follows:

{{$traceid = $this->getInput('traceid');
$time_from = $this->getInput('timestamp') - 1;
$time_to = $this->getInput('timestamp') + 1;}}

The same time range is subsequently passed to the span retrieval API:

{{$spans = API::ApmSpan()->get([
'output' => [
'spanid', 'parent_spanid', 'service_name',
'span_name', 'scope_name', 'timestamp',
'duration', 'events', 'resource_attributes',
'span_attributes'
],
'time_from' => $time_from,
'time_till' => $time_to,
'traceids' => [$traceid],
'sortfield' => 'timestamp',
'sortorder' => ZBX_SORT_UP
]);}}

This limits span retrieval to a two-second window around the supplied trace timestamp.

As a result, spans starting later in a long-running trace are excluded from the API response even when they belong to the requested trace ID.

Tested workaround

After retrieving the root trace and verifying that it exists, the following code was added immediately before the API::ApmSpan()->get() call:

{{// Include all spans belonging to the trace, not only its first seconds.
$trace_duration_seconds = (int) ceil(
$trace['duration'] * SEC_PER_NANOSEC
);

$time_to = $this->getInput('timestamp')
+ max(1, $trace_duration_seconds + 1);}}

This extends the span retrieval window to cover the full duration of the trace, with a small additional margin.

The original time_from value remains unchanged.

Workaround verification

After applying the modification:

• The trace details view correctly displays all 10 spans.

• The complete parent-child hierarchy is visible.

• All operations appear at their expected positions on the trace timeline.

• The displayed durations correspond to the data stored in ClickHouse.

• The tested trace renders correctly without modifying the instrumented application, OpenTelemetry exporter, Zabbix Proxy, or ClickHouse.

Result: The workaround successfully resolves the issue in the tested environment.

Suggested resolution

The span retrieval time range should cover the complete duration of the requested trace rather than using a fixed ±1-second window.

Alternatively, if supported by the underlying API and storage implementation, span retrieval could be based directly on the trace ID without applying an unnecessarily restrictive timestamp window.

The proposed workaround has been validated locally but has not undergone comprehensive regression testing. In particular, traces containing multiple root spans, unusual timestamps, or spans extending beyond the reported trace duration may require additional consideration.

Impact

This issue affects the reliability of APM trace analysis for applications with long-running operations.

Users may incorrectly conclude that spans were lost during OpenTelemetry export or ingestion, even though the complete trace is correctly stored in ClickHouse.

The issue is especially relevant for background jobs, scheduled tasks, batch processing, data collection services, and other workflows that execute sequential operations over multiple seconds.

Conclusion

The underlying trace data is complete and correctly stored. The problem is caused by the time range used when retrieving spans for the APM trace details view.

Extending the retrieval window based on the trace duration restores the complete visualization in the tested environment.