ExportXMLWordPrintable

    • Type: Problem report
    • Resolution: Unresolved
    • Priority: Major
    • None
    • Affects Version/s: 8.0.0
    • Component/s: Proxy (P)

      Environment

      • Zabbix server and proxy 8.0.0 (c81e276), built from tag 8.0.0, using PostgreSQL
      • ClickHouse 26.8.17.4, zabbix.otel_logs created with the official otel_logs_schema.sh, default TTL of 31 days
      • OpenTelemetry Collector Contrib 0.164.0

      Steps to reproduce

      1. In a working server/proxy setup, configure an active proxy with Hostname=repro-proxy, APM TLS disabled and these parameters:

      StartAPMCollector=1
      TelemetryProvider=clickhouse;url=http://<clickhouse>:8123,db=zabbix
      

      2. Register the proxy with APM collection enabled. For example, use these parameters for proxy.create:

      {"name":"repro-proxy","operating_mode":0,"apm":{"data_collection_status":1}}
      

      Wait until the proxy receives its configuration from the server.

      3. Pause TTL merges so the inserted row can be inspected:

      SYSTEM STOP TTL MERGES zabbix.otel_logs;
      

      4. On the Collector host, write a log line without a timestamp:

      echo "observed-only-test plain application log line without timestamp" > /tmp/app.log
      

      5. Save the following configuration as /tmp/filelog-repro.yaml. Replace <proxy> with the proxy address. The Collector must be able to read /tmp/app.log.

      receivers: 
        filelog: 
          include: [/tmp/app.log]
          start_at: beginning
      processors: 
        resource: 
          attributes: 
            - key: service.name
              value: filelog-repro
              action: upsert
      exporters: 
        otlp: 
          endpoint: <proxy>:4317
          tls: 
            insecure: true
        debug: 
          verbosity: detailed
      service: 
        pipelines: 
          logs: 
            receivers: [filelog]
            processors: [resource]
            exporters: [otlp, debug]
      

      Start the Collector:

      otelcol-contrib --config=/tmp/filelog-repro.yaml
      

      6. After the record is exported, check ClickHouse:

      SELECT
          toUnixTimestamp64Nano(Timestamp) AS ns,
          _partition_id,
          Body
      FROM zabbix.otel_logs
      WHERE ServiceName = 'filelog-repro';
      

      7. Resume TTL merges and repeat the SELECT to observe removal:

      SYSTEM START TTL MERGES zabbix.otel_logs;
      

      Result

      Collector emits a record with no event timestamp and a valid observed timestamp:

      ObservedTimestamp: 2026-11-07 03:47:20.034540741 +0000 UTC
      Timestamp: 1970-01-01 00:00:00 +0000 UTC
      

      The proxy stores it with Timestamp 0:

      ns Partition Body
      0 19700101 observed-only-test plain application log line without timestamp

      After TTL merges resume, ClickHouse removes the row.

      Expected

      When time_unix_nano is zero and observed_time_unix_nano is nonzero, the proxy should use the observed timestamp. The record should then be retained according to the configured TTL. Valid nonzero event timestamps should keep precedence.

      OpenTelemetry Logs Data Model recommends the following when mapping logs to a format with one timestamp:

      Use Timestamp if it is present, otherwise use ObservedTimestamp.

      This affects records with an omitted or zero time_unix_nano and a nonzero observed_time_unix_nano. The filelog configuration above produces such records without any timestamp manipulation.

            Assignee:
            Zabbix Development Team
            Reporter:
            Piotr Wegrzyn
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

              Created:
              Updated: