APM proxy silently loses all OTLP data written to ClickHouse: DateTime64 sent as unquoted decimal is rejected

ExportXMLWordPrintable

    • Type: Patch request
    • Resolution: Unresolved
    • Priority: Trivial
    • None
    • Affects Version/s: 8.0.0rc1
    • Component/s: Proxy (P)
    • Environment:

      Steps to reproduce:
      1. Send any OTLP log record to the proxy's APM collector (port 4317).
      2. Query otel_logs: no rows ever appear. APM > Logs shows "No data found".

      • The proxy log shows no error. The APM manager reports "accepted 1, dropped 0 requests".
      • ClickHouse query_log shows the proxy's INSERT INTO otel_logs FORMAT JSONCompactEachRow
        finishing as QueryFinish with no exception (async_insert=1, written_rows=0).
      • system.part_log has no entries for otel_logs: no data is ever written.
      • system.errors counts CANNOT_PARSE_INPUT_ASSERTION_FAILED once per batch (2288 so far):
        Cannot parse input: expected ',' before: '.556000000,"","",0,"ERROR",17,"zabbix-server",...'
        Column 0, name: Timestamp, type: DateTime64(9), parsed text: "1791529987"
        ERROR: garbage after DateTime64(9): ".556000000"

      Cause (source as of commit 154886c, 2026-10-07):

      • src/zabbix_proxy/apm/apm_clickhouse.c, case APM_COL_DATETIME64: the value is formatted as
        "<seconds>.<9-digit nanoseconds>" and added with ZBX_JSON_TYPE_NUMBER (unquoted).
        ClickHouse cannot parse an unquoted fractional number into DateTime64 in JSONCompactEachRow.
        This affects every DateTime64 column, so traces and metrics fail the same way.
      • The insert URL uses async_insert=1&wait_for_async_insert=0, so the parse failure happens
        after ClickHouse has already confirmed the insert. The proxy never learns of it, and the data
        is dropped with no log entry.

      Proof (clickhouse local, 26.1.12.23 and 26.4.4.38):
      [1791529987.556000000] -> Code 27, Cannot parse input (what the proxy sends)
      ["1791529987.556000000"] -> 2026-10-09 07:13:07.556000000 (quoted: OK)
      [1791529987556000000] -> 2026-10-09 07:13:07.556000000 (integer ns: OK)
      date_time_input_format=best_effort and input_format_json_read_numbers_as_strings=1 do not help.

      Suggested fix:
      1. Send DateTime64 values as quoted strings (ZBX_JSON_TYPE_STRING) or as integer nanoseconds.
      2. Make insert failures visible: use wait_for_async_insert=1, or check
      system.asynchronous_insert_log and log failures as warnings.

      Workaround in use:
      OpenTelemetry Collector clickhouse exporter writing directly to the same otel_logs table
      (this works and the data is visible in APM > Logs).

      I wanted to send zabbix server logs into APM and I used otelcol-contrib with this config file

      receivers:
  file_log/zabbix_server:
    include: [/var/log/zabbix/zabbix_server.log]
    start_at: end                                    # 'beginning' to backfill once
    resource:
      service.name: zabbix-server
      host.name: "Zabbix server"                    # what you filter on in APM > Logs
    multiline:                                       # glue continuation lines (e.g. SQL errors)
      line_start_pattern: '^\s*\d+:\d{8}:\d{6}\.\d{3} '
    operators: &zabbix_log_parsing
      # Zabbix line format: "  PID:YYYYMMDD:HHMMSS.mmm message"
      - type: regex_parser
        regex: '(?s)^\s*(?P<pid>\d+):(?P<ts>\d{8}:\d{6}\.\d{3}) (?P<msg>.*)$'
        on_error: send_quiet
        timestamp:
          parse_from: attributes.ts
          layout_type: strptime
          layout: '%Y%m%d:%H%M%S.%L'
          # Zabbix writes local time. If the collector runs in another time zone
          location: UTC
      # Zabbix log lines carry no level, so derive one from keywords
      # (adjust to taste); this feeds the Severity filter in APM > Logs.
      - type: add
        field: attributes.level
        value: INFO
      - type: add
        field: attributes.level
        value: WARN
        if: 'body matches "(?i)(warning|slow query|not supported|unavailable|unreachable|temporarily disabling)"'
      - type: add
        field: attributes.level
        value: ERROR
        if: 'body matches "(?i)(cannot|failed|error|fatal|database is down)"'
      - type: severity_parser
        parse_from: attributes.level
      # Keep only the message text as the log body; keep the PID as an attribute
      - type: move
        from: attributes.msg
        to: body
        on_error: send_quiet
      - type: remove
        field: attributes.ts
        on_error: send_quiet
      - type: remove
        field: attributes.level
        on_error: send_quiet

  file_log/zabbix_proxy:
    include: [/var/log/zabbix/zabbix_proxy.log]
    start_at: end
    resource:
      service.name: zabbix-proxy
    multiline:
      line_start_pattern: '^\s*\d+:\d{8}:\d{6}\.\d{3} '
    operators: *zabbix_log_parsing

processors:
  resource_detection:
    detectors: [system]

exporters:
  debug:
    verbosity: detailed
  otlp_grpc/zabbix_apm:
    endpoint: 172.16.70.4:4317   # proxy with StartAPMCollectors>=1 (APMListenPort)
    tls:
      insecure: true             # if the proxy uses APMTLSCertFile/KeyFil, replace with:
      # ca_file: /etc/otelcol-contrib/zabbix-ca.crt
      # cert_file: /etc/otelcol-contrib/client.crt   # only if APMTLSCAFile is set (mTLS)
      # key_file:  /etc/otelcol-contrib/client.key
  clickhouse:
    endpoint: http://172.16.90.6:8123
    database: default
    username: default
    create_schema: true

service:
  pipelines:
    logs:
      receivers: [file_log/zabbix_server]
      processors: []
      exporters: [debug, otlp_grpc/zabbix_apm]

            Assignee:
            Andrejs Poddubnaks
            Reporter:
            Adam Drmota
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

              Created:
              Updated: