ExportXMLWordPrintable

    • Type: Problem report
    • Resolution: Unresolved
    • Priority: Major
    • None
    • Affects Version/s: 8.0.0
    • Component/s: Proxy (P)

      Environment

      • Zabbix server and proxy 8.0.0, proxy with "StartAPMCollector=1"
      • OpenTelemetry Collector 0.123.0, OTLP/gRPC receiver and exporter

      Steps to reproduce

      1. Create proxy "proxy test". On APM tab enable data collection and set Max messages per second to "1".
      2. Start Collector with block 1. Its receiver listens on 14317, because proxy uses 4317.
      3. Restart proxy. Within 60 seconds send 70 OTLP log requests to Collector, one log each, every 100 ms.
      4. Wait 65 seconds and check Collector log.

      1. Collector configuration

      receivers: 
        otlp: 
          protocols: 
            grpc: 
              endpoint: 127.0.0.1:14317
      exporters: 
        otlp: 
          endpoint: 127.0.0.1:4317
          tls: 
            insecure: true
          retry_on_failure: 
            enabled: true
            initial_interval: 1s
            max_interval: 3s
            max_elapsed_time: 120s
          sending_queue: 
            enabled: true
            num_consumers: 1
            queue_size: 1000
      service: 
        pipelines: 
          logs: 
            receivers: [otlp]
            exporters: [otlp]
        telemetry: 
          logs: 
            level: info

      Result

      Exporting failed. Dropping data. {"error": "not retryable error: Permanent error: rpc error: code = ResourceExhausted desc = export rate limit exceeded", "dropped_items": 1}
      
      Check Result
      Requests over limit 10 of 70 dropped, no retry
      After 65 seconds dropped requests are not sent again
      Proxy stopped for about 10 seconds (UNAVAILABLE) Collector retries ("Will retry the request after interval"), no drop after proxy start

      Expected

      Proxy responds with UNAVAILABLE, as documentation says:

      If the limit is exceeded, the proxy responds with the gRPC status code UNAVAILABLE (temporary error), so the sender may retry sending the message later, depending on its configuration.

            Assignee:
            Zabbix Development Team
            Reporter:
            Piotr Wegrzyn
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

              Created:
              Updated: