Environment
- Zabbix server and proxy 8.0.0, proxy with "StartAPMCollector=1"
- OpenTelemetry Collector 0.123.0, OTLP/gRPC receiver and exporter
Steps to reproduce
- Create proxy "proxy test". On APM tab enable data collection and set Max messages per second to "1".
- Start Collector with block 1. Its receiver listens on 14317, because proxy uses 4317.
- Restart proxy. Within 60 seconds send 70 OTLP log requests to Collector, one log each, every 100 ms.
- Wait 65 seconds and check Collector log.
1. Collector configuration
receivers: otlp: protocols: grpc: endpoint: 127.0.0.1:14317 exporters: otlp: endpoint: 127.0.0.1:4317 tls: insecure: true retry_on_failure: enabled: true initial_interval: 1s max_interval: 3s max_elapsed_time: 120s sending_queue: enabled: true num_consumers: 1 queue_size: 1000 service: pipelines: logs: receivers: [otlp] exporters: [otlp] telemetry: logs: level: info
Result
Exporting failed. Dropping data. {"error": "not retryable error: Permanent error: rpc error: code = ResourceExhausted desc = export rate limit exceeded", "dropped_items": 1}
| Check | Result |
|---|---|
| Requests over limit | 10 of 70 dropped, no retry |
| After 65 seconds | dropped requests are not sent again |
| Proxy stopped for about 10 seconds (UNAVAILABLE) | Collector retries ("Will retry the request after interval"), no drop after proxy start |
Expected
Proxy responds with UNAVAILABLE, as documentation says:
If the limit is exceeded, the proxy responds with the gRPC status code UNAVAILABLE (temporary error), so the sender may retry sending the message later, depending on its configuration.