-
Type:
Problem report
-
Resolution: Unresolved
-
Priority:
Major
-
None
-
Affects Version/s: 8.0.0
-
Component/s: Proxy (P)
Environment
- Zabbix server and proxy 8.0.0, proxy with "StartAPMCollector=1"
- ClickHouse 25.8 behind HTTP load balancer, which returns 503 while backend is unavailable
Steps to reproduce
- Create proxy "APM proxy" with APM data collection enabled. Set proxy "TelemetryProvider" to load balancer URL.
- Create host "APM proxy health", monitored by this proxy, with internal item "APM export statistics", key "zabbix[apm,export]", interval "10s".
- Make load balancer return 503 and send one OTLP log to proxy.
- Make load balancer forward requests again and send one more log.
- Check proxy log, load balancer requests and item value.
Result
got failure response ClickHouse: Service Unavailable (503)
| Check | Result |
|---|---|
| Requests for first log at load balancer | 1 (503), no retry |
| "rows.logs" after second log | 2, only 1 log was forwarded to ClickHouse |
| Load balancer stopped (connection error) | proxy retries ("APM database is down, retrying"), log is forwarded after load balancer start |
Expected
Proxy retries insert rejected with HTTP 503, as it does for connection errors. "rows.logs" counts only rows accepted by ClickHouse.