Steps to reproduce:

1. Zabbix with proxy
2. Active agent with proxy referred to with FQDN/DNS name
3. Observe outgoing DNS requests on agent host
   • E.g. tcpdump -i eth0 -l dst port 53
4. Reject incoming TCP from agent host on proxy host
   • E.g. iptables -I INPUT 1 -s 1.2.3.4 -j REJECT

Result:

Agent will go into a failed state with the log:

active check data upload to [proxy.example.com:10051] started to fail ([connect] cannot connect to [[proxy.example.com]:10051]: [111] Connection refused)

Observe that the agent host sends an A and AAAA DNS request for the proxy each second (looks like this to me. I assume that the agent tries to reconnect every second).

If you happen to get into a state where this happens for every agent you might be dealing with thousands/tens of thousands DNS requests each second. This could be considered a DDOS of the DNS server if every agent host has the same DNS server(s). This is of course setup dependent.

Expected:

The agent should somehow throttle down the reconnection attempts or have some kind of cooldown system when going into this failure state. Maybe this "reconnection throttle" could be added to the agent config (with a default much higher than every second).

Additional:

This happens when the proxy host actively refuses the connection. Stopping the proxy service might lead to the same thing as rejecting the connection in the firewall. As far as i know both RHEL and Ubuntu will actively refuse a connection if it's open in the firewall, but no services are listening on the port. Other OSes/distros might behave differently.