Ubuntu 10.04

indeed, zabbix daemons don't do any dns caching, thus ip usage is suggested. alternatively, nscd or similar name server caching daemons could be used to reduce the amount of remote dns queries

Just a notice in case someone tries to use nscd in Debian or Ubuntu: hosts lookups are not cached by default.

http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=335476

http://sourceware.org/bugzilla/show_bug.cgi?id=4428

You must edit /etc/nscd.conf to enable it, but it is not that reliable.

I have just noticed that zabbix_server has the same issue. When connecting to hosts using DNS name, for every item queried there is name a resolution call.

Has there been any solutions to this? We are experiencing the same issues for Enterprise installations.

As mentioned before, use nscd.
Name service caching is task of the operating system respectively task of systems like nscd. Alternatively one could use a local DNS server for caching.

Caching domain names has no place to be within ZABBIX applications.

I've just checked zabbix agent (2.0.5) under Windows XP x32 and Windows 7 x64 - grab packets by wireshark.
As expected there no problems - windows caches resolved DNS names by default.
It's true as for requesting list of active checks (ServerActive=DNSname) and for agent passive incoming TCP connections (Server=DNSname)

The are windows build-in useful commands which are self-descriptive:
> ipconfig /displaydns
> ipconfig /flushdns

Just a note - some 3rd party software can optionally? disable the caching.
I know at least one - Kaspersky antivirus provides possibility to disable it during installation.

I've performed some additional experiments under Debian 6.0.7
Linux it0 3.2.0-3-amd64 #1 SMP Mon Jul 23 02:45:17 UTC 2012 x86_64 GNU/Linux
Agent is ~2.0.5 compiled with IPv6 support. I'm using only IPv4 addresses in the experiments.

Every time when an agent need to know Server's IP the agent host sends two DNS queries simultaneously: "Type: A (Host address)" and "Type: AAAA (IPv6 address)" and then gets two DNS answers respectively. (ZBX-4252 also mentions this and with some details)

For active checks: if BufferSend is default 5 seconds then agent 12 times per one minute performs queries to DNS server + one query for list of active checks.
This is true disregarding on count of monitored items and their update interval (I hope you don't overflow agent's BufferSize here )
So not so bad

But it more critical for passive checks. For every incoming TCP connection from zabbix server agent performs separate query to DNS server.
So you can calculate agent_items*update interval=number of queries for a period from every agent host to DNS server.

ONLY about agent:

What if will cache only IP of resolved "Server" and "ServerActive" parameters and will refresh it with some "hardcoded|configured|TTL from DNS response" period ?

It will be maybe 1-5 values (remember about multiple servers for active checks support) which agent has to handle and it will resolve current issue.
It should be not so hard to implement, IMO.

I don't see much sense to cache any other IPs possibly used in items.

I've just tested agent on a FreeBSD 8.1 x32 host.
Agent is also compiled with IPv6 support.

Behavior is the same - DNS query for every incoming TCP connection from server.
But order for queries a bit differs from Linux. Here is IPv4 query-response first and then IPv6 ones:

01:50:16.930705 IP 10.20.0.20.20708 > 10.20.0.10.53: 21340+ A? mon2 (33)
01:50:16.930886 IP 10.20.0.10.53 > 10.20.0.20.20708: 21340* 1/0/0 A 10.20.0.32 (49)
01:50:16.930929 IP 10.20.0.20.13973 > 10.20.0.10.53: 21341+ AAAA? mon2 (33)
01:50:16.931051 IP 10.20.0.10.53 > 10.20.0.20.13973: 21341* 0/1/0 (113)

I don't think I need to test something else.

Just additional small test on Debian.
Zabbix server v2.0.5 compiled with and without IPv6 support for single passive item check where host monitored by DNS name:
with IPv6 - performs two queries to DNS;
without IPv6 - performs one query to DNS.

heh, last comment about zabbix sources.

There is some difference - with IPv6 zabbix daemons use "getaddrinfo()" system call, but without IPv6 - another system call - "gethostbyname()".
See /src/libs/zbxcomms/comms.c lines 282 and 348

This is also mentioned in ZBX-6326

if we ever look into implementing this, it should be very well documented and there must be a way to drop this internal dns cache (like we can reload config for server) - lately i've hit some other software doing its own dns caching, and that can be mighty confusing

This issue is somewhat connected to ZBXNEXT-1862. One thing I find worth mentioning, is, that different caching solutions offer different feature sets. I remember comparing nscd, dnsmasq and bind at some point. I think to remember that one would not cache PTR records, the other one would not cache MX records, which can be relevant on the Zabbix server machine. My opinion matches Marc's: I don't consider it a Zabbix issue.

Russian hosting provider RTCOMM have DNS rate limit settings for domains, so with standard linux template zabbix server fault to query agent with error "ZBX_TCP_READ() failed: [4] Interrupted system call". I think we need this feature ASAP.

Lab:

• LAN
• ~100 agents
• 2 DNS server (master-slave)
• 1 zabbix-server 3.0.2 + mysql

Works for me:
1. Zabbix Server-related overload solve by nscd
2. Zabbix client-related overload by:

• setting short hostname in Server= and ServerActive= (like Server=zabbix instead of Server=zabbix.mydomain)
• enshure that /etc/resolv.conf has full domain at first place in 'search' parameter (like 'mydomain.myprovider myprovider', but not 'myprovider mydomain.myprovider'
• add 'option rotate' to resolv.conf

Agent config + correct 'search' parameter make first zabbix agent lookup success. So, it doesnt try other suffixes (only 2 request per connection)
'option rotate' balance loading between master and slave DNS (may not affect on zabbix agent, but balance other requests)

Other way: configure nscd on each agent host. But nscd may cause of mistiming between real network and nscd cache
On zabbix server nscd's negative cache with 20min TTL increase unreachable pollers load

I agree with caching of IP addresses of hostnames supplied on "Server=" and "ServerActive=" parameters (Oleksiy Zagorskyi's suggestion above). For example:

• When Zabbix agent is initializing and parsing its configuration file, it could resolve hostnames set by the administrator and keep IP addresses recorded in memory until the agent is killed by a TERM, HUP or USR1 signal.

• When Zabbix agent accepts a connection request from a server or proxy that is willing to perform a passive check, it could match the client address to one of the cached IP addresses.

The network I manage has a Zabbix deployment composed by a Zabbix server, some geographically dispersed Zabbix proxies, more than 2700 monitored hosts and more than 400000 monitored items. Until my team implements Zabbix muthual authentication, we are relying on hostnames specified in "Server=" agentd parameter to provide monitoring access control. However, Zabbix infrastructure is currently consuming a relevant amount of resources from our internal DNS servers, because the absence of a cache produces one name resolution query for every passive metric collect.

Another approach is to use systemd-resolved if you are on systemd

Despite the benefits of caching systems like NSCD or Systemd-Resolved, it seems that some applications can bypass the host's cache, and Zabbix may be one of them. This is unconfirmed and could not be validated. However, my tests show that it may be true.

Therefore, I believe it would be beneficial if Zabbix processes did not bypass the caching systems.

For passive checks and snmp this should already work after ZBX-26014 is implemented and latest c-ares libraries (As of c-ares 1.31.0), but need to be enabled in older libraries with the following patch:

diff --git a/src/libs/zbxpoller/async_poller.c b/src/libs/zbxpoller/async_poller.c
index ecd4938d60b..dd0f5936963 100644
--- a/src/libs/zbxpoller/async_poller.c
+++ b/src/libs/zbxpoller/async_poller.c
@@ -556,7 +556,7 @@ static void async_poller_dns_init(zbx_poller_config_t *poller_config, zbx_thread
 {
        char                    *timeout;
 #ifdef HAVE_ARES
-       struct ares_options     options;
+       struct ares_options     options = {0};
        int                     optmask, status;
 
        status = ares_library_init(ARES_LIB_INIT_ALL);
@@ -567,10 +567,11 @@ static void       async_poller_dns_init(zbx_poller_config_t *poller_config, zbx_thread
                exit(EXIT_FAILURE);
        }
 
-       optmask = ARES_OPT_SOCK_STATE_CB|ARES_OPT_TIMEOUT;
+       optmask = ARES_OPT_SOCK_STATE_CB|ARES_OPT_TIMEOUT|ARES_OPT_QUERY_CACHE;
        options.sock_state_cb = sock_state_cb;
        options.sock_state_cb_data = poller_config;
        options.timeout = poller_args_in->config_comms->config_timeout;
+       options.qcache_max_ttl = 3600;
 
        status = ares_init_options(&poller_config->channel, &options, optmask);