[ZBX-26589] DNS issue in 7.0.14 Created: 2025 Jun 19 Updated: 2025 Aug 19 Resolved: 2025 Jul 14 |
|
| Status: | Closed |
| Project: | ZABBIX BUGS AND ISSUES |
| Component/s: | Proxy (P), Server (S) |
| Affects Version/s: | None |
| Fix Version/s: | 7.0.16, 7.2.10, 7.4.1rc1, 8.0.0alpha1 |
| Type: | Problem report | Priority: | Major |
| Reporter: | FNkey | Assignee: | Vladislavs Sokurenko |
| Resolution: | Fixed | Votes: | 5 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | 2h | ||
| Original Estimate: | Not Specified | ||
| Attachments: |
|
| Team: | |
| Sprint: | S25-W26/27 |
| Story Points: | 0.25 |
| Description |
|
Steps to reproduce:
Result:
|
| Comments |
| Comment by FNkey [ 2025 Jun 19 ] |
|
Anomaly at screenshot of Pi-hole's dashboard:
Query rate increased from ~350 queries to ~12500 in 10 minutes range. After downgrade to 7.0.13 issue was resolved. |
| Comment by FNkey [ 2025 Jun 19 ] |
# grep zabbix-server /var/log/dpkg.log
2025-06-19 12:08:09 upgrade zabbix-server-mysql:amd64 1:7.0.13-1+debian12 1:7.0.14-1+debian12
2025-06-19 12:08:09 status half-configured zabbix-server-mysql:amd64 1:7.0.13-1+debian12
2025-06-19 12:08:09 status unpacked zabbix-server-mysql:amd64 1:7.0.13-1+debian12
2025-06-19 12:08:09 status half-installed zabbix-server-mysql:amd64 1:7.0.13-1+debian12
2025-06-19 12:08:09 status unpacked zabbix-server-mysql:amd64 1:7.0.14-1+debian12
2025-06-19 12:08:17 configure zabbix-server-mysql:amd64 1:7.0.14-1+debian12 <none>
2025-06-19 12:08:17 status unpacked zabbix-server-mysql:amd64 1:7.0.14-1+debian12
2025-06-19 12:08:19 status half-configured zabbix-server-mysql:amd64 1:7.0.14-1+debian12
2025-06-19 12:08:19 status installed zabbix-server-mysql:amd64 1:7.0.14-1+debian12
2025-06-19 21:59:03 upgrade zabbix-server-mysql:amd64 1:7.0.14-1+debian12 1:7.0.13-1+debian12
2025-06-19 21:59:03 status half-configured zabbix-server-mysql:amd64 1:7.0.14-1+debian12
2025-06-19 21:59:03 status unpacked zabbix-server-mysql:amd64 1:7.0.14-1+debian12
2025-06-19 21:59:03 status half-installed zabbix-server-mysql:amd64 1:7.0.14-1+debian12
2025-06-19 21:59:03 status unpacked zabbix-server-mysql:amd64 1:7.0.13-1+debian12
2025-06-19 21:59:04 configure zabbix-server-mysql:amd64 1:7.0.13-1+debian12 <none>
2025-06-19 21:59:04 status unpacked zabbix-server-mysql:amd64 1:7.0.13-1+debian12
2025-06-19 21:59:17 status half-configured zabbix-server-mysql:amd64 1:7.0.13-1+debian12
2025-06-19 21:59:18 status installed zabbix-server-mysql:amd64 1:7.0.13-1+debian12
|
| Comment by Tomasz Grzechulski [ 2025 Jun 20 ] |
|
Hello, Regards, |
| Comment by FNkey [ 2025 Jun 20 ] |
|
tgrzechulski , I have a simple installation - all Zabbix components on single VM and MySQL database on other host. Only changing zabbix-server version affect DNS. About configuration - basic templates are used: Additional templates: Website by Browser, Mikrotik SNMP, TrueNAS Core by SNMP, TrueNAS SCALE 22.12.1 and up by SNMP, Proxmox VE by HTTP. cat /etc/zabbix/zabbix_server.conf
LogFile=/var/log/zabbix/zabbix_server.log
LogFileSize=2
DebugLevel=3
PidFile=/run/zabbix/zabbix_server.pid
DBHost={{ DB host IP }}
DBName={{ DB name }}
DBUser={{ DB user }}
DBPassword={{ DB pass }}
DBPort=3306
StartPollers=10
StartIPMIPollers=0
StartPollersUnreachable=4
StartTrappers=1
StartPingers=2
StartDiscoverers=2
StartLLDProcessors=1
StartHTTPPollers=2
StartTimers=2
StartEscalators=1
StartJavaPollers=0
StartVMwareCollectors=0
StartBrowserPollers=1
SNMPTrapperFile=/tmp/zabbix_traps.tmp
StartSNMPTrapper=1
HousekeepingFrequency=12
MaxHousekeeperDelete=25000
CacheSize=128M
HistoryCacheSize=128M
HistoryIndexCacheSize=64M
TrendCacheSize=32M
ValueCacheSize=64M
Timeout=30
TrapperTimeout=300
AlertScriptsPath=/usr/lib/zabbix/alertscripts
ExternalScripts=/usr/lib/zabbix/externalscripts
FpingLocation=/usr/bin/fping
Fping6Location=/usr/bin/fping6
LogSlowQueries=3000
StartProxyPollers=1
ProxyConfigFrequency=300
ProxyDataFrequency=5
WebDriverURL={{ driver IP-PORT }}
cat /etc/resolv.conf
domain dom.XXXXXXX
search dom.XXXXXXX
nameserver {{ Pi-hole DNS IP }}
nameserver {{ Second DNS IP }}
nameserver {{ Third DNS IP }}
Logs at /var/log/zabbix/zabbix_server.log doesn't have any records about problem at DebugLevel=3 |
| Comment by FNkey [ 2025 Jun 20 ] |
|
In debug log I see nothing related to problems with communication with Zabbix agents or web-services. Only some errors with SNMP-checks. lsb_release -a No LSB modules are available. Distributor ID: Debian Description: Debian GNU/Linux 12 (bookworm) Release: 12 Codename: bookworm |
| Comment by FNkey [ 2025 Jun 20 ] |
|
The most strange in behavior of version 7.0.14 is that Zabbix Server tries to resolve IP addresses. It looks like zabbix server 7.0.14 mixed up the fields "IP address" and "DNS name" in every Host configuration record. |
| Comment by FNkey [ 2025 Jun 20 ] |
|
For example - almost 900 broken DNS queries for Proxmox IP in 5-10 minutes when I've tested 7.0.14 with debug log level:
|
| Comment by Zeltron [ 2025 Jun 21 ] |
|
Hello everybody, Encountering exactly the same issue with a current 7.2.9 version:. Zabbix Server is flooding DNS server perpetually with the same (bad?) requests which are a few IP addresses (NOT a in-addr.arpa request): root@zabbix:~# tcpdump -i ens18 port 53 18:12:26.227217 IP zabbix.38139 > dns1.domain: 26979+ A? 192.168.86.63. (31) 18:12:26.227229 IP zabbix.38139 > dns1.domain: 25191+ AAAA? 192.168.86.63. (31) 18:12:26.227242 IP zabbix.38139 > dns1.domain: 30611+ A? 172.18.5.55. (29) 18:12:26.227244 IP zabbix.38139 > dns1.domain: 23368+ AAAA? 172.18.5.55. (29) 18:12:26.227253 IP zabbix.38139 > dns1.domain: 45097+ A? 192.168.86.61. (31) 18:12:26.227259 IP zabbix.38139 > dns1.domain: 62821+ AAAA? 192.168.86.61. (31) 18:12:26.227268 IP zabbix.38139 > dns1.domain: 56207+ A? 172.18.35.50. (30) 18:12:26.227270 IP zabbix.38139 > dns1.domain: 38778+ AAAA? 172.18.35.50. (30) 18:12:26.227278 IP zabbix.38139 > dns1.domain: 60569+ A? 172.18.5.55. (29) 18:12:26.227280 IP zabbix.38139 > dns1.domain: 25431+ AAAA? 172.18.5.55. (29) 18:12:26.227603 IP dns1.domain > zabbix.38139: 26979 NXDomain 0/1/0 (106) 18:12:26.227621 IP dns1.domain > zabbix.38139: 30611 NXDomain 0/1/0 (104) 18:12:26.227621 IP dns1.domain > zabbix.38139: 23368 NXDomain 0/1/0 (104) 18:12:26.227649 IP dns1.domain > zabbix.38139: 45097 NXDomain 0/1/0 (106) 18:12:26.227649 IP dns1.domain > zabbix.38139: 62821 NXDomain 0/1/0 (106) 18:12:26.227654 IP dns1.domain > zabbix.38139: 56207 NXDomain 0/1/0 (105) 18:12:26.227665 IP dns1.domain > zabbix.38139: 38778 NXDomain 0/1/0 (105) 18:12:26.227682 IP dns1.domain > zabbix.38139: 60569 NXDomain 0/1/0 (104) 18:12:26.227682 IP dns1.domain > zabbix.38139: 25431 NXDomain 0/1/0 (104) 18:12:26.227822 IP dns1.domain > zabbix.38139: 25191 NXDomain 0/1/0 (106)
It all started after the upgrade from 7.2.7 to 7.2.8, and still occurs after upgrade to 7.2.9::
// espace réservé du coderoot@zabbix:~# dpkg -l | grep zabbix
ii zabbix-agent 1:7.2.9-1+debian12 amd64
ii zabbix-frontend-php 1:7.2.9-1+debian12 all
ii zabbix-nginx-conf 1:7.2.9-1+debian12 all
ii zabbix-release 1:7.2-1+debian12 all
ii zabbix-server-pgsql 1:7.2.9-1+debian12 amd64
ii zabbix-sql-scripts 1:7.2.9-1+debian12 all
Zabbix upgrade apt.log history: Start-Date: 2025-06-19 20:26:07 Commandline: apt upgrade Install: libc-ares2:amd64 (1.18.1-3, automatic) Upgrade: zabbix-sql-scripts:amd64 (1:7.2.7-1+debian12, 1:7.2.8-1+debian12), zabbix-server-pgsql:amd64 (1:7.2.7-1+debian12, 1:7.2.8-1+debian12), zabbix-agent: amd64 (1:7.2.7-1+debian12, 1:7.2.8-1+debian12), zabbix-nginx-conf:amd64 (1:7.2.7-1+debian12, 1:7.2.8-1+debian12), zabbix-frontend-php:amd64 (1:7.2.7-1+debian 12, 1:7.2.8-1+debian12) End-Date: 2025-06-19 20:26:35 Start-Date: 2025-06-20 23:45:38 Commandline: apt upgrade Upgrade: zabbix-sql-scripts:amd64 (1:7.2.8-1+debian12, 1:7.2.9-1+debian12), zabbix-server-pgsql:amd64 (1:7.2.8-1+debian12, 1:7.2.9-1+debian12), zabbix-agent: amd64 (1:7.2.8-1+debian12, 1:7.2.9-1+debian12), zabbix-nginx-conf:amd64 (1:7.2.8-1+debian12, 1:7.2.9-1+debian12), zabbix-frontend-php:amd64 (1:7.2.8-1+debian 12, 1:7.2.9-1+debian12) End-Date: 2025-06-20 23:45:40 How important is the DNS flood requests since 7.2.8 upgrade:
I will be happy to help and send any log/debug/trace that could help in the investigations |
| Comment by Markku Leiniö [ 2025 Jun 21 ] |
|
To be clear, this is not about the DNS entry configuration in the host interface settings, this also happens on 7.0.15 with plain IP address as the host interface address and no DNS name configured. Fortunately this issue is already in the Confirmed state. My current workaround: use local systemd-resolved resolver to cache the requests and not send them to the upstream DNS servers. |
| Comment by Mickael Martin [ 2025 Jun 23 ] |
|
I think I have the bug on zabbix_proxy (Zabbix) 7.2.9 too : |
| Comment by Vladislavs Sokurenko [ 2025 Jun 25 ] |
|
It looks like issue is related to the fact that packages are now built with ares "--with-ares" |
| Comment by Vladislavs Sokurenko [ 2025 Jun 25 ] |
|
I could not reproduce on ubuntu but must likely this is the reason:
diff --git a/src/libs/zbxasyncpoller/asyncpoller.c b/src/libs/zbxasyncpoller/asyncpoller.c
index 2a308f36adb..71b7c4f7282 100644
--- a/src/libs/zbxasyncpoller/asyncpoller.c
+++ b/src/libs/zbxasyncpoller/asyncpoller.c
@@ -333,7 +333,7 @@ void zbx_async_poller_add_task(struct event_base *ev, zbx_channel_t *channel, st
struct ares_addrinfo_hints hints = {.ai_family = PF_UNSPEC, .ai_socktype = SOCK_STREAM};
if (SUCCEED == zbx_is_supported_ip(addr))
- hints.ai_flags = AI_NUMERICHOST;
+ hints.ai_flags = ARES_AI_NUMERICHOST;
ares_getaddrinfo(channel, addr, NULL, &hints, ares_addrinfo_cb, task);
return;
|
| Comment by Vladislavs Sokurenko [ 2025 Jun 26 ] |
|
Fixed in pull request feature/ZBX-26589-7.0 |
| Comment by Vladislavs Sokurenko [ 2025 Jun 26 ] |
|
Fixed in:
|
| Comment by user185953 [ 2025 Jun 30 ] |
|
Thank you for noticing this issue so quickly FNkey (and sorry for ticket spam) ! You saved me lot of problems. |