If host has IPv4 and IPv6 address, both addressess are resolvable to A and AAAA records, only one of them is used as zabbix server address.
$ host zabbix.server.dns
184.108.40.206 (IPv4 address of zabbix server)
xxxx::xxxx (IPv6 address of zabbix server)
Now, when connection comes to IPv4 address, it is rejected as unauthorized IP. Address should be resolved to ipv4 and ipv6 addresses if they exists.