zabbix-agentd is listening on :: (all interfaces, ipv6-system). If a connection comes in as ipv4 it gets masked as ::ffff:192.168.0.1, this will not be recognized by zabbix-agentd if the allowed server is only a hostname or the ipv4-ip. So in this scenario these entries don't work:
Server=zabbix
Server=192.168.0.1
Server=zabbix,192.168.0.1

But this one does:
Server=::ffff:192.168.0.1

Also giving a IPv4 listening-address solves the problem.