According to the manual page Proxy load balancing and high availability the agent configuration Server and ServerActive must/should contain all proxies of the proxy group.

Instead of adding each of them I would just use a DNS record which points to the IP addresses of the proxies. This would also make changes easier to the proxy group much easier since one would just have to adjust the DNS record.
It worked with a test setup but I'm not sure if this is just because of DNS record randomization.

So I ask you to add some information in this regard to the manual. It should be clear if this is expected to work or not.