According to the documentation, the configuration parameter 'server' is not used by the proxy in passive mode.
It should behave like the agent, the parameter containing a list of valid server IPs for passive mode, and the active mode using the first one.

I didn't check the source, so it may be a simple documentation error. (in the wiki and default proxy config)