Ubuntu Linux running Zabbix 5 LTS
- Zabbix Server 5 LTS (Linux), MySQL DB
- Proxy configured using PSK identity and PSK string
- Proxy address is configured as a list of 3 IP addresses (comma-separated)
- Zabbix Proxy 5 installed on a 3-node cluster
- 3 individual IPs (no VIP)
- Proxies configured using identical PSK and PSK-identity
- Proxy only running on one of the 3 nodes at a time
Steps to reproduce:
- Set up 3 Zabbix proxy hosts with individual IPs which work in cluster mode
- Configure 1 Zabbix proxy on Zabbix server side using comma-separted list of proxy nodes in "Proxy address" configuration
- Enable proxy encryption (PSK, not tried yet if issue also occurs when certificates are used)
- Configure some items to be fetched via proxy
- Run the system and let one of the proxies start and connect to the server (the other 2 are shut down)
- Watch latest data to be collected
- Shut down proxy on the active proxy node and run it on any of the other 3 proxy nodes (with identical PSK configuration)
- Watch latest data on server
After a proxy switch (move to one of the other allowed IP addresses) there is no data inserted into Zabbix DB from this proxy any more. It just stops working. However in the logs of the server and proxy we see that the proxy is communicating and also fetching configuration data successfully.
Zabbix proxy seems to properly collect item data and trying to send it to Zabbix server but Zabbix server likely rejecting the data so they never get persisted.
A restart of Zabbix server or also just doing a "check now" on any item from this proxy resolving the problem until the next cluster resource move.
Also after resolving the problem all missing data seems to appear in the database but I assume this is due to the fact the proxy is still caching all those values and pushing them to the proxy again.
Note: After disabling PSK encryption a resource move to any host is showing expected bahavior (proxy moves to another host, Zabbix servre accepting data).
Expected bahavior would be that in case the proxy is configured with 3 "proxy address" configuration a move to any of those 3 addresses would not cause any issues.
There might be some issues with encryption handshake and session keys renewal if the proxy re-connects to the server. Specifically if the re-connect appears from a different source (one of the allowed sources of course).