Type: Incident report
Affects Version/s: 4.2.3
Fix Version/s: None
Component/s: Server (S)
Environment:Zabbix Server (RHEL 7.6), AWS EC2 c4.xlarge
Zabbix frontend (RHEL 7.6), AWS EC2 m4.large
Zabbix DB: AWS RDS (MariaDB 10.0.24), db.m4.xlarge
- Zabbix server trapper busy processes increased significantly from its typical value (~1-3%) to 100%.
- Zabbix proxies cannot connect to Zabbix server. Zabbix_proxy.log contains the following lines:
Agents that are monitored by Zabbix server itself stay green and data is being collected. Zabbix server logs contains no suspicious records, just info about items that became unsupported.
The environment stays in that condition for several hours and Zabbix server starts receiving data from Zabbix proxies only after a reboot via
It could be related to network issues, but we couldn't find any evidences.
Network traffic on Zabbix server:
It's unclear for me why Zabbix trappers suddenly increased to 100% and stuck at that value because TrapperTimeout=15 should prevent trappers from hanging. Could you please tell me what do you think about this situation? I would like to prevent such situation in the future, so maybe you'll point me at where else should I look in order to investigate this incident properly.