Type: Incident report
Resolution: Workaround proposed
Affects Version/s: 3.4.2, 3.4.3
Fix Version/s: None
Component/s: Server (S)
Environment:Operating system: Red Hat Enterprise Linux Server release 7.4 (Maipo)
Hardware: UCSB-B200-M4, 64GB Memory, 1Socket/12Cores
Database: Percona Server 5.7 (local)
Latest software as well as firmware updates applied.
since we updated our Zabbix Server to version 3.4 we are having a lot of trouble regarding the memory usage of preprocessing manager. The memory usage of this process sporadically raises up to 90% of total server memory and results in OOM process kills on OS level -> In most cases the server itself or the database are beeing killed with a loss of data in result.
Strange fact: This mostly happens on saturdays (I already checked cron-/anacrontab but couldn't find anything).
Steps to reproduce:
- High CPU and memory usage of preprocessing manager (see "atop_preprocessingmem.PNG" and "atop_proprocessingcpu.PNG")
- OOM Killer killing processes (see "oom_report.png")
- Loss of data within zabbix (see "Dataloss.png")
Seems like I am not the only one with this issue: https://www.zabbix.com/forum/showthread.php?p=203975#post203975
Tomorrow I will raise the debug level for preprocessing process as mentioned in
Thanks in advance!