|
I'm submitting this based on advice received in ZBXNEXT-5113.
I've run into an issue where our zabbix_server process crashes due to exhaustion of the configuration cache - specifically, lack of an available memory chunk large enough to fulfill the allocation request. A log message suggested increasing the cachesize, but we were already at the 8GB limit. I submitted a feature request, ZBXNEXT-5113, to have the limit increased and provided a proposed patch, requesting that it be reviewed for validity. However, I had also noticed that the configuration cache % free historical data showed 72% free. Reviewing the error messages indicated that the memory is highly fragmented.
memory of total size 12884901512 bytes fragmented into 22921433 chunks
of those, 10416940744 bytes are in 195410 free chunks
of those, 2101217856 bytes are in 22726023 used chunks
I applied the patch I had proposed, built, increased the configuration cache value from 8GB to 12 GB, and waited. Previously, with the 8GB cachesize, we were getting a zabbix server crash every 10-15 days (2-3 days with cachesize previously set to 4GB). With the 12GB cachesize, we went 21 days before a crash occurred (log snippet above is from this crash). At this point I've increased the cachesize to 16GB, but I feel this probably isn't the best fix, as it seems almost all of the configuration cache is tied up in unusable fragmented memory and will inevitably lead to another out of memory scenario.
Is there something that can be done to improve memory allocation so it doesn't become so fragmented? Thank you for your time.
2019-01-24 Incident - cachesize 4GB (similar instances occurred on 2019-01-22, 2019-01-18, 2019-01-15)
2019-03-14 Incident - cachesize 8GB (similar instances occurred on 2019-02-28, 2019-02-19, 2019-02-05)
2019-04-04 Incident - cachesize 12GB
|
configuration-cache-pctfree-30days.png
configuration_syncer.log
Team A
