-
Type:
Problem report
-
Resolution: Unresolved
-
Priority:
Trivial
-
None
-
Affects Version/s: 8.0.0
-
Component/s: Server (S)
-
None
-
Environment:Ubuntu 26.04, PostgreSQL 18.6, Zabbix server 8.0.0 (official packages), ~48k items in history cache
Steps to reproduce:
1. Keep HistoryIndexCacheSize at its default value (4M) on a server with a large number of items (~48k items in history cache).
2. Have some items that return large values, for example agent2 mssql.perfcounter.get (~2 MB per poll).
3. Start the server and wait until the history index cache is exhausted (about 5 minutes after every start in our case).
Result:
The preprocessing manager thread (inside "supervisor #1") hits out of memory while flushing history, logs the error and exits, but it still holds the history cache mutex. After that the server never exits:
- all trappers stay "processing data" and history pollers stop, so data collection stops completely
- "systemctl stop zabbix-server" hangs; only SIGKILL helps
Log:
preprocessing manager #1 file:cachehistory.c,line:2854 __zbx_shmem_realloc(): out of memory (requested 706696 bytes)
preprocessing manager #1 file:cachehistory.c,line:2854 __zbx_shmem_realloc(): please increase HistoryIndexCacheSize configuration parameter
a supervisor unit has exited, terminating
forced supervisor unit shutdown
Backtrace logged by preprocessing manager:
zbx_pp_manager_thread -> zbx_dc_flush_history -> zbx_hashset_insert_ext -> zbx_hashset_reserve -> __zbx_shmem_realloc -> zbx_backtrace
Blocked processes (gdb on trapper):
zbx_dc_flush_history -> __zbx_mutex_lock -> futex wait
Mutex owner TID (pthread __owner) = the preprocessing manager thread, which no longer exists in /proc.
Expected:
The server terminates cleanly with the "please increase HistoryIndexCacheSize" error (as in previous versions), so systemd can restart it. The process should not deadlock on a mutex owned by a dead thread.
Workaround:
HistoryIndexCacheSize=128M. Current usage is ~1.7M of 128M and the hangs have stopped.