ExportXMLWordPrintable

    • Type: Problem report
    • Resolution: Unresolved
    • Priority: Trivial
    • None
    • Affects Version/s: 8.0.0
    • Component/s: Server (S)
    • None
    • Environment:
      Ubuntu 26.04, PostgreSQL 18.6, Zabbix server 8.0.0 (official packages), ~48k items in history cache

      Steps to reproduce:
      1. Keep HistoryIndexCacheSize at its default value (4M) on a server with a large number of items (~48k items in history cache).
      2. Have some items that return large values, for example agent2 mssql.perfcounter.get (~2 MB per poll).
      3. Start the server and wait until the history index cache is exhausted (about 5 minutes after every start in our case).

      Result:
      The preprocessing manager thread (inside "supervisor #1") hits out of memory while flushing history, logs the error and exits, but it still holds the history cache mutex. After that the server never exits:

      • all trappers stay "processing data" and history pollers stop, so data collection stops completely
      • "systemctl stop zabbix-server" hangs; only SIGKILL helps

      Log:
      preprocessing manager #1 file:cachehistory.c,line:2854 __zbx_shmem_realloc(): out of memory (requested 706696 bytes)
      preprocessing manager #1 file:cachehistory.c,line:2854 __zbx_shmem_realloc(): please increase HistoryIndexCacheSize configuration parameter
      a supervisor unit has exited, terminating
      forced supervisor unit shutdown

      Backtrace logged by preprocessing manager:
      zbx_pp_manager_thread -> zbx_dc_flush_history -> zbx_hashset_insert_ext -> zbx_hashset_reserve -> __zbx_shmem_realloc -> zbx_backtrace

      Blocked processes (gdb on trapper):
      zbx_dc_flush_history -> __zbx_mutex_lock -> futex wait
      Mutex owner TID (pthread __owner) = the preprocessing manager thread, which no longer exists in /proc.

      Expected:
      The server terminates cleanly with the "please increase HistoryIndexCacheSize" error (as in previous versions), so systemd can restart it. The process should not deadlock on a mutex owned by a dead thread.

      Workaround:
      HistoryIndexCacheSize=128M. Current usage is ~1.7M of 128M and the hangs have stopped.

            Assignee:
            Aigars Kadikis
            Reporter:
            Ramazan Erdoğan
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

              Created:
              Updated: