Zabbix Server 7.4.11 - Continuous Memory Growth in History Poller Process (leading to OOM)

Summary On a production Zabbix server (v7.4.11), the history poller process shows continuous, unbounded RSS memory growth over time, eventually consuming enough memory (combined with other server processes) to trigger a Linux OOM-kill of the zabbix_server process. This has been observed across two separate incidents.

Environment

• Zabbix server version: 7.4.11
• OS: Linux (RHEL-family, kernel 5.14.x)
• Deployment: HA cluster (active/standby), issue observed on the currently active node
• System RAM: 64 GB
• Scale: approximately 812,000 active items (per diaginfo=valuecache item count)
• StartHistoryPollers: left at default value
• Cache sizes (CacheSize, HistoryCacheSize, HistoryIndexCacheSize, ValueCacheSize): each configured in the low single-digit GB range

Steps to reproduce

1. Run zabbix_server 7.4.11 in an environment with a large item count (800k+ active items), with StartHistoryPollers left at default and cache sizes each in the low single-digit GB range.
2. Let the server run under normal production load for several hours to days (a restart or HA failover to the affected node is a valid starting point).
3. Periodically sample per-subprocess RSS memory, either via "ps -eo pid,vsz,rss,cmd | grep zabbix_server", or via the built-in Zabbix self-monitoring items (proc.mem[\{$PROC_NAME},...,sum,"<process>",rss]), specifically isolating the history poller process type over time.
4. In parallel, run zabbix_server -R diaginfo=valuecache and diaginfo=historycache periodically to monitor shared cache usage.
5. Compare the RSS trend of history poller against other process types (trapper, history syncer, configuration syncer, preprocessing manager) over the same time window.

Result history poller RSS grows continuously without plateauing, and the growth rate accelerates over time rather than slowing down. Observed across three samples taken over several hours after a restart/HA failover:

• history poller: approx 6.8 GB -> 8.9 GB -> 22.9 GB (still climbing, accelerating, approx 788 MB/hour at last sample)
• trapper: approx 15.1 GB -> 18.6 GB -> 21.8 GB (plateaued after initial warm-up)
• history syncer: approx 10.2 GB -> 12.1 GB -> 13.3 GB (plateaued after initial warm-up)
• configuration syncer: approx 4.6 GB (stable across all samples)
• preprocessing manager: approx 4.4 GB (stable across all samples)
• all other process types: low, single/double-digit MB (stable across all samples)

All process types other than history poller show growth consistent with normal cache/working-set warm-up after a restart (rapid initial increase, then a flat plateau). history poller is the only process type that continues to grow without plateauing.

diaginfo=valuecache and diaginfo=historycache output taken during the growth period shows both caches far below their configured limits and not growing in proportion to the observed history poller RSS growth:

• Value cache: approx 4.26 GB configured, only approx 166 MB (about 4%) in use.
• History cache (data): approx 2 GB configured, 0 bytes in use.
• History cache (index): approx 2.1 GB configured, approx 47 MB in use. See attached files: diaginfo_valuecache.txt and diaginfo_historycache.txt.

This indicates the memory growth is internal to the history poller process itself (heap/private memory), not attributable to a shared cache segment being mapped into multiple processes.

Eventually, combined memory usage across process types exceeds available system RAM, and the kernel OOM-killer terminates zabbix_server: "zabbix_server invoked oom-killer: gfp_mask=0x140cca(GFP_HIGHUSER_MOVABLE|__GFP_COMP), order=0, oom_score_adj=0" - a routine single-page (4 KB) user-space allocation failed, indicating the system was already fully exhausted rather than failing on one large allocation. At the time of the first occurrence, zabbix_server RSS had grown to approximately 60 GB on a 64 GB system prior to the kill.

Immediately preceding the first OOM event: a large (approx 190 MB uncompressed) full configuration sync was sent to a proxy, and the housekeeper task started around the same time. Multiple proxies in one proxy group dropped offline (TLS handshake errors on inbound connections) within the same approx 15-second window as the OOM event - consistent with the server running out of memory to service new connections, rather than an unrelated network issue.

See attached: kernel OOM log excerpt, diaginfo output (valuecache/historycache), per-subprocess RSS/VSZ trend data, zabbix_server.log excerpts around both the OOM event and the ongoing growth period.

Expected history poller RSS should stabilize/plateau after initial cache and working-set warm-up, similar to the behavior observed in all other zabbix_server process types (trapper, history syncer, configuration syncer, preprocessing manager), rather than growing continuously and unboundedly until the process is OOM-killed.

Additional questions

1. Is there a known memory leak specific to the history poller process in 7.4.x?
2. What additional diagnostics would help isolate the leak (e.g., specific item types handled by history pollers - DB monitor, SSH, telnet, calculated items - that might be implicated)?
3. Are there any known interactions between HA failover/active-mode switching and history poller memory behavior?

Attachments

• diaginfo_valuecache.txt
• diaginfo_historycache.txt
• Kernel OOM log excerpt (below)
• zabbix_server.log excerpt (below)
• Per-subprocess ps snapshot shortly after restart (below)

Kernel OOM log excerpt: kernel: Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 11/12/2020 kernel: CPU: 5 PID: 1574581 Comm: zabbix_server Kdump: loaded Not tainted 5.14.0-687.23.1.el9_8.x86_64 #1 kernel: zabbix_server invoked oom-killer: gfp_mask=0x140cca(GFP_HIGHUSER_MOVABLE|__GFP_COMP), order=0, oom_score_adj=0

zabbix_server.log excerpt (leading up to the OOM event, hostnames/IPs redacted): sending configuration data to proxy at <redacted> at <redacted>, datalen 189969471, bytes 11447283 with compression ratio 16.6 executing housekeeper Proxy <redacted> changed state from online to offline Proxy <redacted> changed state from online to offline Proxy <redacted> changed state from online to offline Proxy <redacted> changed state from online to offline Proxy group <redacted> changed state from online to offline failed to accept an incoming connection: from <redacted>: unspecified certificate verification error: TLS handshake set result code to 1: file ssl/record/rec_layer_s3.c line 698 func : error:0A000126:SSL routines::unexpected eof while reading: TLS write fatal alert "decode error" Proxy <redacted> changed state from online to offline

Per-subprocess RSS snapshot (ps -eo pid,vsz,rss,cmd --sort=-rss, shortly after a restart, PIDs/hostnames redacted):

• configuration syncer [synced configuration in 52.314343 sec, idle 300 sec] - RSS approx 4.75 GB
• preprocessing manager #1 [queued 5155, processed 5161 values] - RSS approx 4.49 GB
• internal poller #1 - RSS approx 2.23 GB
• history syncer #4 [processed 129892 values] - RSS approx 1.61 GB
• history syncer #1 [processed 142178 values] - RSS approx 1.55 GB
• history syncer #2 [processed 165618 values] - RSS approx 1.41 GB
• history syncer #3 [processed 123121 values] - RSS approx 1.39 GB
• trapper #3 - RSS approx 1.12 GB
• trapper #6 - RSS approx 1.05 GB
• history poller #4 [got 1294 values in 4.998833 sec] - RSS approx 579 MB
• history poller #1 [got 1277 values in 4.997554 sec] - RSS approx 567 MB
• history poller #2 [got 1204 values in 4.999443 sec] - RSS approx 564 MB
• history poller #3 [got 1256 values in 5.009615 sec] - RSS approx 562 MB
• history poller #5 [got 1183 values in 4.993236 sec] - RSS approx 561 MB