Loading...

Type: Problem report
Resolution: Unresolved
Priority: Trivial
Fix Version/s: None
Affects Version/s: None
Component/s: Server (S)
Labels:
- crash
Environment:
ZABBIX VERSION 6.0.30 , operating system 24.04.2 LTS (Noble Numbat) , hardware 16 cpu 124 GB memory , Zabbix operating in cluster mode

Zabbix server switching cluster due to memory allocation problems in lld workers "_mem_malloc"

*ZABBIX SERVER - /etc/zabbix/zabbix_server.conf*

LogFile=/var/log/zabbix/zabbix_server.log
LogFileSize=0
PidFile=/run/zabbix/zabbix_server.pid
SocketDir=/run/zabbix
DBHost=zabbix-prd1
DBName=zabbix
DBUser=lzabbix
DBPassword=xxxxxxx
DBPort=3306
StartPollers=50
StartPollersUnreachable=10
StartHistoryPollers=100
StartTrappers=20
StartPingers=10
StartHTTPPollers=10
StartTimers=10
StartEscalators=60
StartAlerters=30
SNMPTrapperFile=/var/log/snmptrap/snmptrap.log
StartSNMPTrapper=1
HousekeepingFrequency=4
CacheSize=32G
CacheUpdateFrequency=120
StartDBSyncers=70
HistoryCacheSize=1G
HistoryIndexCacheSize=512M
TrendCacheSize=1G
TrendFunctionCacheSize=128M
ValueCacheSize=8G
Timeout=30
UnreachablePeriod=300
UnreachableDelay=30
AlertScriptsPath=/etc/zabbix/alertscripts
ExternalScripts=/etc/zabbix/externalscripts
FpingLocation=/usr/bin/fping
Fping6Location=/usr/bin/fping6
LogSlowQueries=3000
StartLLDProcessors=40
StartODBCPollers=5
HANodeName=zabbix-server01
NodeAddress=zabbix-master01

 

*Error in  2025/11/28 - /var/log/zabbix/zabbix_server.log*


3163991:20251128:163130.290 __mem_malloc: skipped 251628 asked 8612240 skip_min 256 skip_max 8612144
3163991:20251128:163130.330 === memory statistics for configuration cache ===
3163991:20251128:163130.330 free chunks of size     24 bytes:        7
3163991:20251128:163130.330 free chunks of size     32 bytes:       87
3163991:20251128:163130.330 free chunks of size     40 bytes:       21
3163991:20251128:163130.330 free chunks of size     48 bytes:        6
3163991:20251128:163130.330 free chunks of size     56 bytes:       20
3163991:20251128:163130.330 free chunks of size     64 bytes:        3
3163991:20251128:163130.330 free chunks of size     72 bytes:        2
3163991:20251128:163130.330 free chunks of size     80 bytes:        2
3163991:20251128:163130.330 free chunks of size     96 bytes:        1
3163991:20251128:163130.330 free chunks of size    104 bytes:        3
3163991:20251128:163130.330 free chunks of size    112 bytes:        1
3163991:20251128:163130.330 free chunks of size    136 bytes:        2
3163991:20251128:163130.330 free chunks of size    152 bytes:        1
3163991:20251128:163130.330 free chunks of size    176 bytes:        1
3163991:20251128:163130.330 free chunks of size >= 256 bytes:   251628
3163991:20251128:163130.330 min chunk size:         24 bytes
3163991:20251128:163130.330 max chunk size:    8612144 bytes
3163991:20251128:163130.330 memory of total size 33298412032 bytes fragmented into 66332873 chunks
3163991:20251128:163130.330 of those, 26570394288 bytes are in   251785 free chunks
3163991:20251128:163130.330 of those, 6728017744 bytes are in 66081088 used chunks
3163991:20251128:163130.330 of those, 1061325952 bytes are used by allocation overhead
3163991:20251128:163130.330 ================================
3163991:20251128:163130.330 === Backtrace: ===
3163991:20251128:163130.331 12: /usr/sbin/zabbix_server: lld worker #25 [processed 7 LLD rules, idle 4.460559 sec during 5.350972 sec](zbx_backtrace+0x51) [0x560b6ed702b1]
3163991:20251128:163130.331 11: /usr/sbin/zabbix_server: lld worker #25 [processed 7 LLD rules, idle 4.460559 sec during 5.350972 sec](__zbx_mem_malloc+0x68) [0x560b6ed6be28]
3163991:20251128:163130.331 10: /usr/sbin/zabbix_server: lld worker #25 [processed 7 LLD rules, idle 4.460559 sec during 5.350972 sec](zbx_hashset_insert_ext+0xbe) [0x560b6edb647e]
3163991:20251128:163130.331 9: /usr/sbin/zabbix_server: lld worker #25 [processed 7 LLD rules, idle 4.460559 sec during 5.350972 sec](DCstrpool_replace+0x5e) [0x560b6ed0cc2e]
3163991:20251128:163130.331 8: /usr/sbin/zabbix_server: lld worker #25 [processed 7 LLD rules, idle 4.460559 sec during 5.350972 sec](DCconfig_items_apply_changes+0xac) [0x560b6ed23d3c]
3163991:20251128:163130.331 7: /usr/sbin/zabbix_server: lld worker #25 [processed 7 LLD rules, idle 4.460559 sec during 5.350972 sec](lld_worker_thread+0x53f) [0x560b6ecb35df]
3163991:20251128:163130.331 6: /usr/sbin/zabbix_server: lld worker #25 [processed 7 LLD rules, idle 4.460559 sec during 5.350972 sec](zbx_thread_start+0x24) [0x560b6ed72a24]
3163991:20251128:163130.331 5: /usr/sbin/zabbix_server: lld worker #25 [processed 7 LLD rules, idle 4.460559 sec during 5.350972 sec](+0x786b7) [0x560b6ebef6b7]
3163991:20251128:163130.331 4: /usr/sbin/zabbix_server: lld worker #25 [processed 7 LLD rules, idle 4.460559 sec during 5.350972 sec](MAIN_ZABBIX_ENTRY+0x709) [0x560b6ebf0739]
3163991:20251128:163130.331 3: /usr/sbin/zabbix_server: lld worker #25 [processed 7 LLD rules, idle 4.460559 sec during 5.350972 sec](main+0x4cf) [0x560b6ebe76ef]
3163991:20251128:163130.331 2: /lib/x86_64-linux-gnu/libc.so.6(+0x2a1ca) [0x7e59e322a1ca]
3163991:20251128:163130.331 1: /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x8b) [0x7e59e322a28b]
3163991:20251128:163130.331 0: /usr/sbin/zabbix_server: lld worker #25 [processed 7 LLD rules, idle 4.460559 sec during 5.350972 sec](_start+0x25) [0x560b6ebeeb25]
3163991:20251128:163130.331 [file:dbconfig.c,line:92] __zbx_mem_malloc(): out of memory (requested 8612233 bytes)
3163991:20251128:163130.331 [file:dbconfig.c,line:92] __zbx_mem_malloc(): please increase CacheSize configuration parameter

 

*Error in  2025/12/09 - /var/log/zabbix/zabbix_server.log*

939792:20251209:032943.720 __mem_malloc: skipped 208284 asked 9997576 skip_min 256 skip_max 9997480
939792:20251209:032943.752 === memory statistics for configuration cache ===
939792:20251209:032943.752 free chunks of size     24 bytes:       46
939792:20251209:032943.752 free chunks of size     32 bytes:       25
939792:20251209:032943.752 free chunks of size     40 bytes:        5
939792:20251209:032943.752 free chunks of size     48 bytes:        5
939792:20251209:032943.752 free chunks of size     56 bytes:       10
939792:20251209:032943.752 free chunks of size     64 bytes:        4
939792:20251209:032943.752 free chunks of size     72 bytes:        2
939792:20251209:032943.752 free chunks of size     88 bytes:        1
939792:20251209:032943.752 free chunks of size     96 bytes:        4
939792:20251209:032943.752 free chunks of size    104 bytes:        1
939792:20251209:032943.752 free chunks of size    112 bytes:        1
939792:20251209:032943.752 free chunks of size    128 bytes:        1
939792:20251209:032943.752 free chunks of size    136 bytes:        1
939792:20251209:032943.752 free chunks of size    168 bytes:        1
939792:20251209:032943.752 free chunks of size    192 bytes:        1
939792:20251209:032943.752 free chunks of size >= 256 bytes:   208284
939792:20251209:032943.752 min chunk size:         24 bytes
939792:20251209:032943.752 max chunk size:    9997480 bytes
939792:20251209:032943.752 memory of total size 33278894672 bytes fragmented into 67552708 chunks
939792:20251209:032943.752 of those, 26431942696 bytes are in   208392 free chunks
939792:20251209:032943.752 of those, 6846951976 bytes are in 67344316 used chunks
939792:20251209:032943.752 of those, 1080843312 bytes are used by allocation overhead
939792:20251209:032943.752 ================================
939792:20251209:032943.752 === Backtrace: ===
939792:20251209:032943.753 12: /usr/sbin/zabbix_server: lld worker #16 [processed 5 LLD rules, idle 6.424145 sec during 6.740821 sec](zbx_backtrace+0x51) [0x55eb0e3412b1]
939792:20251209:032943.753 11: /usr/sbin/zabbix_server: lld worker #16 [processed 5 LLD rules, idle 6.424145 sec during 6.740821 sec](__zbx_mem_malloc+0x68) [0x55eb0e33ce28]
939792:20251209:032943.753 10: /usr/sbin/zabbix_server: lld worker #16 [processed 5 LLD rules, idle 6.424145 sec during 6.740821 sec](zbx_hashset_insert_ext+0xbe) [0x55eb0e38747e]
939792:20251209:032943.753 9: /usr/sbin/zabbix_server: lld worker #16 [processed 5 LLD rules, idle 6.424145 sec during 6.740821 sec](DCstrpool_replace+0x5e) [0x55eb0e2ddc2e]
939792:20251209:032943.753 8: /usr/sbin/zabbix_server: lld worker #16 [processed 5 LLD rules, idle 6.424145 sec during 6.740821 sec](DCconfig_items_apply_changes+0xac) [0x55eb0e2f4d3c]
939792:20251209:032943.753 7: /usr/sbin/zabbix_server: lld worker #16 [processed 5 LLD rules, idle 6.424145 sec during 6.740821 sec](lld_worker_thread+0x53f) [0x55eb0e2845df]
939792:20251209:032943.753 6: /usr/sbin/zabbix_server: lld worker #16 [processed 5 LLD rules, idle 6.424145 sec during 6.740821 sec](zbx_thread_start+0x24) [0x55eb0e343a24]
939792:20251209:032943.753 5: /usr/sbin/zabbix_server: lld worker #16 [processed 5 LLD rules, idle 6.424145 sec during 6.740821 sec](+0x786b7) [0x55eb0e1c06b7]
939792:20251209:032943.753 4: /usr/sbin/zabbix_server: lld worker #16 [processed 5 LLD rules, idle 6.424145 sec during 6.740821 sec](MAIN_ZABBIX_ENTRY+0x709) [0x55eb0e1c1739]
939792:20251209:032943.753 3: /usr/sbin/zabbix_server: lld worker #16 [processed 5 LLD rules, idle 6.424145 sec during 6.740821 sec](main+0x4cf) [0x55eb0e1b86ef]
939792:20251209:032943.753 2: /lib/x86_64-linux-gnu/libc.so.6(+0x2a1ca) [0x7ba98a42a1ca]
939792:20251209:032943.753 1: /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x8b) [0x7ba98a42a28b]
939792:20251209:032943.753 0: /usr/sbin/zabbix_server: lld worker #16 [processed 5 LLD rules, idle 6.424145 sec during 6.740821 sec](_start+0x25) [0x55eb0e1bfb25]
939792:20251209:032943.753 [file:dbconfig.c,line:92] __zbx_mem_malloc(): out of memory (requested 9997569 bytes)
939792:20251209:032943.753 [file:dbconfig.c,line:92] __zbx_mem_malloc(): please increase CacheSize configuration parameter
3807155:20251209:032943.948 One child process died (PID:939792,exitcode/signal:1). Exiting ...
3807155:20251209:032944.699 PROCESS EXIT: 939792
939502:20251209:032944.699 HA manager has been paused
939502:20251209:032947.796 HA manager has been stopped
3807155:20251209:032948.218 syncing history data...

- - Sort By Name
  - Sort By Date
  - Ascending
  - Descending
  - Thumbnails
  - List
  - Download All

image-2025-12-11-12-44-34-893.png
68 kB
2025 Dec 11 17:44

duplicates

ZBX-27113 Performance impact and memory fragmentation on item unsupported error

Closed

Details

Description

Attachments

Attachments

Issue Links

Activity

People

Dates