[ZBX-22136] After the zabbix-server exits abnormally, it cannot start normally. Created: 2022 Dec 23  Updated: 2023 Apr 19  Resolved: 2023 Apr 19

Status: Closed
Project: ZABBIX BUGS AND ISSUES
Component/s: Server (S)
Affects Version/s: 6.2.6
Fix Version/s: None

Type: Problem report Priority: Trivial
Reporter: liuhua Assignee: Karlis Salins
Resolution: Commercial support required Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

zabbix 6.2.6
ubuntu 22.04

Attachments: PNG File 2022-12-23_184323.png     PNG File image-2023-02-01-16-14-41-712.png     PNG File image-2023-02-01-16-15-50-966.png     PNG File screenshot-1.png    

 Description   

Steps to reproduce:

1、Zabbix server quit due to abnormality.

2、systemctl start zabbix-server

Result:
The command line is unresponsive for a long time, and the log file is unchanged.
Expected:
After Zabbix exits due to insufficient memory, the main process does not close HA Manager, resulting in the inability to close the Zabbix-Server service using systemctl and start the service again.

 Comments   
Comment by liuhua [ 2022 Dec 23 ]

It's not just the HA process

Comment by Tomass Janis Bross [ 2023 Jan 23 ]

Hello!

Thank you for contacting the Zabbix support team.

 

Could you please provide us with your servers hardware information, NVPS value (on the Zabbix frontend: Reports -> System Information), and Zabbix server log of when the issue happens?

 

Cheers,

Tom

Comment by liuhua [ 2023 Feb 01 ]

@tbross    The scenario goes like this: I have a smaller cachesize, and as more devices are added, the lack of memory causes Zabbix to error, and all processes except the HA process exit. At this time, the Zabbix service cannot stop normally and cannot start normally.

Required server performance, new values per second 1973.32


Zabbix server log

2619526:20230201:121404.336 itemid:121771 active range:1953 hits:69 count:25 perc:0.031713%
2619526:20230201:121404.336 itemid:121259 active range:1946 hits:69 count:25 perc:0.031713%
2619526:20230201:121404.336 itemid:122211 active range:1953 hits:69 count:25 perc:0.031713%
2619526:20230201:121404.336 itemid:121555 active range:1946 hits:69 count:25 perc:0.031713%
2619526:20230201:121404.336 itemid:122245 active range:1953 hits:69 count:25 perc:0.031713%
2619526:20230201:121404.336 itemid:125071 active range:1953 hits:69 count:25 perc:0.031713%
2619526:20230201:121404.336 itemid:122545 active range:1953 hits:69 count:25 perc:0.031713%
2619526:20230201:121404.336 itemid:125003 active range:1953 hits:69 count:25 perc:0.031713%
2619526:20230201:121404.336 itemid:121839 active range:1953 hits:69 count:25 perc:0.031713%
2619526:20230201:121404.336 itemid:124673 active range:1953 hits:69 count:25 perc:0.031713%
2619526:20230201:121404.336 itemid:121521 active range:1946 hits:69 count:25 perc:0.031713%
2619526:20230201:121404.336 itemid:124969 active range:1953 hits:69 count:25 perc:0.031713%
2619526:20230201:121404.336 itemid:121737 active range:1953 hits:69 count:25 perc:0.031713%
2619526:20230201:121404.336 itemid:121589 active range:1953 hits:69 count:25 perc:0.031713%
2619526:20230201:121404.336 itemid:125347 active range:1953 hits:66 count:24 perc:0.030444%
2619526:20230201:121404.336 itemid:125753 active range:1953 hits:66 count:24 perc:0.030444%
2619526:20230201:121404.336 itemid:127201 active range:1967 hits:66 count:24 perc:0.030444%
2619526:20230201:121404.336 ==================================================
2619526:20230201:121404.336 === memory statistics for value cache size ===
2619526:20230201:121404.336 free chunks of size     24 bytes:       10
2619526:20230201:121404.336 free chunks of size     32 bytes:      108
2619526:20230201:121404.336 free chunks of size     40 bytes:       14
2619526:20230201:121404.336 free chunks of size     48 bytes:      129
2619526:20230201:121404.336 free chunks of size     56 bytes:       70
2619526:20230201:121404.336 free chunks of size >= 256 bytes:       26
2619526:20230201:121404.336 min chunk size:         24 bytes
2619526:20230201:121404.336 max chunk size:        536 bytes
2619526:20230201:121404.336 memory of total size 6995632 bytes fragmented into 87038 chunks
2619526:20230201:121404.336 of those,      24280 bytes are in      357 free chunks
2619526:20230201:121404.336 of those,    6971352 bytes are in    86681 used chunks
2619526:20230201:121404.336 of those,    1392592 bytes are used by allocation overhead
2619526:20230201:121404.336 ================================
2619526:20230201:121404.336 value cache is fully used: please increase ValueCacheSize configuration parameter
2619507:20230201:121442.635 SNMP agent item "powerdis" on host "10.209.1.48" failed: first network error, wait for 15 seconds
2619511:20230201:121442.675 SNMP agent item "powerdis" on host "10.209.1.51" failed: first network error, wait for 15 seconds
2619501:20230201:121459.586 __mem_malloc: skipped 0 asked 72 skip_min 18446744073709551615 skip_max 0
2619501:20230201:121459.586 === memory statistics for trend cache ===
2619501:20230201:121459.586 free chunks of size     24 bytes:        2
2619501:20230201:121459.586 free chunks of size     32 bytes:        2
2619501:20230201:121459.586 free chunks of size     40 bytes:        2
2619501:20230201:121459.586 free chunks of size     48 bytes:        4
2619501:20230201:121459.586 free chunks of size     56 bytes:        1
2619501:20230201:121459.586 free chunks of size     64 bytes:        2
2619501:20230201:121459.586 min chunk size:         24 bytes
2619501:20230201:121459.586 max chunk size:         64 bytes
2619501:20230201:121459.586 memory of total size 3531200 bytes fragmented into 41421 chunks
2619501:20230201:121459.586 of those,        568 bytes are in       13 free chunks
2619501:20230201:121459.586 of those,    3530632 bytes are in    41408 used chunks
2619501:20230201:121459.586 of those,     662720 bytes are used by allocation overhead
2619501:20230201:121459.586 ================================
2619501:20230201:121459.586 === Backtrace: ===
2619501:20230201:121459.588 18: /usr/sbin/zabbix_server: history syncer #1 [processed 289 values, 339 triggers in 0.127167 sec, syncing history](zbx_backtrace+0x55) [0x55cf72ba416b]
2619501:20230201:121459.588 17: /usr/sbin/zabbix_server: history syncer #1 [processed 289 values, 339 triggers in 0.127167 sec, syncing history](__zbx_shmem_malloc+0x112) [0x55cf72b9e938]
2619501:20230201:121459.588 16: /usr/sbin/zabbix_server: history syncer #1 [processed 289 values, 339 triggers in 0.127167 sec, syncing history](+0x22c89e) [0x55cf72b1189e]
2619501:20230201:121459.588 15: /usr/sbin/zabbix_server: history syncer #1 [processed 289 values, 339 triggers in 0.127167 sec, syncing history](zbx_hashset_insert_ext+0x132) [0x55cf72bf8129]
2619501:20230201:121459.588 14: /usr/sbin/zabbix_server: history syncer #1 [processed 289 values, 339 triggers in 0.127167 sec, syncing history](zbx_hashset_insert+0x31) [0x55cf72bf7ff5]
2619501:20230201:121459.588 13: /usr/sbin/zabbix_server: history syncer #1 [processed 289 values, 339 triggers in 0.127167 sec, syncing history](+0x221e83) [0x55cf72b06e83]
2619501:20230201:121459.588 12: /usr/sbin/zabbix_server: history syncer #1 [processed 289 values, 339 triggers in 0.127167 sec, syncing history](+0x223219) [0x55cf72b08219]
2619501:20230201:121459.588 11: /usr/sbin/zabbix_server: history syncer #1 [processed 289 values, 339 triggers in 0.127167 sec, syncing history](+0x223515) [0x55cf72b08515]
2619501:20230201:121459.588 10: /usr/sbin/zabbix_server: history syncer #1 [processed 289 values, 339 triggers in 0.127167 sec, syncing history](+0x229d04) [0x55cf72b0ed04]
2619501:20230201:121459.588 9: /usr/sbin/zabbix_server: history syncer #1 [processed 289 values, 339 triggers in 0.127167 sec, syncing history](zbx_sync_history_cache+0x91) [0x55cf72b0fb0e]
2619501:20230201:121459.588 8: /usr/sbin/zabbix_server: history syncer #1 [processed 289 values, 339 triggers in 0.127167 sec, syncing history](dbsyncer_thread+0x242) [0x55cf7296f93e]
2619501:20230201:121459.588 7: /usr/sbin/zabbix_server: history syncer #1 [processed 289 values, 339 triggers in 0.127167 sec, syncing history](zbx_thread_start+0x3b) [0x55cf72ba5981]
2619501:20230201:121459.588 6: /usr/sbin/zabbix_server: history syncer #1 [processed 289 values, 339 triggers in 0.127167 sec, syncing history](+0x76c78) [0x55cf7295bc78]
2619501:20230201:121459.588 5: /usr/sbin/zabbix_server: history syncer #1 [processed 289 values, 339 triggers in 0.127167 sec, syncing history](MAIN_ZABBIX_ENTRY+0x99b) [0x55cf7295ce7d]
2619501:20230201:121459.588 4: /usr/sbin/zabbix_server: history syncer #1 [processed 289 values, 339 triggers in 0.127167 sec, syncing history](zbx_daemon_start+0x41e) [0x55cf72ba3c9a]
2619501:20230201:121459.588 3: /usr/sbin/zabbix_server: history syncer #1 [processed 289 values, 339 triggers in 0.127167 sec, syncing history](main+0x39c) [0x55cf7295b0fa]
2619501:20230201:121459.588 2: /lib/x86_64-linux-gnu/libc.so.6(+0x29d90) [0x7f811f261d90]
2619501:20230201:121459.588 1: /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x80) [0x7f811f261e40]
2619501:20230201:121459.588 0: /usr/sbin/zabbix_server: history syncer #1 [processed 289 values, 339 triggers in 0.127167 sec, syncing history](_start+0x25) [0x55cf72959b05]
2619501:20230201:121459.588 file:dbcache.c,line:4508 __zbx_shmem_malloc(): out of memory (requested 72 bytes)
2619501:20230201:121459.588 file:dbcache.c,line:4508 __zbx_shmem_malloc(): please increase TrendCacheSize configuration parameter
2619480:20230201:121459.610 One child process died (PID:2619501,exitcode/signal:1). Exiting ...
2619480:20230201:121500.289 PROCESS EXIT: 2619501
2619481:20230201:121500.290 HA manager has been paused
zabbix_server [2619480]: Error waiting for process with PID 2619501: [10] No child processes
2619480:20230201:121500.405 syncing history data...
2619480:20230201:121500.416 [Z3008] query failed due to primary key constraint: [0] PGRES_FATAL_ERROR:ERROR:  duplicate key value violates unique constraint "177_177_history_pkey"
DETAIL:  Key (itemid, clock, ns)=(142309, 1675224899, 523524868) already exists.

2619480:20230201:121501.874 skipped 6 duplicates
2619480:20230201:121501.874 __mem_malloc: skipped 0 asked 72 skip_min 18446744073709551615 skip_max 0
2619480:20230201:121501.874 === memory statistics for trend cache ===
2619480:20230201:121501.874 free chunks of size     24 bytes:        2
2619480:20230201:121501.874 free chunks of size     32 bytes:        2
2619480:20230201:121501.874 free chunks of size     40 bytes:        2
2619480:20230201:121501.874 free chunks of size     48 bytes:        4
2619480:20230201:121501.874 free chunks of size     56 bytes:        1
2619480:20230201:121501.874 free chunks of size     64 bytes:        2
2619480:20230201:121501.874 min chunk size:         24 bytes
2619480:20230201:121501.874 max chunk size:         64 bytes
2619480:20230201:121501.874 memory of total size 3531200 bytes fragmented into 41421 chunks
2619480:20230201:121501.874 of those,        568 bytes are in       13 free chunks
2619480:20230201:121501.874 of those,    3530632 bytes are in    41408 used chunks
2619480:20230201:121501.874 of those,     662720 bytes are used by allocation overhead
2619480:20230201:121501.874 ================================
2619480:20230201:121501.874 === Backtrace: ===
2619480:20230201:121501.876 18: /usr/sbin/zabbix_server(zbx_backtrace+0x55) [0x55cf72ba416b]
2619480:20230201:121501.876 17: /usr/sbin/zabbix_server(__zbx_shmem_malloc+0x112) [0x55cf72b9e938]
2619480:20230201:121501.876 16: /usr/sbin/zabbix_server(+0x22c89e) [0x55cf72b1189e]
2619480:20230201:121501.876 15: /usr/sbin/zabbix_server(zbx_hashset_insert_ext+0x132) [0x55cf72bf8129]
2619480:20230201:121501.876 14: /usr/sbin/zabbix_server(zbx_hashset_insert+0x31) [0x55cf72bf7ff5]
2619480:20230201:121501.876 13: /usr/sbin/zabbix_server(+0x221e83) [0x55cf72b06e83]
2619480:20230201:121501.876 12: /usr/sbin/zabbix_server(+0x223219) [0x55cf72b08219]
2619480:20230201:121501.876 11: /usr/sbin/zabbix_server(+0x223515) [0x55cf72b08515]
2619480:20230201:121501.876 10: /usr/sbin/zabbix_server(+0x229d04) [0x55cf72b0ed04]
2619480:20230201:121501.876 9: /usr/sbin/zabbix_server(+0x22a6bc) [0x55cf72b0f6bc]
2619480:20230201:121501.876 8: /usr/sbin/zabbix_server(+0x22cf7d) [0x55cf72b11f7d]
2619480:20230201:121501.877 7: /usr/sbin/zabbix_server(free_database_cache+0x48) [0x55cf72b12002]
2619480:20230201:121501.877 6: /usr/sbin/zabbix_server(+0x75c84) [0x55cf7295ac84]
2619480:20230201:121501.877 5: /usr/sbin/zabbix_server(MAIN_ZABBIX_ENTRY+0xf1c) [0x55cf7295d3fe]
2619480:20230201:121501.877 4: /usr/sbin/zabbix_server(zbx_daemon_start+0x41e) [0x55cf72ba3c9a]
2619480:20230201:121501.877 3: /usr/sbin/zabbix_server(main+0x39c) [0x55cf7295b0fa]
2619480:20230201:121501.877 2: /lib/x86_64-linux-gnu/libc.so.6(+0x29d90) [0x7f811f261d90]
2619480:20230201:121501.877 1: /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x80) [0x7f811f261e40]
2619480:20230201:121501.877 0: /usr/sbin/zabbix_server(_start+0x25) [0x55cf72959b05]
2619480:20230201:121501.877 file:dbcache.c,line:4508 __zbx_shmem_malloc(): out of memory (requested 72 bytes)
2619480:20230201:121501.877 file:dbcache.c,line:4508 __zbx_shmem_malloc(): please increase TrendCacheSize configuration parameter
2647422:20230201:150034.418 Starting Zabbix Server. Zabbix 6.2.6 (revision 6981d8b729a).
2647422:20230201:150034.418 ****** Enabled features ******
2647422:20230201:150034.418 SNMP monitoring:           YES
2647422:20230201:150034.418 IPMI monitoring:           YES
2647422:20230201:150034.418 Web monitoring:            YES
2647422:20230201:150034.418 VMware monitoring:         YES
2647422:20230201:150034.418 SMTP authentication:       YES
2647422:20230201:150034.418 ODBC:                      YES
2647422:20230201:150034.418 SSH support:               YES
2647422:20230201:150034.418 IPv6 support:              YES
2647422:20230201:150034.418 TLS support:               YES
2647422:20230201:150034.418 ******************************
2647422:20230201:150034.418 using configuration file: /etc/zabbix/zabbix_server.conf
2647422:20230201:150034.547 current database version (mandatory/optional): 06020000/06020013
2647422:20230201:150034.547 required mandatory version: 06020000
2647423:20230201:150034.648 starting HA manager

 

Generated at Fri Jul 04 07:08:32 EEST 2025 using Jira 9.12.4#9120004-sha1:625303b708afdb767e17cb2838290c41888e9ff0.