-
Incident report
-
Resolution: Cannot Reproduce
-
Trivial
-
None
-
None
-
None
-
None
-
Linux CentOS 7, Server in Docker
Zabbix server is installed with Docker.
Zabbix agent 2 is running on another host.
Steps to reproduce:
- Add template "SMART by Zabbix agent 2" to a monitored host
- Wait for "out of memory" error
Result:
Server logs:
*** Preparing Zabbix server ** Using POSTGRES_USER variable from ENV ** Using POSTGRES_PASSWORD variable from ENV ******************** * DB_SERVER_HOST: postgres-server * DB_SERVER_PORT: 5432 * DB_SERVER_DBNAME: zabbix * DB_SERVER_SCHEMA: public ******************** ** Database 'zabbix' already exists. Please be careful with database owner! ** Table 'zabbix.dbversion' already exists. ** Preparing Zabbix server configuration file ** Updating '/etc/zabbix/zabbix_server.conf' parameter "ListenIP": ''...removed ** Updating '/etc/zabbix/zabbix_server.conf' parameter "ListenPort": ''...removed ** Updating '/etc/zabbix/zabbix_server.conf' parameter "SourceIP": ''...removed ** Updating '/etc/zabbix/zabbix_server.conf' parameter "LogType": 'console'...updated ** Updating '/etc/zabbix/zabbix_server.conf' parameter "LogFile": ''...removed ** Updating '/etc/zabbix/zabbix_server.conf' parameter "LogFileSize": ''...removed ** Updating '/etc/zabbix/zabbix_server.conf' parameter "PidFile": ''...removed ** Updating '/etc/zabbix/zabbix_server.conf' parameter "DebugLevel": '1'...updated ** Updating '/etc/zabbix/zabbix_server.conf' parameter "DBHost": 'postgres-server'...updated ** Updating '/etc/zabbix/zabbix_server.conf' parameter "DBName": 'zabbix'...updated ** Updating '/etc/zabbix/zabbix_server.conf' parameter "DBSchema": 'public'...updated ** Updating '/etc/zabbix/zabbix_server.conf' parameter "DBPort": '5432'...updated ** Updating '/etc/zabbix/zabbix_server.conf' parameter "VaultDBPath": ''...removed ** Updating '/etc/zabbix/zabbix_server.conf' parameter "VaultURL": ''...removed ** Updating '/etc/zabbix/zabbix_server.conf' parameter "DBUser": 'zabbix'...updated ** Updating '/etc/zabbix/zabbix_server.conf' parameter "DBPassword": '****'. Enable DEBUG_MODE to view value ...updated ** Updating '/etc/zabbix/zabbix_server.conf' parameter "StartReportWriters": ''...removed ** Updating '/etc/zabbix/zabbix_server.conf' parameter "WebServiceURL": 'http://zabbix-web-service:10053/report'...updated ** Updating '/etc/zabbix/zabbix_server.conf' parameter "HistoryStorageURL": ''...removed ** Updating '/etc/zabbix/zabbix_server.conf' parameter "HistoryStorageTypes": 'log,text'...updated ** Updating '/etc/zabbix/zabbix_server.conf' parameter "HistoryStorageDateIndex": ''...removed ** Updating '/etc/zabbix/zabbix_server.conf' parameter "DBSocket": ''...removed ** Updating '/etc/zabbix/zabbix_server.conf' parameter "StatsAllowedIP": ''...removed ** Updating '/etc/zabbix/zabbix_server.conf' parameter "StartPollers": ''...removed ** Updating '/etc/zabbix/zabbix_server.conf' parameter "StartIPMIPollers": '3'...added ** Updating '/etc/zabbix/zabbix_server.conf' parameter "StartPollersUnreachable": ''...removed ** Updating '/etc/zabbix/zabbix_server.conf' parameter "StartTrappers": ''...removed ** Updating '/etc/zabbix/zabbix_server.conf' parameter "StartPingers": ''...removed ** Updating '/etc/zabbix/zabbix_server.conf' parameter "StartDiscoverers": ''...removed ** Updating '/etc/zabbix/zabbix_server.conf' parameter "StartHistoryPollers": ''...removed ** Updating '/etc/zabbix/zabbix_server.conf' parameter "StartHTTPPollers": ''...removed ** Updating '/etc/zabbix/zabbix_server.conf' parameter "StartPreprocessors": ''...removed ** Updating '/etc/zabbix/zabbix_server.conf' parameter "StartTimers": ''...removed ** Updating '/etc/zabbix/zabbix_server.conf' parameter "StartEscalators": ''...removed ** Updating '/etc/zabbix/zabbix_server.conf' parameter "StartAlerters": ''...removed ** Updating '/etc/zabbix/zabbix_server.conf' parameter "StartTimers": ''...removed ** Updating '/etc/zabbix/zabbix_server.conf' parameter "StartEscalators": ''...removed ** Updating '/etc/zabbix/zabbix_server.conf' parameter "StartLLDProcessors": ''...removed ** Updating '/etc/zabbix/zabbix_server.conf' parameter "JavaGateway": ''...removed ** Updating '/etc/zabbix/zabbix_server.conf' parameter "JavaGatewayPort": ''...removed ** Updating '/etc/zabbix/zabbix_server.conf' parameter "StartJavaPollers": ''...removed ** Updating '/etc/zabbix/zabbix_server.conf' parameter "StartVMwareCollectors": ''...removed ** Updating '/etc/zabbix/zabbix_server.conf' parameter "VMwareFrequency": ''...removed ** Updating '/etc/zabbix/zabbix_server.conf' parameter "VMwarePerfFrequency": ''...removed ** Updating '/etc/zabbix/zabbix_server.conf' parameter "VMwareCacheSize": ''...removed ** Updating '/etc/zabbix/zabbix_server.conf' parameter "VMwareTimeout": ''...removed ** Updating '/etc/zabbix/zabbix_server.conf' parameter "SNMPTrapperFile": ''...removed ** Updating '/etc/zabbix/zabbix_server.conf' parameter "StartSNMPTrapper": ''...removed ** Updating '/etc/zabbix/zabbix_server.conf' parameter "HousekeepingFrequency": '1'...updated ** Updating '/etc/zabbix/zabbix_server.conf' parameter "MaxHousekeeperDelete": '5000'...updated ** Updating '/etc/zabbix/zabbix_server.conf' parameter "SenderFrequency": ''...removed ** Updating '/etc/zabbix/zabbix_server.conf' parameter "CacheSize": '16G'...added ** Updating '/etc/zabbix/zabbix_server.conf' parameter "CacheUpdateFrequency": ''...removed ** Updating '/etc/zabbix/zabbix_server.conf' parameter "StartDBSyncers": ''...removed ** Updating '/etc/zabbix/zabbix_server.conf' parameter "HistoryCacheSize": '1G'...updated ** Updating '/etc/zabbix/zabbix_server.conf' parameter "HistoryIndexCacheSize": '1G'...updated ** Updating '/etc/zabbix/zabbix_server.conf' parameter "TrendCacheSize": '1G'...updated ** Updating '/etc/zabbix/zabbix_server.conf' parameter "ValueCacheSize": '1G'...updated ** Updating '/etc/zabbix/zabbix_server.conf' parameter "Timeout": ''...removed ** Updating '/etc/zabbix/zabbix_server.conf' parameter "TrapperTimeout": ''...removed ** Updating '/etc/zabbix/zabbix_server.conf' parameter "UnreachablePeriod": ''...removed ** Updating '/etc/zabbix/zabbix_server.conf' parameter "UnavailableDelay": ''...removed ** Updating '/etc/zabbix/zabbix_server.conf' parameter "UnreachableDelay": ''...removed ** Updating '/etc/zabbix/zabbix_server.conf' parameter "AlertScriptsPath": '/usr/lib/zabbix/alertscripts'...updated ** Updating '/etc/zabbix/zabbix_server.conf' parameter "ExternalScripts": '/usr/lib/zabbix/externalscripts'...updated ** Updating '/etc/zabbix/zabbix_server.conf' parameter "FpingLocation": '/usr/bin/fping'...updated ** Updating '/etc/zabbix/zabbix_server.conf' parameter "Fping6Location": '/usr/bin/fping6'...updated ** Updating '/etc/zabbix/zabbix_server.conf' parameter "SSHKeyLocation": '/var/lib/zabbix/ssh_keys'...updated ** Updating '/etc/zabbix/zabbix_server.conf' parameter "LogSlowQueries": ''...removed ** Updating '/etc/zabbix/zabbix_server.conf' parameter "StartProxyPollers": ''...removed ** Updating '/etc/zabbix/zabbix_server.conf' parameter "ProxyConfigFrequency": ''...removed ** Updating '/etc/zabbix/zabbix_server.conf' parameter "ProxyDataFrequency": ''...removed ** Updating '/etc/zabbix/zabbix_server.conf' parameter "SSLCertLocation": '/var/lib/zabbix/ssl/certs/'...updated ** Updating '/etc/zabbix/zabbix_server.conf' parameter "SSLKeyLocation": '/var/lib/zabbix/ssl/keys/'...updated ** Updating '/etc/zabbix/zabbix_server.conf' parameter "SSLCALocation": '/var/lib/zabbix/ssl/ssl_ca/'...updated ** Updating '/etc/zabbix/zabbix_server.conf' parameter "LoadModulePath": '/var/lib/zabbix/modules/'...updated ** Updating '/etc/zabbix/zabbix_server.conf' parameter "TLSCAFile": ''...removed ** Updating '/etc/zabbix/zabbix_server.conf' parameter "TLSCRLFile": ''...removed ** Updating '/etc/zabbix/zabbix_server.conf' parameter "TLSCertFile": ''...removed ** Updating '/etc/zabbix/zabbix_server.conf' parameter "TLSCipherAll": ''...removed ** Updating '/etc/zabbix/zabbix_server.conf' parameter "TLSCipherAll13": ''...removed ** Updating '/etc/zabbix/zabbix_server.conf' parameter "TLSCipherCert": ''...removed ** Updating '/etc/zabbix/zabbix_server.conf' parameter "TLSCipherCert13": ''...removed ** Updating '/etc/zabbix/zabbix_server.conf' parameter "TLSCipherPSK": ''...removed ** Updating '/etc/zabbix/zabbix_server.conf' parameter "TLSCipherPSK13": ''...removed ** Updating '/etc/zabbix/zabbix_server.conf' parameter "TLSKeyFile": ''...removed ** Updating '/etc/zabbix/zabbix_server.conf' parameter "TLSPSKIdentity": ''...removed ** Updating '/etc/zabbix/zabbix_server.conf' parameter "TLSPSKFile": ''...removed ** Updating '/etc/zabbix/zabbix_server.conf' parameter "User": 'zabbix'...updated Starting Zabbix Server. Zabbix 5.4.7 (revision 84dc2ec). Press Ctrl+C to exit. 7:20211110:202342.813 Starting Zabbix Server. Zabbix 5.4.7 (revision 84dc2ec). 7:20211110:202342.813 ****** Enabled features ****** 7:20211110:202342.813 SNMP monitoring: YES 7:20211110:202342.813 IPMI monitoring: YES 7:20211110:202342.813 Web monitoring: YES 7:20211110:202342.813 VMware monitoring: YES 7:20211110:202342.813 SMTP authentication: YES 7:20211110:202342.813 ODBC: YES 7:20211110:202342.813 SSH support: YES 7:20211110:202342.813 IPv6 support: YES 7:20211110:202342.813 TLS support: YES 7:20211110:202342.813 ****************************** 7:20211110:202342.813 using configuration file: /etc/zabbix/zabbix_server.conf 7:20211110:202342.970 current database version (mandatory/optional): 05040000/05040000 7:20211110:202342.970 required mandatory version: 05040000 7:20211110:202343.126 server #0 started [main process] 232:20211110:202343.127 server #1 started [configuration syncer #1] 232:20211110:202343.700 __mem_malloc: skipped 0 asked 256 skip_min 18446744073709551615 skip_max 0 232:20211110:202343.700 [file:dbconfig.c,line:102] __zbx_mem_malloc(): out of memory (requested 256 bytes) 232:20211110:202343.700 [file:dbconfig.c,line:102] __zbx_mem_malloc(): please increase CacheSize configuration parameter 232:20211110:202343.700 === memory statistics for configuration cache === 232:20211110:202343.700 free chunks of size 24 bytes: 23 232:20211110:202343.700 free chunks of size 32 bytes: 5 232:20211110:202343.700 free chunks of size 40 bytes: 3 232:20211110:202343.700 min chunk size: 24 bytes 232:20211110:202343.700 max chunk size: 40 bytes 232:20211110:202343.700 memory of total size 7303704 bytes fragmented into 67784 chunks 232:20211110:202343.700 of those, 832 bytes are in 31 free chunks 232:20211110:202343.700 of those, 7302872 bytes are in 67753 used chunks 232:20211110:202343.700 of those, 1084528 bytes are used by allocation overhead 232:20211110:202343.700 ================================ 232:20211110:202343.700 === Backtrace: === 232:20211110:202343.702 13: /usr/sbin/zabbix_server: configuration syncer [syncing configuration](zbx_backtrace+0x55) [0x55d7b4f1083d] 232:20211110:202343.702 12: /usr/sbin/zabbix_server: configuration syncer [syncing configuration](__zbx_mem_malloc+0x190) [0x55d7b4f0aa63] 232:20211110:202343.702 11: /usr/sbin/zabbix_server: configuration syncer [syncing configuration](+0x1c1fa1) [0x55d7b4e8afa1] 232:20211110:202343.702 10: /usr/sbin/zabbix_server: configuration syncer [syncing configuration](+0x2558e1) [0x55d7b4f1e8e1] 232:20211110:202343.702 9: /usr/sbin/zabbix_server: configuration syncer [syncing configuration](zbx_vector_ptr_append+0x20) [0x55d7b4f1eaac] 232:20211110:202343.702 8: /usr/sbin/zabbix_server: configuration syncer [syncing configuration](+0x1cfd9f) [0x55d7b4e98d9f] 232:20211110:202343.702 7: /usr/sbin/zabbix_server: configuration syncer [syncing configuration](DCsync_configuration+0x137b) [0x55d7b4e9b992] 232:20211110:202343.702 6: /usr/sbin/zabbix_server: configuration syncer [syncing configuration](dbconfig_thread+0x141) [0x55d7b4d345b4] 232:20211110:202343.702 5: /usr/sbin/zabbix_server: configuration syncer [syncing configuration](zbx_thread_start+0x3b) [0x55d7b4f21e6e] 232:20211110:202343.702 4: /usr/sbin/zabbix_server: configuration syncer [syncing configuration](MAIN_ZABBIX_ENTRY+0xc25) [0x55d7b4d20a65] 232:20211110:202343.702 3: /usr/sbin/zabbix_server: configuration syncer [syncing configuration](daemon_start+0x305) [0x55d7b4f10376] 232:20211110:202343.702 2: /usr/sbin/zabbix_server: configuration syncer [syncing configuration](main+0x330) [0x55d7b4d1fd1e] 232:20211110:202343.702 1: /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf3) [0x7f492029f0b3] 232:20211110:202343.702 0: /usr/sbin/zabbix_server: configuration syncer [syncing configuration](_start+0x2e) [0x55d7b4d1e96e] 7:20211110:202343.708 One child process died (PID:232,exitcode/signal:1). Exiting ... 7:20211110:202343.740 Zabbix Server stopped. Zabbix 5.4.7 (revision 84dc2ec).*