[#ZBX-5745] message "increase CacheSize configuration parameter" misleads

Comment by Oleksii Zagorskyi [ 2013 Jan 15 ]

A related article, just in case https://www.zabbix.org/wiki/How_to/configure_shared_memory

Comment by Andris Zeila [ 2013 Jan 16 ]

As only the reporter has experienced this issue it must be something related to his configuration/setup. To make any progress we need to know more details.

Comment by Oleksii Zagorskyi [ 2013 Jan 16 ]

Indeed, I recall this issue and it was a bit strange.

Heh, I did my own investigation:

Copy-paste from issue description (to have all in one place) :

  2147:20121023:220053.055 __mem_malloc: skipped 0 asked 24 skip_min 4294967295 skip_max 0 
  2147:20121023:220053.055 [file:strpool.c,line:57] zbx_mem_malloc(): out of memory (requested 14 bytes) 
  2147:20121023:220053.055 [file:strpool.c,line:57] zbx_mem_malloc(): please increase CacheSize configuration parameter 
  2137:20121023:220053.057 One child process died (PID:2147,exitcode/signal:255). Exiting ...

My experiments with (DebugLevel in 3, 4) and different "CacheSize".
OS is default for Linux:

@sysctl -a | grep -E "shmall|shmmax"
kernel.shmmax = 33554432
kernel.shmall = 2097152

Zabbix installation is very small.

Very big CacheSize (too big for OS)
DL=3 CS=1G:

 25164:20130116:142139.289 cannot allocate shared memory of size 912680551: [22] Invalid argument
 25164:20130116:142139.289 cannot allocate shared memory for configuration cache

DL=4 CS=1G:

 25203:20130116:142214.132 In init_database_cache()
 25203:20130116:142214.132 In zbx_mem_required_size() size:8421080 chunks_num:4 descr:'history cache' param:'HistoryCacheSize'
 25203:20130116:142214.132 End of zbx_mem_required_size() size:8421575
 25203:20130116:142214.132 In zbx_mem_create() descr:'history cache' param:'HistoryCacheSize' size:8421575
 25203:20130116:142214.132 zbx_shmget() removing existing shm_id:237273134
 25203:20130116:142214.132 valid user addresses: [0x7f0b30edc160, 0x7f0b316e40c0] total size: 8421216
 25203:20130116:142214.132 End of zbx_mem_create()
 25203:20130116:142214.132 In zbx_mem_required_size() size:16777216 chunks_num:1 descr:'history text cache' param:'HistoryTextCacheSize'
 25203:20130116:142214.132 End of zbx_mem_required_size() size:16777627
 25203:20130116:142214.132 In zbx_mem_create() descr:'history text cache' param:'HistoryTextCacheSize' size:16777627
 25203:20130116:142214.132 zbx_shmget() removing existing shm_id:237305905
 25203:20130116:142214.132 valid user addresses: [0x7f0b2fedb168, 0x7f0b30edb190] total size: 16777256
 25203:20130116:142214.132 End of zbx_mem_create()
 25203:20130116:142214.132 In init_trend_cache()
 25203:20130116:142214.132 In zbx_mem_required_size() size:4194304 chunks_num:1 descr:'trend cache' param:'TrendCacheSize'
 25203:20130116:142214.132 End of zbx_mem_required_size() size:4194702
 25203:20130116:142214.132 In zbx_mem_create() descr:'trend cache' param:'TrendCacheSize' size:4194702
 25203:20130116:142214.132 zbx_shmget() removing existing shm_id:237338675
 25203:20130116:142214.132 valid user addresses: [0x7f0b2fada158, 0x7f0b2feda188] total size: 4194352
 25203:20130116:142214.132 End of zbx_mem_create()
 25203:20130116:142214.132 End of init_trend_cache()
 25203:20130116:142214.133 End of init_database_cache()
 25203:20130116:142214.133 In init_configuration_cache() size:1073741824
 25203:20130116:142214.133 In zbx_mem_create() descr:'configuration cache' param:'CacheSize' size:912680551
 25203:20130116:142214.133 cannot allocate shared memory of size 912680551: [22] Invalid argument
 25203:20130116:142214.133 cannot allocate shared memory for configuration cache

Very small CacheSize, cannot allocate pool for cache
DL=3 CS=200K:

 24129:20130116:133341.608 __mem_malloc: skipped 1 asked 8072 skip_min 2024 skip_max 2024
 24129:20130116:133341.608 [file:dbconfig.c,line:368] zbx_mem_malloc(): out of memory (requested 8072 bytes)
 24129:20130116:133341.608 [file:dbconfig.c,line:368] zbx_mem_malloc(): please increase CacheSize configuration parameter

DL=4 CS=200K:

 24081:20130116:133220.834 In zbx_mem_required_size() size:8421080 chunks_num:4 descr:'history cache' param:'HistoryCacheSize'
 24081:20130116:133220.834 End of zbx_mem_required_size() size:8421575
 24081:20130116:133220.834 In zbx_mem_create() descr:'history cache' param:'HistoryCacheSize' size:8421575
 24081:20130116:133220.834 valid user addresses: [0x7f13620dd160, 0x7f13628e50c0] total size: 8421216
 24081:20130116:133220.834 End of zbx_mem_create()
 24081:20130116:133220.834 In zbx_mem_required_size() size:16777216 chunks_num:1 descr:'history text cache' param:'HistoryTextCacheSize'
 24081:20130116:133220.834 End of zbx_mem_required_size() size:16777627
 24081:20130116:133220.834 In zbx_mem_create() descr:'history text cache' param:'HistoryTextCacheSize' size:16777627
 24081:20130116:133220.834 valid user addresses: [0x7f13610dc168, 0x7f13620dc190] total size: 16777256
 24081:20130116:133220.834 End of zbx_mem_create()
 24081:20130116:133220.834 In init_trend_cache()
 24081:20130116:133220.834 In zbx_mem_required_size() size:4194304 chunks_num:1 descr:'trend cache' param:'TrendCacheSize'
 24081:20130116:133220.834 End of zbx_mem_required_size() size:4194702
 24081:20130116:133220.834 In zbx_mem_create() descr:'trend cache' param:'TrendCacheSize' size:4194702
 24081:20130116:133220.835 valid user addresses: [0x7f1360cdb158, 0x7f13610db188] total size: 4194352
 24081:20130116:133220.835 End of zbx_mem_create()
 24081:20130116:133220.835 End of init_trend_cache()
 24081:20130116:133220.835 End of init_database_cache()
 24081:20130116:133220.835 In init_configuration_cache() size:204800
 24081:20130116:133220.835 In zbx_mem_create() descr:'configuration cache' param:'CacheSize' size:174080
 24081:20130116:133220.835 valid user addresses: [0x7f1360cb0160, 0x7f1360cda7f8] total size: 173720
 24081:20130116:133220.835 End of zbx_mem_create()
 24081:20130116:133220.835 __mem_malloc: skipped 1 asked 8072 skip_min 2024 skip_max 2024
 24081:20130116:133220.835 [file:dbconfig.c,line:368] zbx_mem_malloc(): out of memory (requested 8072 bytes)
 24081:20130116:133220.835 [file:dbconfig.c,line:368] zbx_mem_malloc(): please increase CacheSize configuration parameter

Too small CacheSize, config cache cannot be filled
DL=3 CS=330K:
server is starting and very soon stopped:

 25638:20130116:143031.812 current database version (mandatory/optional): 02010027/02010027
 25638:20130116:143031.812 required mandatory version: 02010027
 25646:20130116:143031.816 server #1 started [configuration syncer #1]
 25647:20130116:143031.816 server #2 started [db watchdog #1]
 25650:20130116:143031.817 server #5 started [trapper #1]
 25646:20130116:143031.817 __mem_malloc: skipped 0 asked 24 skip_min 4294967295 skip_max 0
 25646:20130116:143031.817 [file:dbconfig.c,line:368] zbx_mem_malloc(): out of memory (requested 24 bytes)
 25646:20130116:143031.817 [file:dbconfig.c,line:368] zbx_mem_malloc(): please increase CacheSize configuration parameter
 25638:20130116:143031.818 One child process died (PID:25646,exitcode/signal:255). Exiting ...
 25638:20130116:143033.819 syncing history data...
 25638:20130116:143033.819 syncing history data done
 25638:20130116:143033.819 syncing trends data...
 25638:20130116:143033.819 syncing trends data done
 25638:20130116:143033.819 Zabbix Server stopped. Zabbix 2.1.0 (revision 32222).

Success start with default caches, DL=4:

 23674:20130116:131951.106 In init_database_cache()
 23674:20130116:131951.106 In zbx_mem_required_size() size:8421080 chunks_num:4 descr:'history cache' param:'HistoryCacheSize'
 23674:20130116:131951.106 End of zbx_mem_required_size() size:8421575
 23674:20130116:131951.106 In zbx_mem_create() descr:'history cache' param:'HistoryCacheSize' size:8421575
 23674:20130116:131951.106 zbx_shmget() removing existing shm_id:234422318
 23674:20130116:131951.106 valid user addresses: [0x7f4a6e1ea160, 0x7f4a6e9f20c0] total size: 8421216
 23674:20130116:131951.106 End of zbx_mem_create()
 23674:20130116:131951.106 In zbx_mem_required_size() size:16777216 chunks_num:1 descr:'history text cache' param:'HistoryTextCacheSize'
 23674:20130116:131951.106 End of zbx_mem_required_size() size:16777627
 23674:20130116:131951.106 In zbx_mem_create() descr:'history text cache' param:'HistoryTextCacheSize' size:16777627
 23674:20130116:131951.106 zbx_shmget() removing existing shm_id:234455089
 23674:20130116:131951.106 valid user addresses: [0x7f4a6d1e9168, 0x7f4a6e1e9190] total size: 16777256
 23674:20130116:131951.106 End of zbx_mem_create()
 23674:20130116:131951.106 In init_trend_cache()
 23674:20130116:131951.106 In zbx_mem_required_size() size:4194304 chunks_num:1 descr:'trend cache' param:'TrendCacheSize'
 23674:20130116:131951.106 End of zbx_mem_required_size() size:4194702
 23674:20130116:131951.106 In zbx_mem_create() descr:'trend cache' param:'TrendCacheSize' size:4194702
 23674:20130116:131951.106 zbx_shmget() removing existing shm_id:234487859
 23674:20130116:131951.106 valid user addresses: [0x7f4a6cde8158, 0x7f4a6d1e8188] total size: 4194352
 23674:20130116:131951.106 End of zbx_mem_create()
 23674:20130116:131951.107 End of init_trend_cache()
 23674:20130116:131951.107 End of init_database_cache()
 23674:20130116:131951.107 In init_configuration_cache() size:8388608
 23674:20130116:131951.107 In zbx_mem_create() descr:'configuration cache' param:'CacheSize' size:7130317
 23674:20130116:131951.107 valid user addresses: [0x7f4a6c71b160, 0x7f4a6cde7cc8] total size: 7129960
 23674:20130116:131951.107 End of zbx_mem_create()
 23674:20130116:131951.107 In zbx_strpool_create()
 23674:20130116:131951.107 In zbx_mem_create() descr:'string pool' param:'CacheSize' size:1258291
 23674:20130116:131951.107 valid user addresses: [0x7f4a6c5e7158, 0x7f4a6c71a328] total size: 1257936
 23674:20130116:131951.107 End of zbx_mem_create()
 23674:20130116:131951.107 End of zbx_strpool_create()
 23674:20130116:131951.107 End of init_configuration_cache()
 23674:20130116:131951.107 In init_selfmon_collector()
 23674:20130116:131951.107 init_selfmon_collector() size:5992
 23674:20130116:131951.107 End of init_selfmon_collector() collector:0x7f4a6c5e5000
 23674:20130116:131951.107 In DCload_config()

Conclusions:

If a very last line in the log is:

cannot allocate shared memory for configuration cache

then it means that server was not able to allocate shared memory pool on start.
Solution -> need to increase allowed shared memory in OS.
Maybe we could add additional message to the log which will be clear what to do for not experienced users, as you wish.

If in end of the log are lines like:

One child process died (PID:25646,exitcode/signal:255). Exiting ...
...
Zabbix Server stopped. Zabbix 2.1.0 (revision 32222).

then zabbix server actually was working but at some point of time cache(s) became empty and server decided to stop.

Note that issue reported had a problem in strpool.c,line:57 but I generated problem in dbconfig.c,line:368
I don't know to to generate such condition, but it's not so important, IMO.

Here nothing to fix, Zabbix is fine, IMO.

Comment by Andris Zeila [ 2013 Jan 21 ]

Yes, the results matches my memory allocator review. Closing.

[ZBX-5745] message "increase CacheSize configuration parameter" misleads Created: 2012 Oct 25 Updated: 2017 May 30 Resolved: 2013 Jan 21
Status:	Closed
Project:	ZABBIX BUGS AND ISSUES
Component/s:	Server (S)
Affects Version/s:	2.0.2
Fix Version/s:	None

[ZBX-5745] message "increase CacheSize configuration parameter" misleads Created: 2012 Oct 25 Updated: 2017 May 30 Resolved: 2013 Jan 21