While investigating value cache free space/fragmentation issues I found also the following minor bugs:

• hashset allocates 3 bytes more than is necessary to store the entry structure with data, fixed in hashset-entry-fix.diff
• memory allocator incorrectly updates the free space when memory is reallocated by merging/splitting the next chunk, fixed in memmalloc-statistics-fix.diff

Also I tried the following improvements regarding memory usage:

• reduced value cache chunk header size by 4 bytes (with padding) - valuecache-small-chunk-header.diff. The chunk is limited to 64KB, so the slot data can be kept in 2 bytes instead of 4
• reverted memory allocator to use 4 bytes to store chunk size - memmalloc-32bit-chunks.diff. This will limit maximum allocation to 2GB - 1 and will result in initial fragmentation of shared memory larger than 2GB. While theoretically it should save 8 bytes per allocation, it did not have any noticeable effect on value cache usage.

I experimented a bit with normalizing value cache chunk sizes to reduce memory fragmentation. While I managed to reduce the fragmentation (at least in my test setup), it did not really affect the size of stored data because of smaller chunks and data storage overhead for each chunk.

Value cache statistics with 32bit memory allocator:

6688:20140308:211224.277 === memory statistics for value cache size ===
6688:20140308:211224.278 free chunks of size     24 bytes:        2
6688:20140308:211224.278 free chunks of size     32 bytes:        1
6688:20140308:211224.278 free chunks of size     56 bytes:        1
6688:20140308:211224.280 free chunks of size     64 bytes:     4963
6688:20140308:211224.280 free chunks of size     80 bytes:     2046
6688:20140308:211224.280 free chunks of size     88 bytes:        1
6688:20140308:211224.281 free chunks of size     96 bytes:     3069
6688:20140308:211224.281 free chunks of size    112 bytes:       43
6688:20140308:211224.281 free chunks of size    128 bytes:       82
6688:20140308:211224.281 free chunks of size    144 bytes:        2
6688:20140308:211224.282 free chunks of size    160 bytes:        3
6688:20140308:211224.282 free chunks of size    176 bytes:        1
6688:20140308:211224.282 free chunks of size    192 bytes:        1
6688:20140308:211224.282 min chunk size:         24 bytes
6688:20140308:211224.282 max chunk size:        198 bytes
6688:20140308:211224.282 memory of total size 8388214 bytes fragmented into 72025 chunks
6688:20140308:211224.282 of those,     792614 bytes are in    10215 free chunks
6688:20140308:211224.282 of those,    7019400 bytes are in    61810 used chunks
6688:20140308:211224.282 ================================

Value cache statistics with 32bit memory allocator and normalized value cache chunk sizes:

31155:20140309:005754.989 === memory statistics for value cache size ===
31155:20140309:005754.989 free chunks of size     24 bytes:        2
31155:20140309:005754.989 free chunks of size     56 bytes:       11
31155:20140309:005754.989 free chunks of size     64 bytes:        1
31155:20140309:005754.989 free chunks of size    200 bytes:        1
31155:20140309:005754.990 min chunk size:         24 bytes
31155:20140309:005754.990 max chunk size:        206 bytes
31155:20140309:005754.990 memory of total size 8388214 bytes fragmented into 84806 chunks
31155:20140309:005754.990 of those,        934 bytes are in       15 free chunks
31155:20140309:005754.990 of those,    7708832 bytes are in    84791 used chunks
31155:20140309:005754.990 ================================

The test case consisted of 500 hosts, 17 triggers per host plus a host with 38 triggers resulting in 8538 items cached by value cache. The items were only of uint or floating type, so basically the only allocations made buy value cache would be to store items, value chunks and the hashset itself. When we discard the overhead of chunk header storage the size of data stored in both cases are quite close:

1. 7019400 - (61810 - 8538) * 28 = 5527784
2. 7708832 - (84791 - 8538) * 28 = 5573748

Those tests were done on clean history, systems with existing history has less fragmentation.

Bottom line - in current form the fragmentation optimizations aren't worth it.

well, that probably explains people reporting value cache "not filling up" and getting stuck at 80-90 % full

wiper you could still get 90% because of fragmentation, but I've seen it switching to low memory mode when only 70% full.

Related issue: ZBX-7741.

wiper Good, so the memory allocator chunk size changes will be resolved there. I attached the 32bit patch to it.

Created ZBX-7919 to handle hashset allocation issue.
Created ZBX-7920 to handle value cache allocation optimization

So only statistics (value cache and memory allocated) fixes will be addressed by this issue.

Fixed in development branch svn://svn.zabbix.com/branches/dev/ZBX-7918

(1) Shall we fix memory statistics for zabbix[vmware] item, too?

wiper Yes, but I think we must create a new issue for that.

wiper Opened a new issue ZBX-7934
CLOSED

Released in:
pre-2.2.3rc1 r43453
pre-2.3.0 r43454