We are seeing a constant decrease in Zabbix history cache writer (% free). Why I suspect some sort of memory leak is:

1) We drastically decreased items/triggers (especially ones that use logging), but rate of decrease remains the same

2) The 4 history sync processes are often either idling, or handling small counts of items with quick pace

I analyzed the memory map of the processes, and as expected, I am seeing the "shared memory segments" growing over time. What is odd, is that these all show up as "deleted"

smap_51122_20200629T1232: 193mb 2020-06-29T12:32:00+00:00
item: total increased 3220
item: 55acee970000-55acf1917000 0 increased 88
item: 7f7a1e135000-7f7a26135000 /SYSV00000000 increased 1788
item: 7f7a26135000-7f7a32135000 /SYSV00000000 increased 112
item: 7f7a34135000-7f7a44135000 /SYSV00000000 increased 1232
smap_51122_20200629T1419: 197mb change: +3mb 107 minutes 1722kb/hr
item: total increased 1808
item: 7f7a1e135000-7f7a26135000 /SYSV00000000 increased 108
item: 7f7a26135000-7f7a32135000 /SYSV00000000 increased 40
item: 7f7a34135000-7f7a44135000 /SYSV00000000 increased 1660
smap_51122_20200629T1451: 198mb change: +1mb 32 minutes 1920kb/hr
item: total increased 1896
item: 7f7a1e135000-7f7a26135000 /SYSV00000000 increased 104
item: 7f7a26135000-7f7a32135000 /SYSV00000000 increased 124
item: 7f7a34135000-7f7a44135000 /SYSV00000000 increased 1664
item: 7fff8a91a000-7fff8a93c000 0 increased 4
smap_51122_20200629T1514: 200mb change: +1mb 23 minutes 2671kb/hr

The two significant ones above are:

7f7a1e135000-7f7a26135000 rw-s 00000000 00:04 4390920 /SYSV00000000 (deleted)
Size: 131072 kB *(Not sure, but could be the ValueCacheSize=128M - though this is stable)*
Rss: 45496 kB

7f7a34135000-7f7a44135000 rw-s 00000000 00:04 4227075 /SYSV00000000 (deleted)
*Size: 262144 kB (This one matches the configuratino HistoryCacheSize=256M)*
Rss: 38160 kB
Pss: 7867 kBPss: 10624 kB

We have had this issue for some time, with Zabbix 4.2, and recently upgraded to 5.0 and still seeing it. Our original solution was to crank up the HistoryCacheSize and restart during weekends, but it winds up taking 24hrs+ to shut down while it syncs the history cache (At the time, 1G) - which is a bit odd considering the DB shows throughput of ~4-5MB/s. But this in itself is a separate issue. We reduced the HistoryCacheSize to 256, and the history sync completes much quicker now, but the "leak" rate stays same (now we have to restart daily)

Any additional steps I can take to diagnose as to what is in those memory segments, and determine why they are growing? We have halved our enabled items at some point to see if it impacts the rate, and it did not.

Steps to reproduce:

1. Not sure how to reproduce as I don't fully understand what is causing the issue