[ZBX-24989] Memory leak on proxy Created: 2024 Aug 07  Updated: 2025 Apr 03  Resolved: 2024 Aug 25

Status: Closed
Project: ZABBIX BUGS AND ISSUES
Component/s: Proxy (P)
Affects Version/s: 7.0.2
Fix Version/s: 7.0.4rc1, 7.2.0alpha1

Type: Problem report Priority: Major
Reporter: Yury Larin Assignee: Vladislavs Sokurenko
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: PNG File Screenshot 2024-08-08 060613.png     PNG File Screenshot 2024-08-08 060640.png     PNG File Screenshot 2024-08-08 060707.png     PNG File Screenshot 2024-08-08 193952.png     PNG File image-2024-08-09-17-40-58-622.png     PNG File image-2024-08-09-17-59-35-965.png     PNG File image-2024-08-09-19-11-29-594.png     PNG File image-2024-08-11-15-45-41-534.png    
Issue Links:
Duplicate
Sub-task
depends on ZBX-25048 Allow setting MaxConcurrentChecksPerP... Confirmed
Team: Team A
Sprint: S24-W32/33, S24-W34/35
Story Points: 1

 Description   

Steps to reproduce:

  1. We have big number of hosts on a proxy via new snmp pollers.
  2. Probably after some downtime of the Zabbix server the proxy snmp poller begin to eat much more RAM.

Result:

root@apaultnms05:~# ps -eo size,pid,user,command --sort -size |     awk '\{ hr=$1/1024 ; printf("%13.2f Mb ",hr) } \{ for ( x=4 ; x<=NF ; x++ ) { printf("%s ",$x) } print "" }' |    cut -d "" -f2 | cut -d "-" -f1
         0.00 Mb COMMAND
      3544.06 Mb /usr/sbin/mariadbd
       135.60 Mb /usr/sbin/zabbix_proxy: http agent poller #2 [got 4 values, queued 4 in 5 sec]
       132.78 Mb /usr/sbin/zabbix_proxy: http agent poller #1 [got 0 values, queued 0 in 5 sec]
       231.28 Mb /usr/sbin/zabbix_agent2
       858.76 Mb /usr/sbin/zabbix_proxy: snmp poller #10 [got 512 values, queued 962 in 5 sec]
       856.32 Mb /usr/sbin/zabbix_proxy: snmp poller #42 [got 0 values, queued 0 in 5 sec]
       820.81 Mb /usr/sbin/zabbix_proxy: snmp poller #15 [got 0 values, queued 0 in 5 sec]
       817.17 Mb /usr/sbin/zabbix_proxy: snmp poller #24 [got 0 values, queued 0 in 5 sec]
       786.20 Mb /usr/sbin/zabbix_proxy: snmp poller #13 [got 0 values, queued 0 in 5 sec]
       781.75 Mb /usr/sbin/zabbix_proxy: snmp poller #44 [got 0 values, queued 0 in 5 sec]
       775.36 Mb /usr/sbin/zabbix_proxy: snmp poller #45 [got 214 values, queued 597 in 5 sec]
       748.81 Mb /usr/sbin/zabbix_proxy: snmp poller #4 [got 915 values, queued 911 in 5 sec]
       322.54 Mb falcon
       742.47 Mb /usr/sbin/zabbix_proxy: snmp poller #26 [got 0 values, queued 0 in 5 sec]
       703.28 Mb /usr/sbin/zabbix_proxy: snmp poller #41 [got 0 values, queued 0 in 5 sec]
       700.03 Mb /usr/sbin/zabbix_proxy: snmp poller #8 [got 3 values, queued 0 in 5 sec]
       691.48 Mb /usr/sbin/zabbix_proxy: snmp poller #48 [got 0 values, queued 0 in 5 sec]
       690.63 Mb /usr/sbin/zabbix_proxy: snmp poller #20 [got 0 values, queued 0 in 5 sec]
       674.15 Mb /usr/sbin/zabbix_proxy: snmp poller #34 [got 0 values, queued 0 in 5 sec]
       673.82 Mb /usr/sbin/zabbix_proxy: snmp poller #49 [got 0 values, queued 0 in 5 sec]
       641.32 Mb /usr/sbin/zabbix_proxy: snmp poller #1 [got 0 values, queued 0 in 5 sec]
       635.46 Mb /usr/sbin/zabbix_proxy: snmp poller #43 [got 0 values, queued 0 in 5 sec]
       613.28 Mb /usr/sbin/zabbix_proxy: snmp poller #7 [got 0 values, queued 0 in 5 sec]
       592.97 Mb /usr/sbin/zabbix_proxy: snmp poller #33 [got 0 values, queued 0 in 5 sec]
       573.11 Mb /usr/sbin/zabbix_proxy: snmp poller #19 [got 0 values, queued 0 in 5 sec]
       543.71 Mb /usr/sbin/zabbix_proxy: snmp poller #36 [got 0 values, queued 0 in 5 sec]
       513.50 Mb /usr/sbin/zabbix_proxy: snmp poller #17 [got 0 values, queued 0 in 5 sec]
       474.89 Mb /usr/sbin/zabbix_proxy: snmp poller #50 [got 0 values, queued 0 in 5 sec]
       461.55 Mb /usr/sbin/zabbix_proxy: snmp poller #6 [got 0 values, queued 0 in 5 sec]
       450.07 Mb /usr/sbin/zabbix_proxy: snmp poller #22 [got 0 values, queued 0 in 5 sec]
       366.57 Mb /usr/sbin/zabbix_proxy: snmp poller #2 [got 0 values, queued 0 in 5 sec]
       368.56 Mb /usr/sbin/zabbix_proxy: snmp poller #14 [got 0 values, queued 0 in 5 sec]
        43.29 Mb /usr/sbin/zabbix_proxy: discovery manager #1 [processing 0 rules, 0 unsaved checks]
       293.50 Mb /usr/sbin/zabbix_proxy: snmp poller #9 [got 0 values, queued 0 in 5 sec]
       276.06 Mb /usr/sbin/zabbix_proxy: snmp poller #16 [got 0 values, queued 0 in 5 sec]
       168.06 Mb /usr/sbin/zabbix_proxy: preprocessing manager #1 [queued 1240, processed 1317 values, idle 4.946487 sec during 5.050130 sec]
       235.54 Mb /opt/fortinet/fortisiem/linux
        82.54 Mb /usr/sbin/zabbix_proxy: http agent poller #3 [got 0 values, queued 0 in 5 sec]
       191.48 Mb /usr/sbin/zabbix_proxy: snmp poller #12 [got 0 values, queued 0 in 5 sec]
       187.00 Mb /usr/sbin/zabbix_proxy: snmp poller #5 [got 0 values, queued 0 in 5 sec]
       170.38 Mb /usr/sbin/zabbix_proxy: snmp poller #23 [got 0 values, queued 0 in 5 sec]
       168.81 Mb /usr/sbin/zabbix_proxy: snmp poller #32 [got 0 values, queued 0 in 5 sec]
       166.75 Mb /usr/sbin/zabbix_proxy: snmp poller #31 [got 0 values, queued 0 in 5 sec]
       143.11 Mb /usr/sbin/zabbix_proxy: snmp poller #3 [got 0 values, queued 0 in 5 sec]
       115.37 Mb /usr/sbin/zabbix_proxy: snmp poller #18 [got 0 values, queued 0 in 5 sec]
        86.99 Mb /usr/sbin/zabbix_proxy: snmp poller #40 [got 0 values, queued 0 in 5 sec]
       101.29 Mb /usr/sbin/zabbix_proxy: snmp poller #38 [got 0 values, queued 0 in 5 sec]
       100.91 Mb /usr/sbin/zabbix_proxy: snmp poller #30 [got 0 values, queued 0 in 5 sec]
        93.50 Mb /usr/sbin/zabbix_proxy: snmp poller #28 [got 0 values, queued 0 in 5 sec]
        62.25 Mb /usr/sbin/zabbix_proxy: snmp poller #29 [got 0 values, queued 0 in 5 sec]
        87.15 Mb /usr/sbin/zabbix_proxy: snmp poller #11 [got 0 values, queued 0 in 5 sec]
        87.20 Mb /usr/sbin/zabbix_proxy: snmp poller #37 [got 0 values, queued 0 in 5 sec]
        75.70 Mb /usr/sbin/zabbix_proxy: snmp poller #39 [got 0 values, queued 0 in 5 sec]
        74.71 Mb /usr/sbin/zabbix_proxy: snmp poller #35 [got 0 values, queued 0 in 5 sec]
        74.46 Mb /usr/sbin/zabbix_proxy: snmp poller #27 [got 0 values, queued 0 in 5 sec]
        74.68 Mb /usr/sbin/zabbix_proxy: snmp poller #25 [got 0 values, queued 0 in 5 sec]
        74.67 Mb /usr/sbin/zabbix_proxy: snmp poller #21 [got 0 values, queued 0 in 5 sec]
        73.89 Mb /usr/sbin/zabbix_proxy: snmp poller #47 [got 0 values, queued 0 in 5 sec]
        46.63 Mb /usr/sbin/zabbix_proxy: snmp poller #46 [got 0 values, queued 0 in 5 sec]
        65.95 Mb /usr/sbin/zabbix_proxy: agent poller #2 [got 0 values, queued 0 in 5 sec]
        65.89 Mb /usr/sbin/zabbix_proxy: agent poller #1 [got 0 values, queued 0 in 5 sec]
        65.75 Mb /usr/sbin/zabbix_proxy: agent poller #3 [got 3 values, queued 3 in 5 sec]
        57.60 Mb /usr/sbin/zabbix_proxy: data sender [sent 473 values in 0.027175 sec, idle 1 sec]
        53.93 Mb /usr/sbin/zabbix_proxy: configuration syncer [synced config 447 bytes in 0.146738 sec, idle 10 sec]

Expected:
It should not use so much RAM.

 Comments   
Comment by Yury Larin [ 2024 Aug 07 ]

Probably it leaks without server downtime. I need more time to check it. It leaks in 1-2 weeks.

Comment by Yury Larin [ 2024 Aug 08 ]

I restarted zabbix-proxy. But the memory leak is still here. Looks like it is not depended on the server reboot.

 

Comment by Yury Larin [ 2024 Aug 08 ]

I used both snmp v2 and snmp v3 on this proxy. The difference in the load on the graph is about 50-100 times.

Could it be the reason?

I created a topic about this on the zabbix forum: New snmp pollers load on snmpv2 vs snmpv3 with encryption. - ZABBIX Forums

Comment by Vladislavs Sokurenko [ 2024 Aug 09 ]

Does it stop growing at some moment ? How many items are there, is it possible to reproduce on smaller installation for example with snmp simulator ? Could you please try reducing number of pollers, 10 could be enough for 10k checks and see if memory stabilises.
If it stabilises then it mean that each poller require memory to process snmp checks and it looks like a memory leak because they take turn to process snmp items and each process reserve some memory at some point but especially noticeable when there is downtime since more pollers will have to take load.

Comment by Yury Larin [ 2024 Aug 09 ]

264k items.

800 checks per sec.

Comment by Vladislavs Sokurenko [ 2024 Aug 09 ]

Thank you and how many snmp interfaces are there ?

Comment by Yury Larin [ 2024 Aug 09 ]

StartSNMPPollers=10 - changed.

I set them to 50 for future load + I have slow links to almost all devices (some 5kbit/s only via SAT links). Due to this I need much bigger timeouts (up to 180 sec now) and more Pollers to wait answers.

Comment by Yury Larin [ 2024 Aug 09 ]

snmp interfaces ? on hosts? should be 1. Only if somewhere by mistake duplicated.

Comment by Vladislavs Sokurenko [ 2024 Aug 09 ]

How many different snmp interfaces as they might be cached by snmp library, perhaps it would be better to allow MaxConcurrentChecksPerPoller to be more than 1000 in that case there will be less processes required and overhead will be less as more cached memory would be reused if that is the case

Comment by Yury Larin [ 2024 Aug 09 ]

I have a few thousand devices with 80-250 items on each. about 800 devices are without bulk tick as snmpv3 on these devices crushes in bulk mode.

So, if it caches interfaces by items - yes. there could be many similar.

The devices have different IP. almost all of them. 

Comment by Vladislavs Sokurenko [ 2024 Aug 09 ]

In this case if devices take very long to respond it might be better to allow MaxConcurrentChecksPerPoller to be set to 10000

Comment by Yury Larin [ 2024 Aug 09 ]

It is said 1000 is max.

Is it possible to set it to 10 000 ?

 

      1. Option: MaxConcurrentChecksPerPoller
  2.       Maximum number of asynchronous checks that can be executed at once by each HTTP agent poller or agent poller.
    #
  3. Mandatory: no
  4. Range: 1-1000
  5. Default:
    MaxConcurrentChecksPerPoller=1000
Comment by Yury Larin [ 2024 Aug 09 ]

So far it is not clear about leak after changing to 10 processes.

Looks like still leaking.

Probably need to wait more.

Comment by Yury Larin [ 2024 Aug 09 ]

Also, why it is said about HTTP agent poller?

Is MaxConcurrentChecksPerPoller parameter for all new pollers?

Comment by Vladislavs Sokurenko [ 2024 Aug 09 ]

Can you try small patch that will allow it?

Comment by Yury Larin [ 2024 Aug 09 ]

I could try to. But I do not have build tools on that system...

If you can explain a bit more how to apply the patch - of cause I will try it.

Comment by Yury Larin [ 2024 Aug 11 ]

Looks like the leak is still here even with 10 snmp pollers:

Comment by Vladislavs Sokurenko [ 2024 Aug 11 ]

Please provide ps out for 10 pollers if possible

Comment by Yury Larin [ 2024 Aug 11 ]

Yes.

ps -eo size,pid,user,command --sort -size |     awk '{ hr=$1/1024 ; printf("%13.2f Mb ",hr) } { for ( x=4 ; x<=NF ; x++ ) { printf("%s ",$x) } print "" }' |    cut -d "" -f2 | cut -d "-" -f1
         0.00 Mb COMMAND
      3546.83 Mb /usr/sbin/mariadbd
       231.28 Mb /usr/sbin/zabbix_agent2
       322.64 Mb falcon
       702.43 Mb /usr/sbin/zabbix_proxy: snmp poller #4 [got 1070 values, queued 1020 in 5 sec]
       634.22 Mb /usr/sbin/zabbix_proxy: snmp poller #5 [got 0 values, queued 0 in 5 sec]
       605.06 Mb /usr/sbin/zabbix_proxy: snmp poller #1 [got 0 values, queued 0 in 5 sec]
       588.01 Mb /usr/sbin/zabbix_proxy: snmp poller #9 [got 556 values, queued 0 in 5 sec]
       472.90 Mb /usr/sbin/zabbix_proxy: snmp poller #6 [got 0 values, queued 0 in 5 sec]
       454.34 Mb /usr/sbin/zabbix_proxy: snmp poller #2 [got 0 values, queued 0 in 5 sec]
        43.29 Mb /usr/sbin/zabbix_proxy: discovery manager #1 [processing 0 rules, 0 unsaved checks]
        91.12 Mb /usr/sbin/zabbix_proxy: http agent poller #2 [got 0 values, queued 0 in 5 sec]
       163.40 Mb /usr/sbin/zabbix_proxy: preprocessing manager #1 [queued 1388, processed 1410 values, idle 4.914197 sec during 5.030357 sec]
       112.07 Mb /usr/sbin/zabbix_proxy: http agent poller #1 [got 11 values, queued 11 in 5 sec]
       211.52 Mb /usr/sbin/zabbix_proxy: snmp poller #8 [got 0 values, queued 0 in 5 sec]
        82.30 Mb /usr/sbin/zabbix_proxy: http agent poller #3 [got 0 values, queued 0 in 5 sec]
       158.50 Mb /usr/sbin/zabbix_proxy: snmp poller #10 [got 0 values, queued 0 in 5 sec]
       141.43 Mb /usr/sbin/zabbix_proxy: snmp poller #3 [got 516 values, queued 1021 in 5 sec]
       124.19 Mb /usr/sbin/zabbix_proxy: snmp poller #7 [got 440 values, queued 448 in 5 sec]
        65.75 Mb /usr/sbin/zabbix_proxy: agent poller #1 [got 0 values, queued 0 in 5 sec]
        65.75 Mb /usr/sbin/zabbix_proxy: agent poller #2 [got 0 values, queued 0 in 5 sec]
        65.35 Mb /usr/sbin/zabbix_proxy: agent poller #3 [got 0 values, queued 0 in 5 sec]
        54.30 Mb /usr/sbin/zabbix_proxy: configuration syncer [synced config 447 bytes in 0.135812 sec, idle 10 sec]
        53.79 Mb /usr/sbin/zabbix_proxy: data sender [sent 1080 values in 0.038072 sec, idle 1 sec]
         9.69 Mb /usr/sbin/zabbix_proxy: history syncer #1 [processed 9 values in 0.000353 sec, idle 1 sec]
         9.37 Mb /usr/sbin/zabbix_proxy: icmp pinger #11 [got 0 values in 0.000072 sec, idle 1 sec]
         9.34 Mb /usr/sbin/zabbix_proxy: icmp pinger #18 [pinging hosts]
         9.34 Mb /usr/sbin/zabbix_proxy: icmp pinger #42 [got 0 values in 0.000027 sec, idle 1 sec]
         9.33 Mb /usr/sbin/zabbix_proxy: icmp pinger #25 [got 0 values in 0.000046 sec, idle 1 sec]
         9.32 Mb /usr/sbin/zabbix_proxy: icmp pinger #47 [got 0 values in 0.000057 sec, idle 1 sec]
         5.80 Mb /usr/sbin/zabbix_proxy: icmp pinger #2 [got 0 values in 0.000044 sec, idle 1 sec]
Comment by Vladislavs Sokurenko [ 2024 Aug 11 ]

Thank you, one process reached 700M, does it grow past that ?

Comment by Yury Larin [ 2024 Aug 11 ]

The process is not too fast. It usually takes about 3-5 days to see the picture. 

I will do ps again tomorrow in the morning (GMT+10). In about 12 hours and we will see for how many mb it will increase. 

Comment by Yury Larin [ 2024 Aug 11 ]

Small change for this night:

ps -eo size,pid,user,command --sort -size |     awk '{ hr=$1/1024 ; printf("%13.2f Mb ",hr) } { for ( x=4 ; x<=NF ; x++ ) { printf("%s ",$x) } print "" }' |    cut -d "" -f2 | cut -d "-" -f1
         0.00 Mb COMMAND
      3546.83 Mb /usr/sbin/mariadbd
       231.28 Mb /usr/sbin/zabbix_agent2
       322.64 Mb falcon
       702.43 Mb /usr/sbin/zabbix_proxy: snmp poller #4 [got 17 values, queued 0 in 5 sec]
       634.22 Mb /usr/sbin/zabbix_proxy: snmp poller #5 [got 0 values, queued 0 in 5 sec]
       611.20 Mb /usr/sbin/zabbix_proxy: snmp poller #1 [got 5 values, queued 0 in 5 sec]
       588.01 Mb /usr/sbin/zabbix_proxy: snmp poller #9 [got 632 values, queued 449 in 5 sec]
       472.90 Mb /usr/sbin/zabbix_proxy: snmp poller #6 [got 0 values, queued 0 in 5 sec]
       454.34 Mb /usr/sbin/zabbix_proxy: snmp poller #2 [got 0 values, queued 0 in 5 sec]
        43.29 Mb /usr/sbin/zabbix_proxy: discovery manager #1 [processing 0 rules, 0 unsaved checks]
        91.39 Mb /usr/sbin/zabbix_proxy: http agent poller #2 [got 0 values, queued 0 in 5 sec]
       165.64 Mb /usr/sbin/zabbix_proxy: preprocessing manager #1 [queued 1494, processed 1580 values, idle 4.940959 sec during 5.000807 sec]
       112.07 Mb /usr/sbin/zabbix_proxy: http agent poller #1 [got 5 values, queued 5 in 5 sec]
       211.52 Mb /usr/sbin/zabbix_proxy: snmp poller #8 [got 0 values, queued 0 in 5 sec]
        82.30 Mb /usr/sbin/zabbix_proxy: http agent poller #3 [got 0 values, queued 0 in 5 sec]
       162.46 Mb /usr/sbin/zabbix_proxy: snmp poller #3 [got 839 values, queued 595 in 5 sec]
       157.57 Mb /usr/sbin/zabbix_proxy: snmp poller #10 [got 10 values, queued 0 in 5 sec]
       119.98 Mb /usr/sbin/zabbix_proxy: snmp poller #7 [got 611 values, queued 620 in 5 sec]
        65.75 Mb /usr/sbin/zabbix_proxy: agent poller #1 [got 4 values, queued 4 in 5 sec]
        65.75 Mb /usr/sbin/zabbix_proxy: agent poller #2 [got 2 values, queued 2 in 5 sec]
        65.35 Mb /usr/sbin/zabbix_proxy: agent poller #3 [got 0 values, queued 0 in 5 sec]
        54.30 Mb /usr/sbin/zabbix_proxy: configuration syncer [synced config 447 bytes in 0.168434 sec, idle 10 sec]
        53.79 Mb /usr/sbin/zabbix_proxy: data sender [sent 604 values in 0.016129 sec, idle 1 sec]
         9.69 Mb /usr/sbin/zabbix_proxy: history syncer #1 [processed 1 values in 0.000245 sec, idle 1 sec]
         9.37 Mb /usr/sbin/zabbix_proxy: icmp pinger #11 [pinging hosts]
         9.34 Mb /usr/sbin/zabbix_proxy: icmp pinger #18 [got 0 values in 0.000024 sec, idle 1 sec]
         9.34 Mb /usr/sbin/zabbix_proxy: icmp pinger #42 [got 0 values in 0.000093 sec, idle 1 sec]
         9.33 Mb /usr/sbin/zabbix_proxy: icmp pinger #25 [got 0 values in 0.000049 sec, idle 1 sec]
         9.32 Mb /usr/sbin/zabbix_proxy: icmp pinger #47 [got 0 values in 0.000038 sec, idle 1 sec]
         5.80 Mb /usr/sbin/zabbix_proxy: icmp pinger #2 [got 0 values in 0.000049 sec, idle 1 sec]
         5.80 Mb /usr/sbin/zabbix_proxy: icmp pinger #3 [pinging hosts]
         5.80 Mb /usr/sbin/zabbix_proxy: icmp pinger #4 [got 0 values in 0.000043 sec, idle 1 sec]
         5.80 Mb /usr/sbin/zabbix_proxy: icmp pinger #5 [got 0 values in 0.000028 sec, idle 1 sec]
         5.80 Mb /usr/sbin/zabbix_proxy: icmp pinger #6 [pinging hosts]

Will check in the next 12 hours.

Comment by Yury Larin [ 2024 Aug 12 ] 
ps -eo size,pid,user,command --sort -size |     awk '{ hr=$1/1024 ; printf("%13.2f Mb ",hr) } { for ( x=4 ; x<=NF ; x++ )
{ printf("%s ",$x) }
print "" }' |    cut d "" -f2 | cut -d "" -f1
         0.00 Mb COMMAND
      3657.35 Mb /usr/sbin/mariadbd
       231.28 Mb /usr/sbin/zabbix_agent2
       322.71 Mb falcon
       702.43 Mb /usr/sbin/zabbix_proxy: snmp poller #4 [got 553 values, queued 565 in 5 sec]
       634.22 Mb /usr/sbin/zabbix_proxy: snmp poller #5 [got 0 values, queued 0 in 5 sec]
       611.20 Mb /usr/sbin/zabbix_proxy: snmp poller #1 [got 1083 values, queued 1100 in 5 sec]
       588.01 Mb /usr/sbin/zabbix_proxy: snmp poller #9 [got 0 values, queued 0 in 5 sec]
       525.98 Mb /usr/sbin/zabbix_proxy: snmp poller #2 [got 549 values, queued 569 in 5 sec]
       472.90 Mb /usr/sbin/zabbix_proxy: snmp poller #6 [got 0 values, queued 0 in 5 sec]
        43.29 Mb /usr/sbin/zabbix_proxy: discovery manager #1 [processing 0 rules, 0 unsaved checks]
       120.21 Mb /usr/sbin/zabbix_proxy: http agent poller #1 [got 4 values, queued 4 in 5 sec]
        91.39 Mb /usr/sbin/zabbix_proxy: http agent poller #2 [got 0 values, queued 0 in 5 sec]
       165.73 Mb /usr/sbin/zabbix_proxy: preprocessing manager #1 [queued 1528, processed 1601 values, idle 4.962162 sec during 5.002428 sec]
       211.52 Mb /usr/sbin/zabbix_proxy: snmp poller #8 [got 0 values, queued 0 in 5 sec]
        82.56 Mb /usr/sbin/zabbix_proxy: http agent poller #3 [got 0 values, queued 0 in 5 sec]
       181.15 Mb /usr/sbin/zabbix_proxy: snmp poller #10 [got 131 values, queued 240 in 5 sec]
       136.61 Mb /usr/sbin/zabbix_proxy: snmp poller #7 [got 540 values, queued 0 in 5 sec]
       112.27 Mb /usr/sbin/zabbix_proxy: snmp poller #3 [got 0 values, queued 0 in 5 sec]
        65.75 Mb /usr/sbin/zabbix_proxy: agent poller #1 [got 2 values, queued 2 in 5 sec]
        65.75 Mb /usr/sbin/zabbix_proxy: agent poller #2 [got 2 values, queued 2 in 5 sec]
        65.35 Mb /usr/sbin/zabbix_proxy: agent poller #3 [got 0 values, queued 0 in 5 sec]
        54.30 Mb /usr/sbin/zabbix_proxy: configuration syncer [synced config 445 bytes in 0.119392 sec, idle 10 sec]
        32.07 Mb /usr/sbin/zabbix_proxy: data sender [sent 1175 values in 0.037201 sec, idle 1 sec]
        10.68 Mb /usr/sbin/zabbix_proxy: history syncer #1 [processed 1 values in 0.000318 sec, idle 1 sec]
         9.38 Mb /usr/sbin/zabbix_proxy: icmp pinger #3 [pinging hosts]
         9.37 Mb /usr/sbin/zabbix_proxy: icmp pinger #11 [got 0 values in 0.000441 sec, idle 1 sec]
         9.34 Mb /usr/sbin/zabbix_proxy: icmp pinger #18 [got 0 values in 0.000314 sec, idle 1 sec]
         9.34 Mb /usr/sbin/zabbix_proxy: icmp pinger #42 [got 0 values in 0.000045 sec, idle 1 sec]
         9.33 Mb /usr/sbin/zabbix_proxy: icmp pinger #25 [got 0 values in 0.000523 sec, idle 1 sec]
         9.32 Mb /usr/sbin/zabbix_proxy: icmp pinger #47 [got 0 values in 0.000123 sec, idle 1 sec]
         5.80 Mb /usr/sbin/zabbix_proxy: icmp pinger #2 [got 0 values in 0.000070 sec, idle 1 sec]
         5.80 Mb /usr/sbin/zabbix_proxy: icmp pinger #4 [got 0 values in 0.000050 sec, idle 1 sec]
         5.80 Mb /usr/sbin/zabbix_proxy: icmp pinger #5 [got 0 values in 0.000814 sec, idle 1 sec]
         5.80 Mb /usr/sbin/zabbix_proxy: icmp pinger #6 [got 0 values in 0.000051 sec, idle 1 sec]
         5.80 Mb /usr/sbin/zabbix_proxy: icmp pinger #9 [pinging hosts]
Comment by Yury Larin [ 2024 Aug 12 ] 
ps -eo size,pid,user,command --sort -size |     awk '{ hr=$1/1024 ; printf("%13.2f Mb ",hr) } { for ( x=4 ; x<=NF ; x++ )
{ printf("%s ",$x) }
print "" }' |    cut d "" -f2 | cut -d "" -f1
         0.00 Mb COMMAND
      3659.30 Mb /usr/sbin/mariadbd
       231.28 Mb /usr/sbin/zabbix_agent2
       322.75 Mb falcon
       726.39 Mb /usr/sbin/zabbix_proxy: snmp poller #6 [got 0 values, queued 0 in 5 sec]
       702.43 Mb /usr/sbin/zabbix_proxy: snmp poller #4 [got 1104 values, queued 530 in 5 sec]
       634.22 Mb /usr/sbin/zabbix_proxy: snmp poller #5 [got 0 values, queued 0 in 5 sec]
       611.20 Mb /usr/sbin/zabbix_proxy: snmp poller #1 [got 0 values, queued 0 in 5 sec]
       588.01 Mb /usr/sbin/zabbix_proxy: snmp poller #9 [got 1505 values, queued 985 in 5 sec]
       525.98 Mb /usr/sbin/zabbix_proxy: snmp poller #2 [got 30 values, queued 0 in 5 sec]
        43.29 Mb /usr/sbin/zabbix_proxy: discovery manager #1 [processing 0 rules, 0 unsaved checks]
       120.36 Mb /usr/sbin/zabbix_proxy: http agent poller #1 [got 3 values, queued 4 in 5 sec]
        91.52 Mb /usr/sbin/zabbix_proxy: http agent poller #2 [got 0 values, queued 0 in 5 sec]
       166.57 Mb /usr/sbin/zabbix_proxy: preprocessing manager #1 [queued 1323, processed 1380 values, idle 5.001455 sec during 5.034993 sec]
       211.52 Mb /usr/sbin/zabbix_proxy: snmp poller #8 [got 0 values, queued 0 in 5 sec]
        82.56 Mb /usr/sbin/zabbix_proxy: http agent poller #3 [got 0 values, queued 0 in 5 sec]
       197.15 Mb /usr/sbin/zabbix_proxy: snmp poller #10 [got 489 values, queued 493 in 5 sec]
       133.63 Mb /usr/sbin/zabbix_proxy: snmp poller #3 [got 0 values, queued 0 in 5 sec]
       119.01 Mb /usr/sbin/zabbix_proxy: snmp poller #7 [got 0 values, queued 0 in 5 sec]
        65.75 Mb /usr/sbin/zabbix_proxy: agent poller #1 [got 1 values, queued 1 in 5 sec]
        65.75 Mb /usr/sbin/zabbix_proxy: agent poller #2 [got 0 values, queued 0 in 5 sec]
        65.35 Mb /usr/sbin/zabbix_proxy: agent poller #3 [got 0 values, queued 0 in 5 sec]
        54.30 Mb /usr/sbin/zabbix_proxy: configuration syncer [loading configuration]
        32.07 Mb /usr/sbin/zabbix_proxy: data sender [sent 583 values in 6.389286 sec, sending data]
        10.68 Mb /usr/sbin/zabbix_proxy: history syncer #1 [processed 1 values in 0.000268 sec, idle 1 sec]
         9.38 Mb /usr/sbin/zabbix_proxy: icmp pinger #3 [got 0 values in 0.000061 sec, idle 1 sec]
         9.37 Mb /usr/sbin/zabbix_proxy: icmp pinger #11 [pinging hosts]
         9.36 Mb /usr/sbin/zabbix_proxy: icmp pinger #17 [pinging hosts]
Comment by Yury Larin [ 2024 Aug 12 ]

 snmp poller #6 has changed RAM usage significantly.

Comment by Yury Larin [ 2024 Aug 13 ]

No big changes so far.

I still keep an eye on it.

Is it ok that it took so much RAM for snmp pollers?

Reducing to 10 pollers helped. Is it worth to allow MaxConcurrentChecksPerPoller be set to 10 000? Probably with default 1000 like now for these people who needs it?

Comment by Vladislavs Sokurenko [ 2024 Aug 13 ]

If it stabilises then it's not exactly memory leak but caching, we will try reproducing the issue and see if something can be done about it, but if there would be 10000 limit then it means it should have used same amount of memory anyway.

Comment by Yury Larin [ 2024 Aug 13 ]

Yes. But could be a bit better about cache issues which you mentioned before?

Also RAM usage increased too much. I thinks it should not take about 8GB in my case.

On 6.4 whole VM was on 2GB with about 1.5 in use. With DB. So, it was about 512MB or less for zabbix-proxy.

I will wait more without proxy reboots and see if it will take more RAM.

May be it can not release used RAM?

Comment by Yury Larin [ 2024 Aug 13 ]

And initial issue was due to have 50 processes were each did cache data and did not release it?

Comment by Vladislavs Sokurenko [ 2024 Aug 13 ]

They cache data in turns so each process can cache same data, they do not share cache unfortunately. This is most likely snmp library cache but we will get back to you on that.

Comment by Yury Larin [ 2024 Aug 13 ]

I saw that it pushed all snmp request to one process first until it reached about 1000 limit. then it added to the next one and so on.

In my case it is usually about 3 snmp pollers are in use.

The problem is I have slow not too stable links and the requests jump between processes and eat RAM.

How difficult is it to stick hosts to exact processes? Like I run 10 snmp pollers on the zabbix-proxy with 1000 hosts on it.

Then just say to the proxy to use snmp poller #1 for hosts 1-100, #2 for 101-200, #3 for 201-300 ....

Divide the load between pollers and do not allow to jump between them..

Is it possible?

Comment by Yury Larin [ 2024 Aug 14 ]

A bit more RAM used:

ps -eo size,pid,user,command --sort -size |     awk '{ hr=$1/1024 ; printf("%13.2f Mb ",hr) } { for ( x=4 ; x<=NF ; x++ )
{ printf("%s ",$x) }
print "" }' |    cut d "" -f2 | cut -d "" -f1
         0.00 Mb COMMAND
      4780.52 Mb /usr/sbin/mariadbd
       231.28 Mb /usr/sbin/zabbix_agent2
       807.65 Mb /usr/sbin/zabbix_proxy: snmp poller #5 [got 0 values, queued 0 in 5 sec]
       778.91 Mb /usr/sbin/zabbix_proxy: snmp poller #8 [got 0 values, queued 0 in 5 sec]
       322.77 Mb falcon
       732.52 Mb /usr/sbin/zabbix_proxy: snmp poller #1 [got 0 values, queued 0 in 5 sec]
       726.39 Mb /usr/sbin/zabbix_proxy: snmp poller #6 [got 44 values, queued 0 in 5 sec]
       660.81 Mb /usr/sbin/zabbix_proxy: snmp poller #3 [got 0 values, queued 0 in 5 sec]
       517.73 Mb /usr/sbin/zabbix_proxy: snmp poller #4 [got 492 values, queued 530 in 5 sec]
       473.60 Mb /usr/sbin/zabbix_proxy: snmp poller #9 [got 0 values, queued 0 in 5 sec]
       458.34 Mb /usr/sbin/zabbix_proxy: snmp poller #2 [got 1441 values, queued 1086 in 5 sec]
        43.29 Mb /usr/sbin/zabbix_proxy: discovery manager #1 [processing 0 rules, 0 unsaved checks]
       120.36 Mb /usr/sbin/zabbix_proxy: http agent poller #1 [got 6 values, queued 5 in 5 sec]
        91.65 Mb /usr/sbin/zabbix_proxy: http agent poller #2 [got 0 values, queued 0 in 5 sec]
       167.82 Mb /usr/sbin/zabbix_proxy: preprocessing manager #1 [queued 1207, processed 1281 values, idle 4.988036 sec during 5.098346 sec]
        82.56 Mb /usr/sbin/zabbix_proxy: http agent poller #3 [got 0 values, queued 0 in 5 sec]
       154.89 Mb /usr/sbin/zabbix_proxy: snmp poller #10 [got 1060 values, queued 1017 in 5 sec]
        93.62 Mb /usr/sbin/zabbix_proxy: snmp poller #7 [got 0 values, queued 0 in 5 sec]
        65.75 Mb /usr/sbin/zabbix_proxy: agent poller #1 [got 2 values, queued 2 in 5 sec]
        65.75 Mb /usr/sbin/zabbix_proxy: agent poller #2 [got 4 values, queued 4 in 5 sec]
        65.35 Mb /usr/sbin/zabbix_proxy: agent poller #3 [got 0 values, queued 0 in 5 sec]
        54.30 Mb /usr/sbin/zabbix_proxy: configuration syncer [synced config 446 bytes in 0.139104 sec, idle 10 sec]
        32.16 Mb /usr/sbin/zabbix_proxy: data sender [sent 726 values in 0.021924 sec, idle 1 sec]
        10.68 Mb /usr/sbin/zabbix_proxy: history syncer #1 [processed 1 values in 0.000136 sec, idle 1 sec]
         9.38 Mb /usr/sbin/zabbix_proxy: icmp pinger #3 [got 0 values in 0.000046 sec, idle 1 sec]
         9.37 Mb /usr/sbin/zabbix_proxy: icmp pinger #11 [pinging hosts]
         9.36 Mb /usr/sbin/zabbix_proxy: icmp pinger #17 [got 0 values in 0.000425 sec, idle 1 sec]
         9.34 Mb /usr/sbin/zabbix_proxy: icmp pinger #18 [got 0 values in 0.000033 sec, idle 1 sec]
         9.34 Mb /usr/sbin/zabbix_proxy: icmp pinger #42 [got 0 values in 0.000025 sec, idle 1 sec]
         9.33 Mb /usr/sbin/zabbix_proxy: icmp pinger #25 [got 0 values in 0.000327 sec, idle 1 sec]
         9.32 Mb /usr/sbin/zabbix_proxy: icmp pinger #47 [got 108 values in 10.010672 sec, idle 1 sec]
         5.88 Mb /usr/sbin/zabbix_proxy: history syncer #4 [processed 0 values in 0.000025 sec, idle 1 sec]
         5.80 Mb /usr/sbin/zabbix_proxy: icmp pinger #2 [pinging hosts]
         5.80 Mb /usr/sbin/zabbix_proxy: icmp pinger #4 [pinging hosts]
Comment by Yury Larin [ 2024 Aug 14 ]

poller # 8 somehow reduced RAM usage a bit:

ps -eo size,pid,user,command --sort -size |     awk '{ hr=$1/1024 ; printf("%13.2f Mb ",hr) } { for ( x=4 ; x<=NF ; x++ ) { printf("%s ",$x) } print "" }' |    cut -d "" -f2 | cut -d "-" -f1
         0.00 Mb COMMAND
      4780.52 Mb /usr/sbin/mariadbd
       231.28 Mb /usr/sbin/zabbix_agent2
       778.91 Mb /usr/sbin/zabbix_proxy: snmp poller #8 [got 65 values, queued 50 in 5 sec]
       749.89 Mb /usr/sbin/zabbix_proxy: snmp poller #5 [got 461 values, queued 949 in 5 sec]
       322.80 Mb falcon
       732.52 Mb /usr/sbin/zabbix_proxy: snmp poller #1 [got 0 values, queued 0 in 5 sec]
       726.39 Mb /usr/sbin/zabbix_proxy: snmp poller #6 [got 1 values, queued 0 in 5 sec]
       660.81 Mb /usr/sbin/zabbix_proxy: snmp poller #3 [got 0 values, queued 0 in 5 sec]
       578.76 Mb /usr/sbin/zabbix_proxy: snmp poller #9 [got 895 values, queued 905 in 5 sec]
       517.73 Mb /usr/sbin/zabbix_proxy: snmp poller #4 [got 0 values, queued 0 in 5 sec]
       373.33 Mb /usr/sbin/zabbix_proxy: snmp poller #2 [got 842 values, queued 180 in 5 sec]
       367.92 Mb /usr/sbin/zabbix_proxy: snmp poller #7 [got 0 values, queued 0 in 5 sec]
        43.29 Mb /usr/sbin/zabbix_proxy: discovery manager #1 [processing 0 rules, 0 unsaved checks]
       120.36 Mb /usr/sbin/zabbix_proxy: http agent poller #1 [got 4 values, queued 4 in 5 sec]
        91.65 Mb /usr/sbin/zabbix_proxy: http agent poller #2 [got 0 values, queued 0 in 5 sec]
       167.82 Mb /usr/sbin/zabbix_proxy: preprocessing manager #1 [queued 1596, processed 1650 values, idle 5.039217 sec during 5.122872 sec]
        82.56 Mb /usr/sbin/zabbix_proxy: http agent poller #3 [got 0 values, queued 0 in 5 sec]
       154.89 Mb /usr/sbin/zabbix_proxy: snmp poller #10 [got 0 values, queued 0 in 5 sec]
        65.75 Mb /usr/sbin/zabbix_proxy: agent poller #1 [got 2 values, queued 2 in 5 sec]
        65.75 Mb /usr/sbin/zabbix_proxy: agent poller #2 [got 1 values, queued 1 in 5 sec]
        65.35 Mb /usr/sbin/zabbix_proxy: agent poller #3 [got 0 values, queued 0 in 5 sec]
        54.30 Mb /usr/sbin/zabbix_proxy: configuration syncer [synced config 446 bytes in 0.517205 sec, idle 10 sec]
        32.16 Mb /usr/sbin/zabbix_proxy: data sender [sent 444 values in 0.016541 sec, idle 1 sec]
        10.68 Mb /usr/sbin/zabbix_proxy: history syncer #1 [processed 0 values in 0.000006 sec, idle 1 sec]
         9.38 Mb /usr/sbin/zabbix_proxy: icmp pinger #3 [got 0 values in 0.000060 sec, idle 1 sec]
         9.38 Mb /usr/sbin/zabbix_proxy: icmp pinger #40 [got 0 values in 0.000016 sec, idle 1 sec]
         9.37 Mb /usr/sbin/zabbix_proxy: icmp pinger #11 [got 0 values in 0.000026 sec, idle 1 sec]
         9.36 Mb /usr/sbin/zabbix_proxy: icmp pinger #17 [got 0 values in 0.000019 sec, idle 1 sec]
         9.34 Mb /usr/sbin/zabbix_proxy: icmp pinger #18 [pinging hosts]
         9.34 Mb /usr/sbin/zabbix_proxy: icmp pinger #42 [got 0 values in 0.000037 sec, idle 1 sec]
         9.33 Mb /usr/sbin/zabbix_proxy: icmp pinger #25 [pinging hosts]
         9.32 Mb /usr/sbin/zabbix_proxy: icmp pinger #47 [pinging hosts]
         5.88 Mb /usr/sbin/zabbix_proxy: history syncer #4 [processed 0 values in 0.000024 sec, idle 1 sec]

 

 

Comment by Yury Larin [ 2024 Aug 15 ] 
ps -eo size,pid,user,command --sort -size |     awk '{ hr=$1/1024 ; printf("%13.2f Mb ",hr) } { for ( x=4 ; x<=NF ; x++ ) { printf("%s ",$x) } print "" }' |    cut -d "" -f2 | cut -d "-" -f1
         0.00 Mb COMMAND
      4780.70 Mb /usr/sbin/mariadbd
       231.28 Mb /usr/sbin/zabbix_agent2
       749.89 Mb /usr/sbin/zabbix_proxy: snmp poller #5 [got 687 values, queued 502 in 5 sec]
       322.86 Mb falcon
       707.75 Mb /usr/sbin/zabbix_proxy: snmp poller #8 [got 511 values, queued 528 in 5 sec]
       662.52 Mb /usr/sbin/zabbix_proxy: snmp poller #3 [got 0 values, queued 0 in 5 sec]
       610.13 Mb /usr/sbin/zabbix_proxy: snmp poller #1 [got 0 values, queued 0 in 5 sec]
       578.76 Mb /usr/sbin/zabbix_proxy: snmp poller #9 [got 1049 values, queued 506 in 5 sec]
       526.74 Mb /usr/sbin/zabbix_proxy: snmp poller #6 [got 0 values, queued 0 in 5 sec]
       439.66 Mb /usr/sbin/zabbix_proxy: snmp poller #4 [got 0 values, queued 0 in 5 sec]
       373.33 Mb /usr/sbin/zabbix_proxy: snmp poller #2 [got 414 values, queued 942 in 5 sec]
       367.92 Mb /usr/sbin/zabbix_proxy: snmp poller #7 [got 0 values, queued 0 in 5 sec]
        43.29 Mb /usr/sbin/zabbix_proxy: discovery manager #1 [processing 0 rules, 0 unsaved checks]
       120.36 Mb /usr/sbin/zabbix_proxy: http agent poller #1 [got 10 values, queued 9 in 5 sec]
        91.95 Mb /usr/sbin/zabbix_proxy: http agent poller #2 [got 0 values, queued 0 in 5 sec]
       168.98 Mb /usr/sbin/zabbix_proxy: preprocessing manager #1 [queued 1769, processed 1826 values, idle 4.942024 sec during 5.001221 sec]
        82.56 Mb /usr/sbin/zabbix_proxy: http agent poller #3 [got 0 values, queued 0 in 5 sec]
       154.89 Mb /usr/sbin/zabbix_proxy: snmp poller #10 [got 0 values, queued 0 in 5 sec]
        65.75 Mb /usr/sbin/zabbix_proxy: agent poller #1 [got 0 values, queued 0 in 5 sec]
        65.75 Mb /usr/sbin/zabbix_proxy: agent poller #2 [got 0 values, queued 0 in 5 sec]
        65.35 Mb /usr/sbin/zabbix_proxy: agent poller #3 [got 0 values, queued 0 in 5 sec]
        59.00 Mb /usr/sbin/zabbix_proxy: configuration syncer [synced config 444 bytes in 0.127440 sec, idle 10 sec]
        33.66 Mb /usr/sbin/zabbix_proxy: data sender [sent 0 values in 0.023018 sec, idle 1 sec]
        17.50 Mb /usr/sbin/zabbix_proxy: history syncer #1 [processed 1 values in 0.000314 sec, idle 1 sec]
         9.38 Mb /usr/sbin/zabbix_proxy: icmp pinger #3 [got 0 values in 0.000019 sec, idle 1 sec]
         9.38 Mb /usr/sbin/zabbix_proxy: icmp pinger #40 [got 0 values in 0.000050 sec, idle 1 sec]
         9.37 Mb /usr/sbin/zabbix_proxy: icmp pinger #11 [got 0 values in 0.000177 sec, idle 1 sec]
         9.36 Mb /usr/sbin/zabbix_proxy: icmp pinger #17 [got 0 values in 0.000026 sec, idle 1 sec]
         9.34 Mb /usr/sbin/zabbix_proxy: icmp pinger #18 [got 0 values in 0.000022 sec, idle 1 sec]
         9.34 Mb /usr/sbin/zabbix_proxy: icmp pinger #42 [got 0 values in 0.000143 sec, idle 1 sec]
Comment by Vladislavs Sokurenko [ 2024 Aug 16 ]

Could you please be so kind and try executing zabbix_server -R snmp_cache_reload and see if it becomes better for some time

Comment by Yury Larin [ 2024 Aug 16 ]

 

zabbix_server -R snmp_cache_reload
Runtime control command was forwarded successfully

No effect.

Comment by Yury Larin [ 2024 Aug 16 ] 
ps -eo size,pid,user,command --sort -size |     awk '{ hr=$1/1024 ; printf("%13.2f Mb ",hr) } { for ( x=4 ; x<=NF ; x++ ) { printf("%s ",$x) } print "" }' |    cut -d "" -f2 | cut -d "-" -f1
         0.00 Mb COMMAND
      4868.86 Mb /usr/sbin/mariadbd
       231.28 Mb /usr/sbin/zabbix_agent2
       870.21 Mb /usr/sbin/zabbix_proxy: snmp poller #4 [got 1373 values, queued 997 in 5 sec]
       322.91 Mb falcon
       707.75 Mb /usr/sbin/zabbix_proxy: snmp poller #8 [got 0 values, queued 0 in 5 sec]
       694.51 Mb /usr/sbin/zabbix_proxy: snmp poller #5 [got 740 values, queued 647 in 5 sec]
       662.52 Mb /usr/sbin/zabbix_proxy: snmp poller #3 [got 775 values, queued 927 in 5 sec]
       610.13 Mb /usr/sbin/zabbix_proxy: snmp poller #1 [got 0 values, queued 0 in 5 sec]
       578.76 Mb /usr/sbin/zabbix_proxy: snmp poller #9 [got 343 values, queued 508 in 5 sec]
       526.74 Mb /usr/sbin/zabbix_proxy: snmp poller #6 [got 0 values, queued 0 in 5 sec]
       373.33 Mb /usr/sbin/zabbix_proxy: snmp poller #2 [got 0 values, queued 0 in 5 sec]
       367.92 Mb /usr/sbin/zabbix_proxy: snmp poller #7 [got 76 values, queued 63 in 5 sec]
        43.29 Mb /usr/sbin/zabbix_proxy: discovery manager #1 [processing 0 rules, 0 unsaved checks]
       120.49 Mb /usr/sbin/zabbix_proxy: http agent poller #1 [got 3 values, queued 3 in 5 sec]
        91.95 Mb /usr/sbin/zabbix_proxy: http agent poller #2 [got 0 values, queued 0 in 5 sec]
       169.57 Mb /usr/sbin/zabbix_proxy: preprocessing manager #1 [queued 1599, processed 1654 values, idle 4.907716 sec during 5.002101 sec]
        82.56 Mb /usr/sbin/zabbix_proxy: http agent poller #3 [got 0 values, queued 0 in 5 sec]
       154.89 Mb /usr/sbin/zabbix_proxy: snmp poller #10 [got 0 values, queued 0 in 5 sec]
        65.75 Mb /usr/sbin/zabbix_proxy: agent poller #1 [got 3 values, queued 3 in 5 sec]
        65.75 Mb /usr/sbin/zabbix_proxy: agent poller #2 [got 1 values, queued 1 in 5 sec]
        65.35 Mb /usr/sbin/zabbix_proxy: agent poller #3 [got 0 values, queued 0 in 5 sec]
        54.30 Mb /usr/sbin/zabbix_proxy: configuration syncer [synced config 446 bytes in 0.158896 sec, idle 10 sec]
        32.16 Mb /usr/sbin/zabbix_proxy: data sender [sent 230 values in 0.010541 sec, idle 1 sec]
        17.50 Mb /usr/sbin/zabbix_proxy: history syncer #1 [processed 1 values in 0.000412 sec, idle 1 sec]
         9.38 Mb /usr/sbin/zabbix_proxy: icmp pinger #3 [pinging hosts]
         9.38 Mb /usr/sbin/zabbix_proxy: icmp pinger #40 [got 0 values in 0.000025 sec, idle 1 sec]
         9.37 Mb /usr/sbin/zabbix_proxy: icmp pinger #11 [got 0 values in 0.000016 sec, idle 1 sec]
         9.36 Mb /usr/sbin/zabbix_proxy: icmp pinger #17 [got 0 values in 0.000061 sec, idle 1 sec]
         9.34 Mb /usr/sbin/zabbix_proxy: icmp pinger #18 [got 0 values in 0.000055 sec, idle 1 sec]
         9.34 Mb /usr/sbin/zabbix_proxy: icmp pinger #42 [got 0 values in 0.000087 sec, idle 1 sec]
         9.33 Mb /usr/sbin/zabbix_proxy: icmp pinger #25 [got 0 values in 0.000512 sec, idle 1 sec]
         9.32 Mb /usr/sbin/zabbix_proxy: icmp pinger #47 [got 0 values in 0.000027 sec, idle 1 sec]
         5.88 Mb /usr/sbin/zabbix_proxy: history syncer #4 [processed 0 values in 0.000025 sec, idle 1 sec]
Comment by Yury Larin [ 2024 Aug 17 ]

I tried:

zabbix_proxy -R snmp_cache_reload
Runtime control command was forwarded successfully

No effect as well.

 

Comment by Yury Larin [ 2024 Aug 17 ] 
ps -eo size,pid,user,command --sort -size |     awk '{ hr=$1/1024 ; printf("%13.2f Mb ",hr) } { for ( x=4 ; x<=NF ; x++ ) { printf("%s ",$x) } print "" }' |    cut -d "" -f2 | cut -d "-" -f1
         0.00 Mb COMMAND
      4868.86 Mb /usr/sbin/mariadbd
       231.28 Mb /usr/sbin/zabbix_agent2
       870.21 Mb /usr/sbin/zabbix_proxy: snmp poller #4 [got 11 values, queued 0 in 5 sec]
       322.91 Mb falcon
       707.75 Mb /usr/sbin/zabbix_proxy: snmp poller #8 [got 39 values, queued 0 in 5 sec]
       694.51 Mb /usr/sbin/zabbix_proxy: snmp poller #5 [got 18 values, queued 0 in 5 sec]
       662.52 Mb /usr/sbin/zabbix_proxy: snmp poller #3 [got 0 values, queued 0 in 5 sec]
       610.13 Mb /usr/sbin/zabbix_proxy: snmp poller #1 [got 1061 values, queued 996 in 5 sec]
       578.76 Mb /usr/sbin/zabbix_proxy: snmp poller #9 [got 0 values, queued 0 in 5 sec]
       471.94 Mb /usr/sbin/zabbix_proxy: snmp poller #6 [got 1364 values, queued 1001 in 5 sec]
       373.33 Mb /usr/sbin/zabbix_proxy: snmp poller #2 [got 535 values, queued 1026 in 5 sec]
       367.92 Mb /usr/sbin/zabbix_proxy: snmp poller #7 [got 0 values, queued 0 in 5 sec]
        43.29 Mb /usr/sbin/zabbix_proxy: discovery manager #1 [processing 0 rules, 0 unsaved checks]
       120.49 Mb /usr/sbin/zabbix_proxy: http agent poller #1 [got 3 values, queued 3 in 5 sec]
        91.95 Mb /usr/sbin/zabbix_proxy: http agent poller #2 [got 0 values, queued 0 in 5 sec]
       169.57 Mb /usr/sbin/zabbix_proxy: preprocessing manager #1 [queued 1738, processed 1797 values, idle 4.939684 sec during 5.023037 sec]
        82.56 Mb /usr/sbin/zabbix_proxy: http agent poller #3 [got 0 values, queued 0 in 5 sec]
       154.89 Mb /usr/sbin/zabbix_proxy: snmp poller #10 [got 0 values, queued 0 in 5 sec]
        65.75 Mb /usr/sbin/zabbix_proxy: agent poller #1 [got 0 values, queued 0 in 5 sec]
        65.75 Mb /usr/sbin/zabbix_proxy: agent poller #2 [got 0 values, queued 0 in 5 sec]
        65.35 Mb /usr/sbin/zabbix_proxy: agent poller #3 [got 0 values, queued 0 in 5 sec]
        54.30 Mb /usr/sbin/zabbix_proxy: configuration syncer [synced config 446 bytes in 0.451857 sec, idle 10 sec]
        32.16 Mb /usr/sbin/zabbix_proxy: data sender [sent 490 values in 0.019445 sec, idle 1 sec]
        17.50 Mb /usr/sbin/zabbix_proxy: history syncer #1 [processed 1 values in 0.000329 sec, idle 1 sec]
         9.38 Mb /usr/sbin/zabbix_proxy: icmp pinger #3 [got 0 values in 0.000063 sec, idle 1 sec]
         9.38 Mb /usr/sbin/zabbix_proxy: icmp pinger #40 [got 0 values in 0.000930 sec, idle 1 sec]
         9.37 Mb /usr/sbin/zabbix_proxy: icmp pinger #11 [pinging hosts]
         9.36 Mb /usr/sbin/zabbix_proxy: icmp pinger #17 [got 0 values in 0.000061 sec, idle 1 sec]
         9.34 Mb /usr/sbin/zabbix_proxy: icmp pinger #18 [got 0 values in 0.000033 sec, idle 1 sec]
         9.34 Mb /usr/sbin/zabbix_proxy: icmp pinger #42 [pinging hosts]
         9.33 Mb /usr/sbin/zabbix_proxy: icmp pinger #25 [got 0 values in 0.000051 sec, idle 1 sec]
         9.32 Mb /usr/sbin/zabbix_proxy: icmp pinger #47 [pinging hosts]
         5.88 Mb /usr/sbin/zabbix_proxy: history syncer #4 [processed 0 values in 0.000013 sec, idle 1 sec]
         5.82 Mb /usr/sbin/zabbix_proxy: history syncer #3 [processed 0 values in 0.000017 sec, idle 1 sec]
Comment by Yury Larin [ 2024 Aug 17 ]

A bit more free RAM:

ps -eo size,pid,user,command --sort -size |     awk '{ hr=$1/1024 ; printf("%13.2f Mb ",hr) } { for ( x=4 ; x<=NF ; x++ ) { printf("%s ",$x) } print "" }' |    cut -d "" -f2 | cut -d "-" -f1
         0.00 Mb COMMAND
      4868.86 Mb /usr/sbin/mariadbd
       231.28 Mb /usr/sbin/zabbix_agent2
       870.21 Mb /usr/sbin/zabbix_proxy: snmp poller #4 [got 0 values, queued 0 in 5 sec]
       322.91 Mb falcon
       619.44 Mb /usr/sbin/zabbix_proxy: snmp poller #5 [got 0 values, queued 0 in 5 sec]
       610.13 Mb /usr/sbin/zabbix_proxy: snmp poller #1 [got 27 values, queued 0 in 5 sec]
       578.76 Mb /usr/sbin/zabbix_proxy: snmp poller #9 [got 18 values, queued 0 in 5 sec]
       561.03 Mb /usr/sbin/zabbix_proxy: snmp poller #3 [got 0 values, queued 0 in 5 sec]
       471.94 Mb /usr/sbin/zabbix_proxy: snmp poller #6 [got 1217 values, queued 696 in 5 sec]
       373.33 Mb /usr/sbin/zabbix_proxy: snmp poller #2 [got 0 values, queued 0 in 5 sec]
        43.29 Mb /usr/sbin/zabbix_proxy: discovery manager #1 [processing 0 rules, 0 unsaved checks]
       120.49 Mb /usr/sbin/zabbix_proxy: http agent poller #1 [got 6 values, queued 8 in 5 sec]
        91.95 Mb /usr/sbin/zabbix_proxy: http agent poller #2 [got 0 values, queued 0 in 5 sec]
       169.57 Mb /usr/sbin/zabbix_proxy: preprocessing manager #1 [queued 1597, processed 1693 values, idle 4.944020 sec during 5.007903 sec]
       252.95 Mb /usr/sbin/zabbix_proxy: snmp poller #8 [got 1457 values, queued 935 in 5 sec]
        82.56 Mb /usr/sbin/zabbix_proxy: http agent poller #3 [got 0 values, queued 0 in 5 sec]
       154.89 Mb /usr/sbin/zabbix_proxy: snmp poller #10 [got 0 values, queued 0 in 5 sec]
       103.02 Mb /usr/sbin/zabbix_proxy: snmp poller #7 [got 0 values, queued 0 in 5 sec]
        65.75 Mb /usr/sbin/zabbix_proxy: agent poller #1 [got 1 values, queued 1 in 5 sec]
        65.75 Mb /usr/sbin/zabbix_proxy: agent poller #2 [got 1 values, queued 1 in 5 sec]
        65.35 Mb /usr/sbin/zabbix_proxy: agent poller #3 [got 0 values, queued 0 in 5 sec]
        54.30 Mb /usr/sbin/zabbix_proxy: configuration syncer [synced config 2 bytes in 0.131447 sec, idle 10 sec]
        32.16 Mb /usr/sbin/zabbix_proxy: data sender [sent 580 values in 0.127352 sec, idle 1 sec]
        17.50 Mb /usr/sbin/zabbix_proxy: history syncer #1 [processed 1 values in 0.000219 sec, idle 1 sec]
         9.38 Mb /usr/sbin/zabbix_proxy: icmp pinger #3 [got 0 values in 0.000067 sec, idle 1 sec]
         9.38 Mb /usr/sbin/zabbix_proxy: icmp pinger #40 [got 0 values in 0.000054 sec, idle 1 sec]
         9.37 Mb /usr/sbin/zabbix_proxy: icmp pinger #11 [got 0 values in 0.000055 sec, idle 1 sec]
         9.36 Mb /usr/sbin/zabbix_proxy: icmp pinger #17 [pinging hosts]
         9.34 Mb /usr/sbin/zabbix_proxy: icmp pinger #18 [got 0 values in 0.000040 sec, idle 1 sec]
         9.34 Mb /usr/sbin/zabbix_proxy: icmp pinger #42 [pinging hosts]
         9.33 Mb /usr/sbin/zabbix_proxy: icmp pinger #25 [got 0 values in 0.000062 sec, idle 1 sec]
         9.32 Mb /usr/sbin/zabbix_proxy: icmp pinger #47 [got 0 values in 0.000320 sec, idle 1 sec]
         5.88 Mb /usr/sbin/zabbix_proxy: history syncer #4 [processed 0 values in 0.000023 sec, idle 1 sec]
         5.82 Mb /usr/sbin/zabbix_proxy: history syncer #3 [processed 0 values in 0.000017 sec, idle 1 sec]
Comment by Yury Larin [ 2024 Aug 18 ]

2 days after cache reload:

ps -eo size,pid,user,command --sort -size |     awk '{ hr=$1/1024 ; printf("%13.2f Mb ",hr) } { for ( x=4 ; x<=NF ; x++ ) { printf("%s ",$x) } print "" }' |    cut -d "" -f2 | cut -d "-" -f1
         0.00 Mb COMMAND
      4868.86 Mb /usr/sbin/mariadbd
       231.28 Mb /usr/sbin/zabbix_agent2
       870.21 Mb /usr/sbin/zabbix_proxy: snmp poller #4 [got 230 values, queued 115 in 5 sec]
       827.39 Mb /usr/sbin/zabbix_proxy: snmp poller #2 [got 924 values, queued 793 in 5 sec]
       806.59 Mb /usr/sbin/zabbix_proxy: snmp poller #9 [got 0 values, queued 0 in 5 sec]
       322.96 Mb falcon
       619.44 Mb /usr/sbin/zabbix_proxy: snmp poller #5 [got 0 values, queued 0 in 5 sec]
       610.13 Mb /usr/sbin/zabbix_proxy: snmp poller #1 [got 682 values, queued 744 in 5 sec]
       565.50 Mb /usr/sbin/zabbix_proxy: snmp poller #3 [got 0 values, queued 0 in 5 sec]
       471.94 Mb /usr/sbin/zabbix_proxy: snmp poller #6 [got 68 values, queued 339 in 5 sec]
        43.29 Mb /usr/sbin/zabbix_proxy: discovery manager #1 [processing 0 rules, 0 unsaved checks]
       120.49 Mb /usr/sbin/zabbix_proxy: http agent poller #1 [got 4 values, queued 5 in 5 sec]
        91.95 Mb /usr/sbin/zabbix_proxy: http agent poller #2 [got 0 values, queued 0 in 5 sec]
       169.61 Mb /usr/sbin/zabbix_proxy: preprocessing manager #1 [queued 1136, processed 1216 values, idle 4.934276 sec during 5.007030 sec]
       252.95 Mb /usr/sbin/zabbix_proxy: snmp poller #8 [got 0 values, queued 0 in 5 sec]
       228.77 Mb /usr/sbin/zabbix_proxy: snmp poller #7 [got 322 values, queued 491 in 5 sec]
        82.56 Mb /usr/sbin/zabbix_proxy: http agent poller #3 [got 0 values, queued 0 in 5 sec]
       194.55 Mb /usr/sbin/zabbix_proxy: snmp poller #10 [got 134 values, queued 153 in 5 sec]
        65.75 Mb /usr/sbin/zabbix_proxy: agent poller #1 [got 1 values, queued 1 in 5 sec]
        65.75 Mb /usr/sbin/zabbix_proxy: agent poller #2 [got 0 values, queued 0 in 5 sec]
        65.35 Mb /usr/sbin/zabbix_proxy: agent poller #3 [got 0 values, queued 0 in 5 sec]
        54.30 Mb /usr/sbin/zabbix_proxy: configuration syncer [synced config 2 bytes in 0.207794 sec, idle 10 sec]
        32.16 Mb /usr/sbin/zabbix_proxy: data sender [sent 906 values in 0.048013 sec, idle 1 sec]
        17.50 Mb /usr/sbin/zabbix_proxy: history syncer #1 [processed 3 values in 0.000309 sec, idle 1 sec]
         9.38 Mb /usr/sbin/zabbix_proxy: icmp pinger #3 [got 0 values in 0.000027 sec, idle 1 sec]
         9.38 Mb /usr/sbin/zabbix_proxy: icmp pinger #40 [got 0 values in 0.000051 sec, idle 1 sec]
         9.37 Mb /usr/sbin/zabbix_proxy: icmp pinger #11 [got 0 values in 0.000131 sec, idle 1 sec]
         9.36 Mb /usr/sbin/zabbix_proxy: icmp pinger #4 [got 0 values in 0.000030 sec, idle 1 sec]
         9.36 Mb /usr/sbin/zabbix_proxy: icmp pinger #17 [got 0 values in 0.000026 sec, idle 1 sec]
         9.34 Mb /usr/sbin/zabbix_proxy: icmp pinger #18 [got 0 values in 0.000055 sec, idle 1 sec]
         9.34 Mb /usr/sbin/zabbix_proxy: icmp pinger #42 [got 0 values in 0.000057 sec, idle 1 sec]
         9.33 Mb /usr/sbin/zabbix_proxy: icmp pinger #25 [got 0 values in 0.000039 sec, idle 1 sec]
Comment by Vladislavs Sokurenko [ 2024 Aug 19 ]

Based in data its looks like memory significantly dropped after cache reload:

From:

707.75 Mb /usr/sbin/zabbix_proxy: snmp poller #8 [got 39 values, queued 0 in 5 sec]

To:

252.95 Mb /usr/sbin/zabbix_proxy: snmp poller #8 [got 1457 values, queued 935 in 5 sec]

But it would be more correct to check pss like this please:

ps -eo pss,pid,user,command --sort -pss
Comment by Vladislavs Sokurenko [ 2024 Aug 19 ]

Reproduced issue when device is restarted frequently then cache grows until snmp_cache_reload is called, it is proposed to periodically clear this cache to avoid such situations.

Comment by Yury Larin [ 2024 Aug 19 ] 
ps -eo pss,pid,user,command --sort -pss
  PSS     PID USER     COMMAND
2521516 780949 mysql   /usr/sbin/mariadbd
899224 3689927 root    /usr/bin/python3 /usr/lib/zabbix/externalscripts/arp-zabbix-sync.py
879967 846567 zabbix   /usr/sbin/zabbix_proxy: snmp poller #4 [got 1486 values, queued 1072 in 5 sec]
835646 846563 zabbix   /usr/sbin/zabbix_proxy: snmp poller #2 [got 984 values, queued 962 in 5 sec]
805016 846577 zabbix   /usr/sbin/zabbix_proxy: snmp poller #9 [got 0 values, queued 0 in 5 sec]
622078 846570 zabbix   /usr/sbin/zabbix_proxy: snmp poller #5 [got 0 values, queued 0 in 5 sec]
611533 846561 zabbix   /usr/sbin/zabbix_proxy: snmp poller #1 [got 2 values, queued 0 in 5 sec]
560993 846565 zabbix   /usr/sbin/zabbix_proxy: snmp poller #3 [got 0 values, queued 0 in 5 sec]
461965 846571 zabbix   /usr/sbin/zabbix_proxy: snmp poller #6 [got 0 values, queued 0 in 5 sec]
242675 846575 zabbix   /usr/sbin/zabbix_proxy: snmp poller #8 [got 0 values, queued 0 in 5 sec]
189503 846174 zabbix   /usr/sbin/zabbix_proxy: preprocessing manager #1 [queued 1773, processed 1829 v
185301 846579 zabbix   /usr/sbin/zabbix_proxy: snmp poller #10 [got 937 values, queued 465 in 5 sec]
164992 846175 zabbix   /usr/sbin/zabbix_proxy: data sender [sent 253 values in 0.027739 sec, idle 1 se
162446 846573 zabbix   /usr/sbin/zabbix_proxy: snmp poller #7 [got 74 values, queued 528 in 5 sec]
148865 846180 zabbix   /usr/sbin/zabbix_proxy: history syncer #1 [processed 2 values in 0.000237 sec,
54082  846134 zabbix   /usr/sbin/zabbix_proxy: configuration syncer [synced config 447 bytes in 0.1252
52363     159 Debian-+ /usr/sbin/snmptrapd -LOw -f udp:162 udp6:162
39791 1501524 root     /lib/systemd/systemd-journald
28825      77 root     falcon-sensor
25869     156 zabbix   /usr/sbin/zabbix_agent2 -c /etc/zabbix/zabbix_agent2.conf
10955  846182 zabbix   /usr/sbin/zabbix_proxy: history syncer #3 [processed 0 values in 0.000015 sec,
 9269  846183 zabbix   /usr/sbin/zabbix_proxy: history syncer #4 [processed 0 values in 0.000021 sec,
 8943  846181 zabbix   /usr/sbin/zabbix_proxy: history syncer #2 [processed 0 values in 0.000014 sec,
 6580  846523 zabbix   /usr/sbin/zabbix_proxy: icmp pinger #25 [got 0 values in 0.000034 sec, idle 1 s
 6341  846551 zabbix   /usr/sbin/zabbix_proxy: http agent poller #1 [got 3 values, queued 3 in 5 sec]
Comment by Yury Larin [ 2024 Aug 19 ] 
zabbix_proxy -R snmp_cache_reload
Runtime control command was forwarded successfully
Comment by Yury Larin [ 2024 Aug 19 ] 
ps -eo pss,pid,user,command --sort -pss
  PSS     PID USER     COMMAND
2522048 780949 mysql   /usr/sbin/mariadbd
899357 3689927 root    /usr/bin/python3 /usr/lib/zabbix/externalscripts/arp-zabbix-sync.py
879967 846567 zabbix   /usr/sbin/zabbix_proxy: snmp poller #4 [got 20 values, queued 0 in 5 sec]
835702 846563 zabbix   /usr/sbin/zabbix_proxy: snmp poller #2 [got 20 values, queued 0 in 5 sec]
622082 846570 zabbix   /usr/sbin/zabbix_proxy: snmp poller #5 [got 929 values, queued 1370 in 5 sec]
611534 846561 zabbix   /usr/sbin/zabbix_proxy: snmp poller #1 [got 11 values, queued 0 in 5 sec]
494441 846565 zabbix   /usr/sbin/zabbix_proxy: snmp poller #3 [got 0 values, queued 0 in 5 sec]
461965 846571 zabbix   /usr/sbin/zabbix_proxy: snmp poller #6 [got 1307 values, queued 719 in 5 sec]
242675 846575 zabbix   /usr/sbin/zabbix_proxy: snmp poller #8 [got 0 values, queued 0 in 5 sec]
209559 846577 zabbix   /usr/sbin/zabbix_proxy: snmp poller #9 [got 1030 values, queued 1061 in 5 sec]
189517 846174 zabbix   /usr/sbin/zabbix_proxy: preprocessing manager #1 [queued 1711, processed 1786 v
185302 846579 zabbix   /usr/sbin/zabbix_proxy: snmp poller #10 [got 0 values, queued 0 in 5 sec]
164993 846175 zabbix   /usr/sbin/zabbix_proxy: data sender [sent 917 values in 0.030096 sec, idle 1 se
162447 846573 zabbix   /usr/sbin/zabbix_proxy: snmp poller #7 [got 0 values, queued 0 in 5 sec]
148865 846180 zabbix   /usr/sbin/zabbix_proxy: history syncer #1 [processed 1 values in 0.000080 sec,
54083  846134 zabbix   /usr/sbin/zabbix_proxy: configuration syncer [synced config 447 bytes in 0.1279
52363     159 Debian-+ /usr/sbin/snmptrapd -LOw -f udp:162 udp6:162
40975 1501524 root     /lib/systemd/systemd-journald
28828      77 root     falcon-sensor
26133     156 zabbix   /usr/sbin/zabbix_agent2 -c /etc/zabbix/zabbix_agent2.conf
10955  846182 zabbix   /usr/sbin/zabbix_proxy: history syncer #3 [processed 0 values in 0.000019 sec,
 9269  846183 zabbix   /usr/sbin/zabbix_proxy: history syncer #4 [processed 0 values in 0.000013 sec,
 8943  846181 zabbix   /usr/sbin/zabbix_proxy: history syncer #2 [processed 0 values in 0.000013 sec,
Comment by Yury Larin [ 2024 Aug 20 ]

Probably server has the same problem. But I do not use server for so many slow devices.

Comment by Vladislavs Sokurenko [ 2024 Aug 20 ]

It will not free memory this way due to a way that memory management works if it already grown but if there would be script that would call zabbix_proxy -R snmp_cache_reload every hour then it should have not grown

Comment by Yury Larin [ 2024 Aug 20 ]

Since I reduced snmp pollers to 10 it eats not too much RAM but much more than should be.

So, I will wait the fix and hopefully the issue will be fixed.

What about:

How difficult is it to stick hosts to exact processes? Like I run 10 snmp pollers on the zabbix-proxy with 1000 hosts on it.

Then just say to the proxy to use snmp poller #1 for hosts 1-100, #2 for 101-200, #3 for 201-300 ....

Divide the load between pollers and do not allow to jump between them..

Is it possible?

Comment by Vladislavs Sokurenko [ 2024 Aug 20 ]

Current fix will handle devices being constantly added and removed but remaining in cache after removal. It can be possible to pin pollers to hosts but still will need to cleanup removed devices.

Comment by Vladislavs Sokurenko [ 2024 Aug 20 ]

Fixed in pull request feature/ZBX-24989-7.0

Comment by Vladislavs Sokurenko [ 2024 Aug 22 ]

Fixed in:

Comment by Vladislavs Sokurenko [ 2024 Nov 28 ]

Please also check if ZBXNEXT-9625 helps

Generated at Sun Apr 06 09:10:34 EEST 2025 using Jira 9.12.4#9120004-sha1:625303b708afdb767e17cb2838290c41888e9ff0.