Zabbix Server (RHEL 7.6), AWS EC2 c4.xlarge
Zabbix frontend (RHEL 7.6), AWS EC2 m4.large
Zabbix DB: AWS RDS (MariaDB 10.0.24), db.m4.xlarge

Hello Valeriy,

Can you please attach performance graphs from Monitoring → Graphs → select your Zabbix Server showing time period of 1 day and the same graphs for one of the proxies used:

• Zabbix cache usage, % free
• Zabbix data gathering process busy %
• Zabbix internal process busy %
• Zabbix server performance

Also required:

• Zabbix Server configuration file;
• Zabbix Proxy configuration file and total number of proxies used;
• What traffic filtering systems are used and what value of the RTT and latency between Zabbix Server and Proxies;

Do you have network performance graphs? With latency, packet loss, etc..?

Regards,
Edgar

Hello Edgar,

We don't have some metrics you've requested but I can enable them now and attach them a bit later if it's okay for you. For now, here's what I have:

Zabbix server config:
zabbix_server.conf

Zabbix proxy config:
zabbix_proxy.conf
We have 74 Zabbix proxies located across different regions in AWS. They have typical configuration provisioned by Puppet. Usually we adjust proxy cache, config frequency, proxy cache size, pollers, pollers unreachable and java pollers (rarely). Just in case, here's another Zabbix proxy config from a proxy with the heaviest load in our infrastructure:
zabbix_proxy.conf

^{We don't filter traffic. Connections to Zabbix server are limited by RHEL 7.6 firewall.}

^{Ping from Zabbix server to proxy:}

[ec2-user*********** ~]$ ping *********** 
PING *********** (*********** ) 56(84) bytes of data.
64 bytes from *********** : icmp_seq=1 ttl=64 time=0.280 ms
64 bytes from *********** : icmp_seq=2 ttl=64 time=0.259 ms
64 bytes from *********** : icmp_seq=3 ttl=64 time=0.211 ms
64 bytes from *********** : icmp_seq=4 ttl=64 time=0.231 ms
64 bytes from *********** : icmp_seq=5 ttl=64 time=0.284 ms
64 bytes from *********** : icmp_seq=6 ttl=64 time=0.228 ms
64 bytes from *********** : icmp_seq=7 ttl=64 time=0.247 ms
64 bytes from *********** : icmp_seq=8 ttl=64 time=0.248 ms
^C
--- ***********  ping statistics ---
8 packets transmitted, 8 received, 0% packet loss, time 7000ms
rtt min/avg/max/mdev = 0.211/0.248/0.284/0.028 ms

Unfortunately, we don't have any network performance graphs except ones we have in AWS.
Zabbix proxy:

^{Zabbix server:}

^

^

^{That's the most of the information I was able to gather. Feel free to ask if you need anything else.}

^{Best regards,}

^Valery

Just in case, here's the list of item keys from metrics we are currently collection on our Zabbix server:

agent.ping

system.run["mysql -h {$DBINSTANCE} -u{$DBUSER} -p{$DBPASS} -NBL <{$DBQUERY} | grep -w history"]

system.run["mysql -h {$DBINSTANCE} -u{$DBUSER} -p{$DBPASS} -NBL <{$DBQUERY} | grep -w history_log"]

system.run["mysql -h {$DBINSTANCE} -u{$DBUSER} -p{$DBPASS} -NBL <{$DBQUERY} | grep -w history_str"]

system.run["mysql -h {$DBINSTANCE} -u{$DBUSER} -p{$DBPASS} -NBL <{$DBQUERY} | grep -w history_text"]

system.run["mysql -h {$DBINSTANCE} -u{$DBUSER} -p{$DBPASS} -NBL <{$DBQUERY} | grep -w history_uint"]

system.run["mysql -h {$DBINSTANCE} -u{$DBUSER} -p{$DBPASS} -NBL <{$DBQUERY} | grep -w trends"]

system.run["mysql -h {$DBINSTANCE} -u{$DBUSER} -p{$DBPASS} -NBL <{$DBQUERY} | grep -w trends_uint"]

zabbix[lld_queue]
zabbix[process,alerter,avg,busy]
zabbix[process,configuration syncer,avg,busy]
zabbix[process,db watchdog,avg,busy]
zabbix[process,discoverer,avg,busy]
zabbix[process,escalator,avg,busy]
zabbix[process,history syncer,avg,busy]
zabbix[process,housekeeper,avg,busy]
zabbix[process,http poller,avg,busy]
zabbix[process,java poller,avg,busy]
zabbix[process,poller,avg,busy]
zabbix[process,proxy poller,avg,busy]
zabbix[process,self-monitoring,avg,busy]
zabbix[process,timer,avg,busy]
zabbix[process,trapper,avg,busy]
zabbix[process,unreachable poller,avg,busy]
zabbix[queue,1m]
zabbix[queue,5m]
zabbix[queue,10m]
zabbix[rcache,buffer,pfree]
zabbix[vcache,buffer,pfree]
zabbix[vcache,cache,hits]
zabbix[vcache,cache,misses]
zabbix[vcache,cache,mode]
zabbix[wcache,history,pfree]
zabbix[wcache,index,pfree]
zabbix[wcache,trend,pfree]
zabbix[wcache,values]

Hello Valeriy,

We need all requested information, also please note network performance could not be tested with small packet size (64 byte). Also please clarify the number of conntrack connections on Zabbix Server.

Regards,
Edgar

Could you please advise me a better way to test network performance which will be acceptable for you?

So, here's the graphs for Zabbix server. We don't use Zabbix frontend graphs, instead we use Grafana, hope it's okay. Some poller processes are disabled (vmware, ipmi and some other), so they are not present on the graph.

Zabbix proxy graphs within 1 day.

Hello Valeriy,

Please provide strace output for trapper process with log file (debug 4 required for the processes)

ps aux|grep -i trapper ### to get trapper process number 
zabbix_server -R log_level_increase=trapper,<Process number>
sudo strace -s 256 -T -tt -f -o /tmp/zabbix.trapper.strace.out -p <PID_from_first_step>
sudo strace -c -s 100 -T -tt -fp <PID_from_first_step>

Regards,
Edgar