Server: performance improvements (ZBXNEXT-318)

[ZBXNEXT-325] Parrallel writing of historical data by N processes Created: 2010 Apr 23  Updated: 2017 May 31  Resolved: 2010 Apr 29

Status: Closed
Project: ZABBIX FEATURE REQUESTS
Component/s: None
Affects Version/s: None
Fix Version/s: 1.8.3

Type: Change Request (Sub-task) Priority: Major
Reporter: Alexei Vladishev Assignee: Unassigned
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Comments   
Comment by Alexei Vladishev [ 2010 Apr 26 ]

It will be implemented by introducing a new structure, which will keep list of items currently processed by DB Syncers.

If an itemid is buing processed, other DBSyncer won't pick it up from the history buffer.

Comment by Alexander Vladishev [ 2010 Apr 29 ]

Added new configuration parameter StartDBSyncers.
Available from v. pre1.8.3, r11801

Comment by michael chan [ 2012 Oct 11 ]

How will we know if this is working?

Reason I ask, is that I have Zabbix monitoring with a small setup:
Hosts: 127/20/64
Items: 4615/117/31
Triggers: 1934/17
Required NVPS 132.67

And the issue is that 27 of these hosts I am actively logging log data (e.g. error.log, access.log, etc.), roughly 2 logs each host, at 10 lines per second. So the theoretical maximum is 540 lines/sec of log(text) data. Also I have a trigger assigned to all the logs - nothing special, just a simple REGEX on each, and performing an alert if it is matched.

Now when I do a mysql processlist, I almost always see just 1 thread running, even if DBSyncers is 4, 16, 32. The DBSyncers value to be honest is irrelevant.

Now when I increased my lines/sec to 20 lps on all my Items that gather log data, The Zabix text write cache & History write cache start decreasing slowly, basically unable to write the data out to the DB quick enough.

I have tried increasing the server size, increase the database memory & # of cpus (I am running all this with an Amazon c1.medium EC2 instance and m1.small instance, and have increased the EC2 instance to a c1.xlarge and RDS to m1.large) and have not seen ANY improvement at all. The larger Amazon instances I have tried each have roughly 4x the memory and 4x the # of cores.

Is there any way to see what the issue is? Why are there not more dbsyncers actively writing data and not all sleeping? The Amazon RDS mysql instance has all the defaults which are for the most part set it and forget it, and I have tried tweaking the Zabbix Server parameters and nothing makes any difference. Zabbix acts almost as if it is single-threaded, and is not scaling according to the # of cpus (neither server cpus nor database/RDS cpus).

Thanks for any help you can provide.

Generated at Tue Apr 23 15:24:20 EEST 2024 using Jira 9.12.4#9120004-sha1:625303b708afdb767e17cb2838290c41888e9ff0.