We currently have about 6K hosts (and expect to have noticeably more in the future), and a config size of around 1.5GB. The problem is that this configuration takes over 2 minutes load, and while it does it places a significant strain on the database, to the point where we are getting alerts every time the configuration is loaded. This is compounded by the fact that almost all our hosts are virtual hosts on the cloud, and hence we have a fairly high churn of hosts being added and disabled/removed - each time a host comes up or goes down we create/update/delete the relevant host entry in zabbix and trigger a configuration reload (CacheUpdateFrequency is set to 1h, since we almost always explicitly trigger configuration reloads). Besides the database load, the fact that it takes 2 or 3 minutes to reload the config means that it's not uncommon to get alerts when hosts are going down, since the hosts will have terminated by the time the configuration is finally reloaded (though we've been able to mitigate this somewhat, but if things continue to grow and we see 5 or more minutes to reload the configuration this will become more of an issue again).

Now, the interesting thing about the above is that in the vast majority of cases when the configuration is reloaded only a small fraction of the config has actually changed - typically just one or at most a handful of hosts. This leads to the idea that it would be great if zabbix could only reload that part that changed. One idea (and I'm sure you have others and better ones) is simple: track the list of changed hosts (including templates), i.e. any time a configuration change is made create an entry in a change table specifying which host(s) got affected; then when reloading the configuration, only reload the data for those hosts (and clear the change table). In the case of templates, you'd need to also walk up the template-links to find the upstream templates and hosts that are affected in order to get the full list of hosts/templates to reload.

One nice side-effect of something like the above would be that one could probably get rid of CacheUpdateFrequency and just check for changes every minute or even more often, since if the change-list is empty there would be nothing to do.