Network traffic compression was advertised to provide 5 times improvement in required bandwidth with "no impact on CPU or memory usage". Why can't it be used for sending LLD data to preprocessing workers?
Complexity of preprocessing steps shouldn't be the issue since it only affects preprocessing workers and you can have as many of them as you like.
wiper: I would say - no noticeable impact on CPU usage. But that's with large data. With smaller data packets it might have negative result. Currently we are limited by preprocessing manager performance, so we are looking either how to reduce or share it.
I did test pure preprocessing performance, without pushing data into history cache. With contested history cache the results would be even worse.
On the other hand if the compression is done in data gathering processes (is that what you were thinking?), then preprocessing manager has only to cache/forward the compressed data - so that should help with performance when large data (LLD) is processed.
cyclone Honestly speaking, I wasn't thinking that far. 
But it sounds like a sensible approach to offload the preprocessing manager. And of course you are not obliged to compress all, small data can go uncompressed. It is a matter of protocol design to allow both types of messages. Since the protocol isn't for public use you are free to design it the way you like it.
As far as I recall, one and the only preprocessing manager was needed to keep the guarantee that the data which came first will be processed by triggers and actions first regardless of preprocessing steps it has to go through.
wiper: yes, but that guarantee is lost in history cache.
cyclone Well, I meant for items in one group of interdependent triggers... Isn't this guarantee still true if we neglect the fact that internal item processing is prioritized?
Is there a reason why preprocessing can't be moved to zabbix-agent side? this way the load can be distributed. Also with options like threshold/discard, the final transmitted data amount can be considerably smaller.