Currently a single poller can execute only one check at the same time. The overall throughput can be increased by running multiple pollers, leading to some setups with 1000 pollers (and even that is not enough in some cases).
Majority of the checks are not cpu intensive and spend most of the time waiting for network response. This means the polling process could be parallelized by starting new checks while others are still waiting on network.
Add new poller processes capable of executing multiple checks of specific type at the same time:
- zabbix agent
- snmp agent
- ... ?
Those pollers will have separate queues.
Checks supporting asynchronous execution must be excluded from 'unreachable poller' logic (when timeouted checks are moved to unreachable poller to avoid blocking working checks).
There must be maximum concurrent check limit for each new poller type. At first stage hardcoded limit should be used while later a configurable way will be added.