-
New Feature Request
-
Resolution: Unresolved
-
Medium
-
None
-
4.0.19
-
None
We currently maintain a Zabbix 4.0 installation with an elasticsearch-backed history store. It maintains about 30 to 60 days of historical data. We have recently run into a problem where the Zabbix server sporadically starts initiating an abnormally high volume of scroll queries against the Elasticsearch cluster; the cause of this is unknown and we are currently investigating with support.
This aside we have noticed that when Zabbix makes a query against Elasticsearch, it does so seemingly with no time bounds. https://github.com/zabbix/zabbix/blob/da0e68e91a55320fdd1d5f42154788364f02f999/src/libs/zbxhistory/history_elastic.c#L665
"%s/%s*/_search?scroll=10s"
"host/data_type*/_search?scroll=10s"
We would like to bound these queries in some way; we don't generally use data past 1-2 days for trigger evaluation, having the queries unbounded causes massive search overhead.
The Elasticsearch query can be time bounded either:
- directly in the query with a time range
- or with date math index names https://www.elastic.co/guide/en/elasticsearch/reference/current/date-math-index-names.html
i.e. /<uint-{now/d{yyyy-MM-dd}}>,<uint-{now/d-1d{yyyy-MM-dd}}>/_search?scroll=10s instead of /uint*/_search?scroll=10s
last 2 days of data
The massive query load has caused monitoring downtime for us, so we went ahead and mitigated this with an nginx proxy to rewrite elasticsearch queries, properly bounding them to our constraints. I will link the workaround in the comments below.
Ideally, for us, the Zabbix Server would allow an admin operation to either a relative time range (2days) or a specific date math index pattern to bound the server's search queries. This would reduce load for trigger evaluation. The Zabbix GUI would be unaffected and still using the wildcard suffix, so it would be able to query all of the data.
I am envisioning a server configuration that allows an administrator to replace the wildcarded "uint*" index suffix with a more target data math selection, such as "<uint-{now/d{yyyy-MM-dd}}>,<uint-{now/d-1d{yyyy-MM-dd}}>"; of course per data-type. This can significantly reduce query load for large data sets.
Query examples:
POST /log*/values/_search?scroll=10s HTTP/1.1 DELETE /_search/scroll/DnF1ZXJ5VGhlbkZldGNoBQAAAAACKgVNFlBaRnA2dzYyUnltNXMxT0I2cFNlc0EAAAAAAioFUBZQWkZwNnc2MlJ5bTVzMU9CNnBTZXNBAAAAAAIqBU4WUFpGcDZ3NjJSeW01czFPQjZwU2VzQQAAAAACKgVPFlBaRnA2dzYyUnltNXMxT0I2cFNlc0EAAAAAAioFURZQWkZwNnc2MlJ5bTVzMU9CNnBTZXNB HTTP/1.1 POST /log*/values/_search?scroll=10s HTTP/1.1 DELETE /_search/scroll/DnF1ZXJ5VGhlbkZldGNoBQAAAAACKgVSFlBaRnA2dzYyUnltNXMxT0I2cFNlc0EAAAAAAioFUxZQWkZwNnc2MlJ5bTVzMU9CNnBTZXNBAAAAAAIqBVUWUFpGcDZ3NjJSeW01czFPQjZwU2VzQQAAAAACKgVWFlBaRnA2dzYyUnltNXMxT0I2cFNlc0EAAAAAAioFVBZQWkZwNnc2MlJ5bTVzMU9CNnBTZXNB HTTP/1.1 POST /uint*/values/_search?scroll=10s HTTP/1.1 DELETE /_search/scroll/DnF1ZXJ5VGhlbkZldGNoCgAAAAACKgVXFlBaRnA2dzYyUnltNXMxT0I2cFNlc0EAAAAAAioFWBZQWkZwNnc2MlJ5bTVzMU9CNnBTZXNBAAAAAAIqBVkWUFpGcDZ3NjJSeW01czFPQjZwU2VzQQAAAAACKgVaFlBaRnA2dzYyUnltNXMxT0I2cFNlc0EAAAAAAioFXxZQWkZwNnc2MlJ5bTVzMU9CNnBTZXNBAAAAAAIqBWAWUFpGcDZ3NjJSeW01czFPQjZwU2VzQQAAAAACKgVeFlBaRnA2dzYyUnltNXMxT0I2cFNlc0EAAAAAAioFWxZQWkZwNnc2MlJ5bTVzMU9CNnBTZXNBAAAAAAIqBVwWUFpGcDZ3NjJSeW01czF POST /log*/values/_search?scroll=10s HTTP/1.1 DELETE /_search/scroll/DnF1ZXJ5VGhlbkZldGNoBQAAAAACKgVhFlBaRnA2dzYyUnltNXMxT0I2cFNlc0EAAAAAAioFYhZQWkZwNnc2MlJ5bTVzMU9CNnBTZXNBAAAAAAIqBWMWUFpGcDZ3NjJSeW01czFPQjZwU2VzQQAAAAACKgVlFlBaRnA2dzYyUnltNXMxT0I2cFNlc0EAAAAAAioFZBZQWkZwNnc2MlJ5bTVzMU9CNnBTZXNB HTTP/1.1 POST /dbl*/values/_search?scroll=10s HTTP/1.1 DELETE /_search/scroll/DnF1ZXJ5VGhlbkZldGNoCgAAAAACKgVmFlBaRnA2dzYyUnltNXMxT0I2cFNlc0EAAAAAAioFZxZQWkZwNnc2MlJ5bTVzMU9CNnBTZXNBAAAAAAIqBW8WUFpGcDZ3NjJSeW01czFPQjZwU2VzQQAAAAACKgVoFlBaRnA2dzYyUnltNXMxT0I2cFNlc0EAAAAAAioFahZQWkZwNnc2MlJ5bTVzMU9CNnBTZXNBAAAAAAIqBW0WUFpGcDZ3NjJSeW01czFPQjZwU2VzQQAAAAACKgVuFlBaRnA2dzYyUnltNXMxT0I2cFNlc0EAAAAAAioFaRZQWkZwNnc2MlJ5bTVzMU9CNnBTZXNBAAAAAAIqBWsWUFpGcDZ3NjJSeW01czF POST /log*/values/_search?scroll=10s HTTP/1.1 DELETE /_search/scroll/DnF1ZXJ5VGhlbkZldGNoBQAAAAACKgVwFlBaRnA2dzYyUnltNXMxT0I2cFNlc0EAAAAAAioFcRZQWkZwNnc2MlJ5bTVzMU9CNnBTZXNBAAAAAAIqBXQWUFpGcDZ3NjJSeW01czFPQjZwU2VzQQAAAAACKgVzFlBaRnA2dzYyUnltNXMxT0I2cFNlc0EAAAAAAioFchZQWkZwNnc2MlJ5bTVzMU9CNnBTZXNB HTTP/1.1 POST /uint*/values/_search?scroll=10s HTTP/1.1 DELETE /_search/scroll/DnF1ZXJ5VGhlbkZldGNoCgAAAAACKgV1FlBaRnA2dzYyUnltNXMxT0I2cFNlc0EAAAAAAioFdhZQWkZwNnc2MlJ5bTVzMU9CNnBTZXNBAAAAAAIqBX0WUFpGcDZ3NjJSeW01czFPQjZwU2VzQQAAAAACKgV8FlBaRnA2dzYyUnltNXMxT0I2cFNlc0EAAAAAAioFdxZQWkZwNnc2MlJ5bTVzMU9CNnBTZXNBAAAAAAIqBXkWUFpGcDZ3NjJSeW01czFPQjZwU2VzQQAAAAACKgV-FlBaRnA2dzYyUnltNXMxT0I2cFNlc0EAAAAAAioFeBZQWkZwNnc2MlJ5bTVzMU9CNnBTZXNBAAAAAAIqBXoWUFpGcDZ3NjJSeW01czF POST /uint*/values/_search?scroll=10s HTTP/1.1 DELETE /_search/scroll/DnF1ZXJ5VGhlbkZldGNoCgAAAAACKgV_FlBaRnA2dzYyUnltNXMxT0I2cFNlc0EAAAAAAioFgBZQWkZwNnc2MlJ5bTVzMU9CNnBTZXNBAAAAAAIqBYgWUFpGcDZ3NjJSeW01czFPQjZwU2VzQQAAAAACKgWCFlBaRnA2dzYyUnltNXMxT0I2cFNlc0EAAAAAAioFhhZQWkZwNnc2MlJ5bTVzMU9CNnBTZXNBAAAAAAIqBYUWUFpGcDZ3NjJSeW01czFPQjZwU2VzQQAAAAACKgWHFlBaRnA2dzYyUnltNXMxT0I2cFNlc0EAAAAAAioFgRZQWkZwNnc2MlJ5bTVzMU9CNnBTZXNBAAAAAAIqBYMWUFpGcDZ3NjJSeW01czF POST /log*/values/_search?scroll=10s HTTP/1.1 DELETE /_search/scroll/DnF1ZXJ5VGhlbkZldGNoBQAAAAACKgWJFlBaRnA2dzYyUnltNXMxT0I2cFNlc0EAAAAAAioFihZQWkZwNnc2MlJ5bTVzMU9CNnBTZXNBAAAAAAIqBY0WUFpGcDZ3NjJSeW01czFPQjZwU2VzQQAAAAACKgWLFlBaRnA2dzYyUnltNXMxT0I2cFNlc0EAAAAAAioFjBZQWkZwNnc2MlJ5bTVzMU9CNnBTZXNB HTTP/1.1 POST /log*/values/_search?scroll=10s HTTP/1.1 DELETE /_search/scroll/DnF1ZXJ5VGhlbkZldGNoBQAAAAACKgWOFlBaRnA2dzYyUnltNXMxT0I2cFNlc0EAAAAAAioFkRZQWkZwNnc2MlJ5bTVzMU9CNnBTZXNBAAAAAAIqBY8WUFpGcDZ3NjJSeW01czFPQjZwU2VzQQAAAAACKgWQFlBaRnA2dzYyUnltNXMxT0I2cFNlc0EAAAAAAioFkhZQWkZwNnc2MlJ5bTVzMU9CNnBTZXNB HTTP/1.1 POST /uint*/values/_search?scroll=10s HTTP/1.1 DELETE /_search/scroll/DnF1ZXJ5VGhlbkZldGNoCgAAAAACKgWTFlBaRnA2dzYyUnltNXMxT0I2cFNlc0EAAAAAAioFlBZQWkZwNnc2MlJ5bTVzMU9CNnBTZXNBAAAAAAIqBZUWUFpGcDZ3NjJSeW01czFPQjZwU2VzQQAAAAACKgWaFlBaRnA2dzYyUnltNXMxT0I2cFNlc0EAAAAAAioFmxZQWkZwNnc2MlJ5bTVzMU9CNnBTZXNBAAAAAAIqBZwWUFpGcDZ3NjJSeW01czFPQjZwU2VzQQAAAAACKgWWFlBaRnA2dzYyUnltNXMxT0I2cFNlc0EAAAAAAioFlxZQWkZwNnc2MlJ5bTVzMU9CNnBTZXNBAAAAAAIqBZgWUFpGcDZ3NjJSeW01czF POST /dbl*/values/_search?scroll=10s HTTP/1.1 POST /_search/scroll HTTP/1.1 DELETE /_search/scroll/DnF1ZXJ5VGhlbkZldGNoCgAAAAACKgWdFlBaRnA2dzYyUnltNXMxT0I2cFNlc0EAAAAAAioFnhZQWkZwNnc2MlJ5bTVzMU9CNnBTZXNBAAAAAAIqBaQWUFpGcDZ3NjJSeW01czFPQjZwU2VzQQAAAAACKgWjFlBaRnA2dzYyUnltNXMxT0I2cFNlc0EAAAAAAioFpRZQWkZwNnc2MlJ5bTVzMU9CNnBTZXNBAAAAAAIqBaYWUFpGcDZ3NjJSeW01czFPQjZwU2VzQQAAAAACKgWgFlBaRnA2dzYyUnltNXMxT0I2cFNlc0EAAAAAAioFnxZQWkZwNnc2MlJ5bTVzMU9CNnBTZXNBAAAAAAIqBaEWUFpGcDZ3NjJSeW01czF POST /log*/values/_search?scroll=10s HTTP/1.1 DELETE /_search/scroll/DnF1ZXJ5VGhlbkZldGNoBQAAAAACKgWqFlBaRnA2dzYyUnltNXMxT0I2cFNlc0EAAAAAAioFpxZQWkZwNnc2MlJ5bTVzMU9CNnBTZXNBAAAAAAIqBasWUFpGcDZ3NjJSeW01czFPQjZwU2VzQQAAAAACKgWoFlBaRnA2dzYyUnltNXMxT0I2cFNlc0EAAAAAAioFqRZQWkZwNnc2MlJ5bTVzMU9CNnBTZXNB HTTP/1.1 POST /uint*/values/_search?scroll=10s HTTP/1.1 POST /log*/values/_search?scroll=10s HTTP/1.1 DELETE /_search/scroll/DnF1ZXJ5VGhlbkZldGNoBQAAAAACKgWzFlBaRnA2dzYyUnltNXMxT0I2cFNlc0EAAAAAAioFtBZQWkZwNnc2MlJ5bTVzMU9CNnBTZXNBAAAAAAIqBbUWUFpGcDZ3NjJSeW01czFPQjZwU2VzQQAAAAACKgW2FlBaRnA2dzYyUnltNXMxT0I2cFNlc0EAAAAAAioFtxZQWkZwNnc2MlJ5bTVzMU9CNnBTZXNB HTTP/1.1 POST /log*/values/_search?scroll=10s HTTP/1.1 DELETE /_search/scroll/DnF1ZXJ5VGhlbkZldGNoBQAAAAACKgW7FlBaRnA2dzYyUnltNXMxT0I2cFNlc0EAAAAAAioFvRZQWkZwNnc2MlJ5bTVzMU9CNnBTZXNBAAAAAAIqBb4WUFpGcDZ3NjJSeW01czFPQjZwU2VzQQAAAAACKgW_FlBaRnA2dzYyUnltNXMxT0I2cFNlc0EAAAAAAioFvBZQWkZwNnc2MlJ5bTVzMU9CNnBTZXNB HTTP/1.1 DELETE /_search/scroll/DnF1ZXJ5VGhlbkZldGNoCgAAAAACKgWuFlBaRnA2dzYyUnltNXMxT0I2cFNlc0EAAAAAAioFrBZQWkZwNnc2MlJ5bTVzMU9CNnBTZXNBAAAAAAIqBbkWUFpGcDZ3NjJSeW01czFPQjZwU2VzQQAAAAACKgW4FlBaRnA2dzYyUnltNXMxT0I2cFNlc0EAAAAAAioFshZQWkZwNnc2MlJ5bTVzMU9CNnBTZXNBAAAAAAIqBa0WUFpGcDZ3NjJSeW01czFPQjZwU2VzQQAAAAACKgW6FlBaRnA2dzYyUnltNXMxT0I2cFNlc0EAAAAAAioFrxZQWkZwNnc2MlJ5bTVzMU9CNnBTZXNBAAAAAAIqBbAWUFpGcDZ3NjJSeW01czF POST /log*/values/_search?scroll=10s HTTP/1.1 DELETE /_search/scroll/DnF1ZXJ5VGhlbkZldGNoBQAAAAACKgXAFlBaRnA2dzYyUnltNXMxT0I2cFNlc0EAAAAAAioFxBZQWkZwNnc2MlJ5bTVzMU9CNnBTZXNBAAAAAAIqBcEWUFpGcDZ3NjJSeW01czFPQjZwU2VzQQAAAAACKgXCFlBaRnA2dzYyUnltNXMxT0I2cFNlc0EAAAAAAioFwxZQWkZwNnc2MlJ5bTVzMU9CNnBTZXNB HTTP/1.1 POST /uint*/values/_search?scroll=10s HTTP/1.1 DELETE /_search/scroll/DnF1ZXJ5VGhlbkZldGNoCgAAAAACKgXFFlBaRnA2dzYyUnltNXMxT0I2cFNlc0EAAAAAAioFxhZQWkZwNnc2MlJ5bTVzMU9CNnBTZXNBAAAAAAIqBccWUFpGcDZ3NjJSeW01czFPQjZwU2VzQQAAAAACKgXIFlBaRnA2dzYyUnltNXMxT0I2cFNlc0EAAAAAAioFzRZQWkZwNnc2MlJ5bTVzMU9CNnBTZXNBAAAAAAIqBc4WUFpGcDZ3NjJSeW01czFPQjZwU2VzQQAAAAACKgXMFlBaRnA2dzYyUnltNXMxT0I2cFNlc0EAAAAAAioFyhZQWkZwNnc2MlJ5bTVzMU9CNnBTZXNBAAAAAAIqBckWUFpGcDZ3NjJSeW01czF POST /log*/values/_search?scroll=10s HTTP/1.1 DELETE /_search/scroll/DnF1ZXJ5VGhlbkZldGNoBQAAAAACKgXPFlBaRnA2dzYyUnltNXMxT0I2cFNlc0EAAAAAAioF0BZQWkZwNnc2MlJ5bTVzMU9CNnBTZXNBAAAAAAIqBdEWUFpGcDZ3NjJSeW01czFPQjZwU2VzQQAAAAACKgXSFlBaRnA2dzYyUnltNXMxT0I2cFNlc0EAAAAAAioF0xZQWkZwNnc2MlJ5bTVzMU9CNnBTZXNB HTTP/1.1 POST /log*/values/_search?scroll=10s HTTP/1.1 DELETE /_search/scroll/DnF1ZXJ5VGhlbkZldGNoBQAAAAACKgXUFlBaRnA2dzYyUnltNXMxT0I2cFNlc0EAAAAAAioF1RZQWkZwNnc2MlJ5bTVzMU9CNnBTZXNBAAAAAAIqBdgWUFpGcDZ3NjJSeW01czFPQjZwU2VzQQAAAAACKgXWFlBaRnA2dzYyUnltNXMxT0I2cFNlc0EAAAAAAioF1xZQWkZwNnc2MlJ5bTVzMU9CNnBTZXNB HTTP/1.1 POST /log*/values/_search?scroll=10s HTTP/1.1 DELETE /_search/scroll/DnF1ZXJ5VGhlbkZldGNoBQAAAAACKgXZFlBaRnA2dzYyUnltNXMxT0I2cFNlc0EAAAAAAioF2hZQWkZwNnc2MlJ5bTVzMU9CNnBTZXNBAAAAAAIqBdsWUFpGcDZ3NjJSeW01czFPQjZwU2VzQQAAAAACKgXdFlBaRnA2dzYyUnltNXMxT0I2cFNlc0EAAAAAAioF3BZQWkZwNnc2MlJ5bTVzMU9CNnBTZXNB HTTP/1.1 POST /log*/values/_search?scroll=10s HTTP/1.1 DELETE /_search/scroll/DnF1ZXJ5VGhlbkZldGNoBQAAAAACKgXeFlBaRnA2dzYyUnltNXMxT0I2cFNlc0EAAAAAAioF3xZQWkZwNnc2MlJ5bTVzMU9CNnBTZXNBAAAAAAIqBeIWUFpGcDZ3NjJSeW01czFPQjZwU2VzQQAAAAACKgXgFlBaRnA2dzYyUnltNXMxT0I2cFNlc0EAAAAAAioF4RZQWkZwNnc2MlJ5bTVzMU9CNnBTZXNB HTTP/1.1 POST /log*/values/_search?scroll=10s HTTP/1.1 DELETE /_search/scroll/DnF1ZXJ5VGhlbkZldGNoBQAAAAACKgXjFlBaRnA2dzYyUnltNXMxT0I2cFNlc0EAAAAAAioF5BZQWkZwNnc2MlJ5bTVzMU9CNnBTZXNBAAAAAAIqBeUWUFpGcDZ3NjJSeW01czFPQjZwU2VzQQAAAAACKgXmFlBaRnA2dzYyUnltNXMxT0I2cFNlc0EAAAAAAioF5xZQWkZwNnc2MlJ5bTVzMU9CNnBTZXNB HTTP/1.1 POST /log*/values/_search?scroll=10s HTTP/1.1 DELETE /_search/scroll/DnF1ZXJ5VGhlbkZldGNoBQAAAAACKgXoFlBaRnA2dzYyUnltNXMxT0I2cFNlc0EAAAAAAioF6hZQWkZwNnc2MlJ5bTVzMU9CNnBTZXNBAAAAAAIqBekWUFpGcDZ3NjJSeW01czFPQjZwU2VzQQAAAAACKgXrFlBaRnA2dzYyUnltNXMxT0I2cFNlc0EAAAAAAioF7BZQWkZwNnc2MlJ5bTVzMU9CNnBTZXNB HTTP/1.1 POST /log*/values/_search?scroll=10s HTTP/1.1 DELETE /_search/scroll/DnF1ZXJ5VGhlbkZldGNoBQAAAAACKgXtFlBaRnA2dzYyUnltNXMxT0I2cFNlc0EAAAAAAioF8RZQWkZwNnc2MlJ5bTVzMU9CNnBTZXNBAAAAAAIqBe4WUFpGcDZ3NjJSeW01czFPQjZwU2VzQQAAAAACKgXvFlBaRnA2dzYyUnltNXMxT0I2cFNlc0EAAAAAAioF8BZQWkZwNnc2MlJ5bTVzMU9CNnBTZXNB HTTP/1.1 POST /dbl*/values/_search?scroll=10s HTTP/1.1