-
Change Request
-
Resolution: Fixed
-
Critical
-
3.4.5
-
Sprint 33, Sprint 34
-
1
In order to properly scale history data in elasticsearch, we should use multiple indices per type, and the common way to split them is by date.
With only one index, we can't dynamically add shards as index grows (a reindex is needed otherwise), we can't store fresh data in hot nodes and move old ones to cold nodes, etc..
IMO there are two ways to achieve date based indices setup :
- let zabbix server put data directly in the good index (automatically created through adequate template, by ES)
Example (with by-day splitting) :POST /_bulk?refresh=true {"index":{"_index":"uint-2018.01.08","_type":"values"}} {"itemid":56085,"value":"5363994624","clock":1515406785,"ns":760871775,"ttl":31536000} {"index":{"_index":"uint-2018.01.08","_type":"values"}} {"itemid":25425,"value":"1078325248","clock":1515406785,"ns":761604920,"ttl":31536000}
Frontend would have to search in every indices
Example :POST dbl*/values/_search?scroll=10s {"query":{"bool":{"must":[{"terms":{"itemid":["69859","69860"]}},{"range":{"clock":{"gte":1515404223}}},{"range":{"clock":{"lte":1515407277}}}]}},"sort":["clock"],"size":10000}
- Let Elasticsearch put data in appropriate index, using "Date Index Name Processor Pipeline" (https://www.elastic.co/guide/en/elasticsearch/reference/6.1/date-index-name-processor.html) :
Example :
https://support.zabbix.com/browse/ZBXNEXT-4002?focusedCommentId=254700&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-254700
No changes needed in Frontend
- is duplicated by
-
ZBX-13746 ElasticSearch: max documents per index - the limit is 2,147,483,519
- Closed