-
Problem report
-
Resolution: Unresolved
-
Major
-
None
-
6.0.6
Steps to reproduce:
- Setup Elasticsearch 7.x cluster and use it as a history storage(all data: HistoryStorageTypes=uint,dbl,str,log,text)
- Fill zabbix server with data. It works when hosts number is about 3000
- Navigate to /zabbix.php?action=dashboard.view
Result:
See log files of Zabbix server and Elastisearch. There will be a big packs of errors like **
zabbix server log:
5610:20220816:192015.350 cannot get values from elasticsearch, HTTP status code: 400
5609:20220816:192015.365 cannot get values from elasticsearch, HTTP status code: 400
5614:20220816:192015.367 cannot get values from elasticsearch, HTTP status code: 400
5613:20220816:192015.368 cannot get values from elasticsearch, HTTP status code: 400{*}{*}
elasticsearch:
Aug 16 19:01:03 myhost elasticsearch[130]: Caused by: org.elasticsearch.ElasticsearchException: Trying to create too many scroll contexts. Must be less than or equal to: [40960]. This limit can be set by changing the [search.max_open_scroll_context] setting.
Aug 16 19:01:03 myhost elasticsearch[130]: at org.elasticsearch.search.SearchService.createAndPutReaderContext(SearchService.java:649) ~[elasticsearch-7.10.2.jar:7.10.2]
Aug 16 19:01:03 myhost elasticsearch[130]: at org.elasticsearch.search.SearchService.createOrGetReaderContext(SearchService.java:633) ~[elasticsearch-7.10.2.jar:7.10.2]
Aug 16 19:01:03 myhost elasticsearch[130]: at org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:426) ~[elasticsearch-7.10.2.jar:7.10.2]
Aug 16 19:01:03 myhost elasticsearch[130]: at org.elasticsearch.search.SearchService.access$500(SearchService.java:141) ~[elasticsearch-7.10.2.jar:7.10.2]
Aug 16 19:01:03 myhost elasticsearch[130]: at org.elasticsearch.search.SearchService$2.lambda$onResponse$0(SearchService.java:401) ~[elasticsearch-7.10.2.jar:7.10.2]
Aug 16 19:01:03 myhost elasticsearch[130]: at org.elasticsearch.action.ActionRunnable.lambda$supply$0(ActionRunnable.java:58) [elasticsearch-7.10.2.jar:7.10.2]
Aug 16 19:01:03 myhost elasticsearch[130]: at org.elasticsearch.action.ActionRunnable$2.doRun(ActionRunnable.java:73) ~[elasticsearch-7.10.2.jar:7.10.2]
Aug 16 19:01:03 myhost elasticsearch[130]: at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.10.2.jar:7.10.2]
Aug 16 19:01:03 myhost elasticsearch[130]: at org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:44) [elasticsearch-7.10.2.jar:7.10.2]
Aug 16 19:01:03 myhost elasticsearch[130]: at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:743) [elasticsearch-7.10.2.jar:7.10.2]
Aug 16 19:01:03 myhost elasticsearch[130]: at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.10.2.jar:7.10.2]
Aug 16 19:01:03 myhost elasticsearch[130]: at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) [?:?]
Aug 16 19:01:03 myhost elasticsearch[130]: at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) [?:?]
Aug 16 19:01:03 myhost elasticsearch[130]: at java.lang.Thread.run(Thread.java:832) [?:?]
Aug 16 19:01:03 myhost elasticsearch[130]: [o8dlAKgRRLeKPxLYECKi6g][uint][0]: Failed to execute [SearchRequest\{searchType=QUERY_THEN_FETCH, indices=[uint*], indicesOptions=IndicesOptions[ignore_unavailable=false, allow_no_indices=true, expand_wildcards_open=true, expand_wildcards_closed=false, expand_wildcards_hidden=false, allow_aliases_to_multiple_indices=true, forbid_closed_indices=true, ignore_aliases=false, ignore_throttled=true], types=[], routing='null', preference='null', requestCache=null, scroll=Scroll{keepAlive=10s}, maxConcurrentShardRequests=0, batchedReduceSize=512, preFilterShardSize=null, allowPartialSearchResults=true, localClusterAlias=null, getOrCreateAbsoluteStartMillis=-1, ccsMinimizeRoundtrips=true, source=\{"size":2,"query":{"bool":{"must":[{"match":{"itemid":{"query":580577,"operator":"OR","prefix_length":0,"max_expansions":50,"fuzzy_transpositions":true,"lenient":false,"zero_terms_query":"NONE","auto_generate_synonyms_phrase_query":true,"boost":1.0}}}],"filter":[\{"range":{"clock":{"from":1660665542,"to":1660665663,"include_lower":false,"include_upper":true,"boost":1.0}}}],"adjust_pure_negative":true,"boost":1.0}},"sort":[\{"clock":{"order":"desc"}}]}}] lastShard [false]
Aug 16 19:01:03 myhost elasticsearch[130]: org.elasticsearch.transport.RemoteTransportException: [myhost2][10.25.13.90:9300][indices:data/read/search[phase/query]]
**
I've tried to increase the value of this parameter search.max_open_scroll_context up to 256000, but it was not helpful.
I've increased RAM, but it wasn't helpful too.
The requests like this are very expensive and frequently used by Zabbix server. It generates a great amount of errors very quickly. It causes very large log files and huge traffic to log collectors excluding search erors as is.
Elasticsearch 7 documentation recommends use search_after instead of scroll api:
https://www.elastic.co/guide/en/elasticsearch/reference/7.17/scroll-api.html
Zabbix hasn't configuration parameters which can change this behavior or, may be I do something wrong...