[ZBX-21481] Zabbix 6.0 problems with Elasticsearch 7.x scroll api Created: 2022 Aug 16 Updated: 2025 Jan 07 |
|
Status: | Confirmed |
Project: | ZABBIX BUGS AND ISSUES |
Component/s: | Frontend (F), Server (S) |
Affects Version/s: | 6.0.6 |
Fix Version/s: | None |
Type: | Problem report | Priority: | Major |
Reporter: | Artem Hohlov | Assignee: | Zabbix Development Team |
Resolution: | Unresolved | Votes: | 0 |
Labels: | Elasticsearch | ||
Remaining Estimate: | Not Specified | ||
Time Spent: | Not Specified | ||
Original Estimate: | Not Specified |
Attachments: |
![]() |
||||||||||||
Issue Links: |
|
Description |
Steps to reproduce:
Result: zabbix server log: 5610:20220816:192015.350 cannot get values from elasticsearch, HTTP status code: 400 5609:20220816:192015.365 cannot get values from elasticsearch, HTTP status code: 400 5614:20220816:192015.367 cannot get values from elasticsearch, HTTP status code: 400 5613:20220816:192015.368 cannot get values from elasticsearch, HTTP status code: 400{*}{*} elasticsearch: Aug 16 19:01:03 myhost elasticsearch[130]: Caused by: org.elasticsearch.ElasticsearchException: Trying to create too many scroll contexts. Must be less than or equal to: [40960]. This limit can be set by changing the [search.max_open_scroll_context] setting. Aug 16 19:01:03 myhost elasticsearch[130]: at org.elasticsearch.search.SearchService.createAndPutReaderContext(SearchService.java:649) ~[elasticsearch-7.10.2.jar:7.10.2] Aug 16 19:01:03 myhost elasticsearch[130]: at org.elasticsearch.search.SearchService.createOrGetReaderContext(SearchService.java:633) ~[elasticsearch-7.10.2.jar:7.10.2] Aug 16 19:01:03 myhost elasticsearch[130]: at org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:426) ~[elasticsearch-7.10.2.jar:7.10.2] Aug 16 19:01:03 myhost elasticsearch[130]: at org.elasticsearch.search.SearchService.access$500(SearchService.java:141) ~[elasticsearch-7.10.2.jar:7.10.2] Aug 16 19:01:03 myhost elasticsearch[130]: at org.elasticsearch.search.SearchService$2.lambda$onResponse$0(SearchService.java:401) ~[elasticsearch-7.10.2.jar:7.10.2] Aug 16 19:01:03 myhost elasticsearch[130]: at org.elasticsearch.action.ActionRunnable.lambda$supply$0(ActionRunnable.java:58) [elasticsearch-7.10.2.jar:7.10.2] Aug 16 19:01:03 myhost elasticsearch[130]: at org.elasticsearch.action.ActionRunnable$2.doRun(ActionRunnable.java:73) ~[elasticsearch-7.10.2.jar:7.10.2] Aug 16 19:01:03 myhost elasticsearch[130]: at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.10.2.jar:7.10.2] Aug 16 19:01:03 myhost elasticsearch[130]: at org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:44) [elasticsearch-7.10.2.jar:7.10.2] Aug 16 19:01:03 myhost elasticsearch[130]: at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:743) [elasticsearch-7.10.2.jar:7.10.2] Aug 16 19:01:03 myhost elasticsearch[130]: at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.10.2.jar:7.10.2] Aug 16 19:01:03 myhost elasticsearch[130]: at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) [?:?] Aug 16 19:01:03 myhost elasticsearch[130]: at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) [?:?] Aug 16 19:01:03 myhost elasticsearch[130]: at java.lang.Thread.run(Thread.java:832) [?:?] Aug 16 19:01:03 myhost elasticsearch[130]: [o8dlAKgRRLeKPxLYECKi6g][uint][0]: Failed to execute [SearchRequest\{searchType=QUERY_THEN_FETCH, indices=[uint*], indicesOptions=IndicesOptions[ignore_unavailable=false, allow_no_indices=true, expand_wildcards_open=true, expand_wildcards_closed=false, expand_wildcards_hidden=false, allow_aliases_to_multiple_indices=true, forbid_closed_indices=true, ignore_aliases=false, ignore_throttled=true], types=[], routing='null', preference='null', requestCache=null, scroll=Scroll{keepAlive=10s}, maxConcurrentShardRequests=0, batchedReduceSize=512, preFilterShardSize=null, allowPartialSearchResults=true, localClusterAlias=null, getOrCreateAbsoluteStartMillis=-1, ccsMinimizeRoundtrips=true, source=\{"size":2,"query":{"bool":{"must":[{"match":{"itemid":{"query":580577,"operator":"OR","prefix_length":0,"max_expansions":50,"fuzzy_transpositions":true,"lenient":false,"zero_terms_query":"NONE","auto_generate_synonyms_phrase_query":true,"boost":1.0}}}],"filter":[\{"range":{"clock":{"from":1660665542,"to":1660665663,"include_lower":false,"include_upper":true,"boost":1.0}}}],"adjust_pure_negative":true,"boost":1.0}},"sort":[\{"clock":{"order":"desc"}}]}}] lastShard [false] Aug 16 19:01:03 myhost elasticsearch[130]: org.elasticsearch.transport.RemoteTransportException: [myhost2][10.25.13.90:9300][indices:data/read/search[phase/query]] ** I've tried to increase the value of this parameter search.max_open_scroll_context up to 256000, but it was not helpful. I've increased RAM, but it wasn't helpful too.
The requests like this are very expensive and frequently used by Zabbix server. It generates a great amount of errors very quickly. It causes very large log files and huge traffic to log collectors excluding search erors as is.
Elasticsearch 7 documentation recommends use search_after instead of scroll api: https://www.elastic.co/guide/en/elasticsearch/reference/7.17/scroll-api.html Zabbix hasn't configuration parameters which can change this behavior or, may be I do something wrong... |
Comments |
Comment by Edgar Akhmetshin [ 2022 Aug 17 ] |
Hello Artem, Thank you for reporting the issue. Confirmed. Regards, |
Comment by Vladislavs Sokurenko [ 2024 Dec 18 ] |
Please also see if |
Comment by Vladislavs Sokurenko [ 2025 Jan 06 ] |
Could try if avoiding scroll helps when retrieving limited datasets, please see ZBX-21481-7.0-search.diff |