[ZBX-21481] Zabbix 6.0 problems with Elasticsearch 7.x scroll api Created: 2022 Aug 16  Updated: 2025 Jan 07

Status: Confirmed
Project: ZABBIX BUGS AND ISSUES
Component/s: Frontend (F), Server (S)
Affects Version/s: 6.0.6
Fix Version/s: None

Type: Problem report Priority: Major
Reporter: Artem Hohlov Assignee: Zabbix Development Team
Resolution: Unresolved Votes: 0
Labels: Elasticsearch
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: File ZBX-21481-7.0-search.diff    
Issue Links:
Sub-task
part of ZBX-24549 Zabbix server 6.4.12 probably bug or ... Closed
part of ZBX-19511 Why zabbix uses the scroll api to acc... Need info

 Description   

Steps to reproduce:

  1. Setup Elasticsearch 7.x cluster and use it as a history storage(all data: HistoryStorageTypes=uint,dbl,str,log,text)
  2. Fill zabbix server with data. It works when hosts number is about 3000  
  3. Navigate to /zabbix.php?action=dashboard.view

Result:
See log files of Zabbix server and Elastisearch. There will be a big packs of errors like **

zabbix server log:


  5610:20220816:192015.350 cannot get values from elasticsearch, HTTP status code: 400
  5609:20220816:192015.365 cannot get values from elasticsearch, HTTP status code: 400
  5614:20220816:192015.367 cannot get values from elasticsearch, HTTP status code: 400
  5613:20220816:192015.368 cannot get values from elasticsearch, HTTP status code: 400{*}{*}


elasticsearch:


Aug 16 19:01:03 myhost elasticsearch[130]: Caused by: org.elasticsearch.ElasticsearchException: Trying to create too many scroll contexts. Must be less than or equal to: [40960]. This limit can be set by changing the [search.max_open_scroll_context] setting.
Aug 16 19:01:03 myhost elasticsearch[130]: at org.elasticsearch.search.SearchService.createAndPutReaderContext(SearchService.java:649) ~[elasticsearch-7.10.2.jar:7.10.2]
Aug 16 19:01:03 myhost elasticsearch[130]: at org.elasticsearch.search.SearchService.createOrGetReaderContext(SearchService.java:633) ~[elasticsearch-7.10.2.jar:7.10.2]
Aug 16 19:01:03 myhost elasticsearch[130]: at org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:426) ~[elasticsearch-7.10.2.jar:7.10.2]
Aug 16 19:01:03 myhost elasticsearch[130]: at org.elasticsearch.search.SearchService.access$500(SearchService.java:141) ~[elasticsearch-7.10.2.jar:7.10.2]
Aug 16 19:01:03 myhost elasticsearch[130]: at org.elasticsearch.search.SearchService$2.lambda$onResponse$0(SearchService.java:401) ~[elasticsearch-7.10.2.jar:7.10.2]
Aug 16 19:01:03 myhost elasticsearch[130]: at org.elasticsearch.action.ActionRunnable.lambda$supply$0(ActionRunnable.java:58) [elasticsearch-7.10.2.jar:7.10.2]
Aug 16 19:01:03 myhost elasticsearch[130]: at org.elasticsearch.action.ActionRunnable$2.doRun(ActionRunnable.java:73) ~[elasticsearch-7.10.2.jar:7.10.2]
Aug 16 19:01:03 myhost elasticsearch[130]: at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.10.2.jar:7.10.2]
Aug 16 19:01:03 myhost elasticsearch[130]: at org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:44) [elasticsearch-7.10.2.jar:7.10.2]
Aug 16 19:01:03 myhost elasticsearch[130]: at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:743) [elasticsearch-7.10.2.jar:7.10.2]
Aug 16 19:01:03 myhost elasticsearch[130]: at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.10.2.jar:7.10.2]
Aug 16 19:01:03 myhost elasticsearch[130]: at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) [?:?]
Aug 16 19:01:03 myhost elasticsearch[130]: at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) [?:?]
Aug 16 19:01:03 myhost elasticsearch[130]: at java.lang.Thread.run(Thread.java:832) [?:?]
Aug 16 19:01:03 myhost elasticsearch[130]: [o8dlAKgRRLeKPxLYECKi6g][uint][0]: Failed to execute [SearchRequest\{searchType=QUERY_THEN_FETCH, indices=[uint*], indicesOptions=IndicesOptions[ignore_unavailable=false, allow_no_indices=true, expand_wildcards_open=true, expand_wildcards_closed=false, expand_wildcards_hidden=false, allow_aliases_to_multiple_indices=true, forbid_closed_indices=true, ignore_aliases=false, ignore_throttled=true], types=[], routing='null', preference='null', requestCache=null, scroll=Scroll{keepAlive=10s}, maxConcurrentShardRequests=0, batchedReduceSize=512, preFilterShardSize=null, allowPartialSearchResults=true, localClusterAlias=null, getOrCreateAbsoluteStartMillis=-1, ccsMinimizeRoundtrips=true, source=\{"size":2,"query":{"bool":{"must":[{"match":{"itemid":{"query":580577,"operator":"OR","prefix_length":0,"max_expansions":50,"fuzzy_transpositions":true,"lenient":false,"zero_terms_query":"NONE","auto_generate_synonyms_phrase_query":true,"boost":1.0}}}],"filter":[\{"range":{"clock":{"from":1660665542,"to":1660665663,"include_lower":false,"include_upper":true,"boost":1.0}}}],"adjust_pure_negative":true,"boost":1.0}},"sort":[\{"clock":{"order":"desc"}}]}}] lastShard [false]
Aug 16 19:01:03 myhost elasticsearch[130]: org.elasticsearch.transport.RemoteTransportException: [myhost2][10.25.13.90:9300][indices:data/read/search[phase/query]]

**

I've tried to increase the value of this parameter search.max_open_scroll_context up to 256000, but it was not helpful.

I've increased RAM, but it wasn't helpful too.

 

The requests like this are very expensive and frequently used by Zabbix server. It generates a great amount of errors very quickly. It causes very large log files and huge traffic to log collectors excluding search erors as is.

 

Elasticsearch 7 documentation recommends use search_after  instead of scroll api:

https://www.elastic.co/guide/en/elasticsearch/reference/7.17/scroll-api.html

Zabbix hasn't configuration parameters which can change this behavior or, may be I do something wrong...



 Comments   
Comment by Edgar Akhmetshin [ 2022 Aug 17 ]

Hello Artem,

Thank you for reporting the issue. Confirmed.

Regards,
Edgar

Comment by Vladislavs Sokurenko [ 2024 Dec 18 ]

Please also see if ZBX-24549 is related

Comment by Vladislavs Sokurenko [ 2025 Jan 06 ]

Could try if avoiding scroll helps when retrieving limited datasets, please see ZBX-21481-7.0-search.diff

Generated at Mon Apr 28 07:22:09 EEST 2025 using Jira 9.12.4#9120004-sha1:625303b708afdb767e17cb2838290c41888e9ff0.