Uploaded image for project: 'ZABBIX BUGS AND ISSUES'
  1. ZABBIX BUGS AND ISSUES
  2. ZBX-21481

Zabbix 6.0 problems with Elasticsearch 7.x scroll api

    XMLWordPrintable

Details

    • Problem report
    • Status: Confirmed
    • Major
    • Resolution: Unresolved
    • 6.0.6
    • None
    • Frontend (F), Server (S)

    Description

      Steps to reproduce:

      1. Setup Elasticsearch 7.x cluster and use it as a history storage(all data: HistoryStorageTypes=uint,dbl,str,log,text)
      2. Fill zabbix server with data. It works when hosts number is about 3000  
      3. Navigate to /zabbix.php?action=dashboard.view

      Result:
      See log files of Zabbix server and Elastisearch. There will be a big packs of errors like **

      zabbix server log:
      
      
        5610:20220816:192015.350 cannot get values from elasticsearch, HTTP status code: 400
        5609:20220816:192015.365 cannot get values from elasticsearch, HTTP status code: 400
        5614:20220816:192015.367 cannot get values from elasticsearch, HTTP status code: 400
        5613:20220816:192015.368 cannot get values from elasticsearch, HTTP status code: 400{*}{*}
      
      
      elasticsearch:
      
      
      Aug 16 19:01:03 myhost elasticsearch[130]: Caused by: org.elasticsearch.ElasticsearchException: Trying to create too many scroll contexts. Must be less than or equal to: [40960]. This limit can be set by changing the [search.max_open_scroll_context] setting.
      Aug 16 19:01:03 myhost elasticsearch[130]: at org.elasticsearch.search.SearchService.createAndPutReaderContext(SearchService.java:649) ~[elasticsearch-7.10.2.jar:7.10.2]
      Aug 16 19:01:03 myhost elasticsearch[130]: at org.elasticsearch.search.SearchService.createOrGetReaderContext(SearchService.java:633) ~[elasticsearch-7.10.2.jar:7.10.2]
      Aug 16 19:01:03 myhost elasticsearch[130]: at org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:426) ~[elasticsearch-7.10.2.jar:7.10.2]
      Aug 16 19:01:03 myhost elasticsearch[130]: at org.elasticsearch.search.SearchService.access$500(SearchService.java:141) ~[elasticsearch-7.10.2.jar:7.10.2]
      Aug 16 19:01:03 myhost elasticsearch[130]: at org.elasticsearch.search.SearchService$2.lambda$onResponse$0(SearchService.java:401) ~[elasticsearch-7.10.2.jar:7.10.2]
      Aug 16 19:01:03 myhost elasticsearch[130]: at org.elasticsearch.action.ActionRunnable.lambda$supply$0(ActionRunnable.java:58) [elasticsearch-7.10.2.jar:7.10.2]
      Aug 16 19:01:03 myhost elasticsearch[130]: at org.elasticsearch.action.ActionRunnable$2.doRun(ActionRunnable.java:73) ~[elasticsearch-7.10.2.jar:7.10.2]
      Aug 16 19:01:03 myhost elasticsearch[130]: at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.10.2.jar:7.10.2]
      Aug 16 19:01:03 myhost elasticsearch[130]: at org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:44) [elasticsearch-7.10.2.jar:7.10.2]
      Aug 16 19:01:03 myhost elasticsearch[130]: at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:743) [elasticsearch-7.10.2.jar:7.10.2]
      Aug 16 19:01:03 myhost elasticsearch[130]: at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.10.2.jar:7.10.2]
      Aug 16 19:01:03 myhost elasticsearch[130]: at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) [?:?]
      Aug 16 19:01:03 myhost elasticsearch[130]: at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) [?:?]
      Aug 16 19:01:03 myhost elasticsearch[130]: at java.lang.Thread.run(Thread.java:832) [?:?]
      Aug 16 19:01:03 myhost elasticsearch[130]: [o8dlAKgRRLeKPxLYECKi6g][uint][0]: Failed to execute [SearchRequest\{searchType=QUERY_THEN_FETCH, indices=[uint*], indicesOptions=IndicesOptions[ignore_unavailable=false, allow_no_indices=true, expand_wildcards_open=true, expand_wildcards_closed=false, expand_wildcards_hidden=false, allow_aliases_to_multiple_indices=true, forbid_closed_indices=true, ignore_aliases=false, ignore_throttled=true], types=[], routing='null', preference='null', requestCache=null, scroll=Scroll{keepAlive=10s}, maxConcurrentShardRequests=0, batchedReduceSize=512, preFilterShardSize=null, allowPartialSearchResults=true, localClusterAlias=null, getOrCreateAbsoluteStartMillis=-1, ccsMinimizeRoundtrips=true, source=\{"size":2,"query":{"bool":{"must":[{"match":{"itemid":{"query":580577,"operator":"OR","prefix_length":0,"max_expansions":50,"fuzzy_transpositions":true,"lenient":false,"zero_terms_query":"NONE","auto_generate_synonyms_phrase_query":true,"boost":1.0}}}],"filter":[\{"range":{"clock":{"from":1660665542,"to":1660665663,"include_lower":false,"include_upper":true,"boost":1.0}}}],"adjust_pure_negative":true,"boost":1.0}},"sort":[\{"clock":{"order":"desc"}}]}}] lastShard [false]
      Aug 16 19:01:03 myhost elasticsearch[130]: org.elasticsearch.transport.RemoteTransportException: [myhost2][10.25.13.90:9300][indices:data/read/search[phase/query]]
      
      

      **

      I've tried to increase the value of this parameter search.max_open_scroll_context up to 256000, but it was not helpful.

      I've increased RAM, but it wasn't helpful too.

       

      The requests like this are very expensive and frequently used by Zabbix server. It generates a great amount of errors very quickly. It causes very large log files and huge traffic to log collectors excluding search erors as is.

       

      Elasticsearch 7 documentation recommends use search_after  instead of scroll api:

      https://www.elastic.co/guide/en/elasticsearch/reference/7.17/scroll-api.html

      Zabbix hasn't configuration parameters which can change this behavior or, may be I do something wrong...

      Attachments

        Activity

          People

            zabbix.dev Zabbix Development Team
            artem.khokhlov Artem Hohlov
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated: