[ZBX-18333] Error "elasticsearch version is not compatible with zabbix server" Created: 2020 Sep 02  Updated: 2024 Apr 10  Resolved: 2023 Apr 19

Status: Closed
Project: ZABBIX BUGS AND ISSUES
Component/s: Server (S)
Affects Version/s: 5.0.3, 5.0.5
Fix Version/s: None

Type: Incident report Priority: Minor
Reporter: Misha Assignee: Dmitrijs Goloscapovs (Inactive)
Resolution: Cannot Reproduce Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

Ubuntu 18.04.4
zabbix-server-pgsql 5.0.3
elasticsearch 7.8.0
zabbix-server-mysql 5.0.5

Team: Team A
Sprint: Sprint 96 (Jan 2023), Sprint 97 (Feb 2023), Sprint 98 (Mar 2023), Sprint 99 (Apr 2023)

 Description   

I got an error when trying to save historical Zabbix data in Elasticsearch: "elasticsearch version is not compatible with zabbix server". 

Which version of elasticsearch are supported?

  • /etc/zabbix/zabbix_server.conf: 
    #Elasticsearch
HistoryStorageURL=http://elastic_fqdn:9200
HistoryStorageTypes=uint,dbl,str,log,text
HistoryStorageDateIndex=1
  • Elasticsearch:
    • dbl_template: 
      {
  "version": 1,
  "order": 0,
  "index_patterns": [
    "dbl*"
  ],
  "settings": {
    "index": {
      "lifecycle": {
        "name": "zabbix_ilm"
      },
      "number_of_shards": "3",
      "number_of_replicas": "1"
    }
  },
  "aliases": {},
  "mappings": {
    "_doc": {
      "_routing": {
        "required": false
      },
      "_meta": {},
      "_source": {
        "excludes": [],
        "includes": [],
        "enabled": true
      },
      "dynamic": false,
      "dynamic_templates": [],
      "properties": {
        "itemid": {
          "type": "long"
        },
        "clock": {
          "format": "epoch_second",
          "type": "date"
        },
        "value": {
          "coerce": true,
          "index": true,
          "ignore_malformed": false,
          "store": false,
          "type": "double",
          "doc_values": true
        }
      }
    }
  }
}
    • log_template 
      {
  "version": 1,
  "order": 0,
  "index_patterns": [
    "log*"
  ],
  "settings": {
    "index": {
      "lifecycle": {
        "name": "zabbix_ilm"
      },
      "number_of_shards": "3",
      "number_of_replicas": "1"
    }
  },
  "aliases": {},
  "mappings": {
    "_doc": {
      "_routing": {
        "required": false
      },
      "_meta": {},
      "_source": {
        "excludes": [],
        "includes": [],
        "enabled": true
      },
      "dynamic": false,
      "dynamic_templates": [],
      "properties": {
        "itemid": {
          "type": "long"
        },
        "clock": {
          "format": "epoch_second",
          "type": "date"
        }
      }
    }
  }
}
    • str_template 
      {
  "version": 1,
  "order": 0,
  "index_patterns": [
    "str*"
  ],
  "settings": {
    "index": {
      "lifecycle": {
        "name": "zabbix_ilm"
      },
      "number_of_shards": "3",
      "number_of_replicas": "1"
    }
  },
  "aliases": {},
  "mappings": {
    "_doc": {
      "_routing": {
        "required": false
      },
      "_meta": {},
      "_source": {
        "excludes": [],
        "includes": [],
        "enabled": true
      },
      "dynamic": false,
      "dynamic_templates": [],
      "properties": {
        "itemid": {
          "type": "long"
        },
        "clock": {
          "format": "epoch_second",
          "type": "date"
        },
        "value": {
          "index": false,
          "type": "text",
          "fields": {
            "analyzed": {
              "analyzer": "standard",
              "index": true,
              "type": "text"
            }
          }
        }
      }
    }
  }
}
    • text_template 
      {
  "version": 1,
  "order": 0,
  "index_patterns": [
    "text*"
  ],
  "settings": {
    "index": {
      "lifecycle": {
        "name": "zabbix_ilm"
      },
      "number_of_shards": "3",
      "number_of_replicas": "1"
    }
  },
  "aliases": {},
  "mappings": {
    "_doc": {
      "_routing": {
        "required": false
      },
      "_meta": {},
      "_source": {
        "excludes": [],
        "includes": [],
        "enabled": true
      },
      "dynamic": false,
      "dynamic_templates": [],
      "properties": {
        "itemid": {
          "type": "long"
        },
        "clock": {
          "format": "epoch_second",
          "type": "date"
        },
        "value": {
          "index": false,
          "type": "text",
          "fields": {
            "analyzed": {
              "analyzer": "standard",
              "index": true,
              "type": "text"
            }
          }
        }
      }
    }
  }
}
    • uint_template 
      {
  "order": 0,
  "index_patterns": [
    "uint*"
  ],
  "settings": {
    "index": {
      "lifecycle": {
        "name": "zabbix_ilm"
      },
      "number_of_shards": "3",
      "number_of_replicas": "1"
    }
  },
  "aliases": {},
  "mappings": {
    "_doc": {
      "_source": {
        "excludes": [],
        "includes": [],
        "enabled": true
      },
      "_meta": {},
      "_routing": {
        "required": false
      },
      "dynamic": false,
      "dynamic_templates": [],
      "properties": {
        "clock": {
          "format": "epoch_second",
          "type": "date"
        },
        "itemid": {
          "type": "long"
        },
        "value": {
          "type": "long",
          "index": true,
          "ignore_malformed": false,
          "coerce": true,
          "doc_values": true,
          "store": false
        }
      }
    }
  }
}


 Comments   
Comment by Aigars Kadikis [ 2020 Sep 21 ]

Hello Misha,

Thank you for reaching out. Sorry, by I cannot face the error with Zabbix 5.0.3 and ElasticSearch 7.8.0.

I did use the suggested settings at our official documentation page:

https://www.zabbix.com/documentation/current/manual/appendix/install/elastic_search_setup

Please attach a full page screenshot and zabbix_server.log to show where the message is comming from.

Comment by Misha [ 2020 Sep 21 ]

Hi.

This error appears only if the configuration file contains the HistoryStorageDateIndex = 1 option. Also, with this configuration, zabbix writes all data to the index
uint-YYYY-MM-DD (e.g. uint-2020-09-21). 

Screenshot:

 https://drive.google.com/file/d/1vzwI4H_kRWZzSKc_ejvQbDh8tuy2AycJ/view?usp=sharing

 

If in the configuration file HistoryStorageDateIndex = 0, then there are no errors with the text "elasticsearch version is not compatible with zabbix server". The data is written to the correct indices, but zabbix writes data to elastic without specifying the pipeline name, because of this, all data is written to the indices without specifying the date (i.e., the indices are named: uint, dbl, str, text). There are pipelines, though. For example:

[
  {
    "date_index_name": {
      "field": "clock",
      "date_formats": [
        "UNIX"
      ],
      "index_name_prefix": "unit-",
      "date_rounding": "d"
    }
  }
]
Comment by Aigars Kadikis [ 2020 Nov 26 ]

Yes, there is a problem with 5.0.5 backend while using HistoryStorageDateIndex=1.

To reproduce with containers:

docker run --name mysql8 \
-e MYSQL_ROOT_PASSWORD="zabbix" \
-e TZ="Europe/Riga" \
-p 3306:3306 -d mysql:8.0.21

docker exec -it mysql8 mysql -uroot -pzabbix -e "
CREATE USER 'zabbix'@'%' IDENTIFIED BY 'zabbix';
ALTER USER 'zabbix'@'%' IDENTIFIED WITH mysql_native_password BY 'zabbix';
CREATE DATABASE zabbix CHARACTER SET utf8 COLLATE utf8_bin;
GRANT ALL PRIVILEGES ON zabbix.* TO 'zabbix'@'%';
FLUSH PRIVILEGES
"

docker run -d --name elasticsearch780 \
-p 9200:9200 -p 9300:9300 \
-e "discovery.type=single-node" elasticsearch:7.8.0

docker run --name z50srv \
-e DB_SERVER_HOST="10.133.112.87" \
-e DB_SERVER_PORT="3306" \
-e MYSQL_DATABASE="zabbix" \
-e MYSQL_USER="zabbix" \
-e MYSQL_PASSWORD="zabbix" \
-e ZBX_HISTORYSTORAGEURL="10.133.112.87:9200" \
-e ZBX_HISTORYSTORAGETYPES="uint,dbl,str,log,text" \
-e ZBX_HISTORYSTORAGEDATEINDEX=1 \
-d zabbix/zabbix-server-mysql:alpine-5.0.5
 

By looking into server logs it says errors per 'uint-pipeline' and 'dbl-pipeline':

   218:20201126:212853.656 elastic_writer_flush() cannot send data to elasticsearch: index:uint status:400 type:illegal_argument_exception reason:pipeline with id [uint-pipeline] does not exist
   218:20201126:212904.583 elastic_writer_flush() cannot send data to elasticsearch: index:dbl status:400 type:illegal_argument_exception reason:pipeline with id [dbl-pipeline] does not exist
   218:20201126:212904.583 elastic_writer_flush() cannot send data to elasticsearch: index:uint status:400 type:illegal_argument_exception reason:pipeline with id [uint-pipeline] does not exist
   218:20201126:212923.084 elastic_writer_flush() cannot send data to elasticsearch: index:uint status:400 type:illegal_argument_exception reason:pipeline with id [uint-pipeline] does not exist
   218:20201126:212933.136 elastic_writer_flush() cannot send data to elasticsearch: index:dbl status:400 type:illegal_argument_exception reason:pipeline with id [dbl-pipeline] does not exist
   218:20201126:212943.185 elastic_writer_flush() cannot send data to elasticsearch: index:dbl status:400 type:illegal_argument_exception reason:pipeline with id [dbl-pipeline] does not exist
   218:20201126:212943.185 elastic_writer_flush() cannot send data to elasticsearch: index:uint status:400 type:illegal_argument_exception reason:pipeline with id [uint-pipeline] does not exist
   218:20201126:212953.459 elastic_writer_flush() cannot send data to elasticsearch: index:uint status:400 type:illegal_argument_exception reason:pipeline with id [uint-pipeline] does not exist
   218:20201126:212953.459 elastic_writer_flush() cannot send data to elasticsearch: index:dbl status:400 type:illegal_argument_exception reason:pipeline with id [dbl-pipeline] does not exist
   218:20201126:213003.547 elastic_writer_flush() cannot send data to elasticsearch: index:dbl status:400 type:illegal_argument_exception reason:pipeline with id [dbl-pipeline] does not exist
   218:20201126:213022.604 elastic_writer_flush() cannot send data to elasticsearch: index:uint status:400 type:illegal_argument_exception reason:pipeline with id [uint-pipeline] does not exist
   218:20201126:213032.624 elastic_writer_flush() cannot send data to elasticsearch: index:dbl status:400 type:illegal_argument_exception reason:pipeline with id [dbl-pipeline] does not exist
   218:20201126:213042.718 elastic_writer_flush() cannot send data to elasticsearch: index:dbl status:400 type:illegal_argument_exception reason:pipeline with id [dbl-pipeline] does not exist
   218:20201126:213042.722 elastic_writer_flush() cannot send data to elasticsearch: index:uint status:400 type:illegal_argument_exception reason:pipeline with id [uint-pipeline] does not exist
   218:20201126:213052.996 elastic_writer_flush() cannot send data to elasticsearch: index:uint status:400 type:illegal_argument_exception reason:pipeline with id [uint-pipeline] does not exist
   218:20201126:213052.999 elastic_writer_flush() cannot send data to elasticsearch: index:dbl status:400 type:illegal_argument_exception reason:pipeline with id [dbl-pipeline] does not exist
Comment by Dmitrijs Goloscapovs (Inactive) [ 2023 Jan 17 ]

Hello, msolonitsin

Could you please be so kind and share:

  • Zabbix Server log entries that are related to ES (increase DebugLevel to 5, grep 'elastic') without sensitive information
  • if it is possible - pipeline definitions

Zabbix can throw 'elasticsearch is not compatible..' due to bad unexpected responses in some cases.

Comment by Dmitrijs Goloscapovs (Inactive) [ 2023 Jan 17 ]

Failed to reproduce on release/5.0 and ES 7.8.0 with configuration (including pipeline definitions) as per our manual.

Generated at Wed Nov 05 21:07:17 EET 2025 using Jira 9.12.4#9120004-sha1:625303b708afdb767e17cb2838290c41888e9ff0.