-
Type:
Problem report
-
Resolution: Unresolved
-
Priority:
Trivial
-
Affects Version/s: 8.0.0rc2
-
Component/s: Server (S)
-
Environment:Ubuntu 24.04, zabbix-server-pgsql 2:8.0.0~rc2-1+ubuntu24.04, Elasticsearch 9.5.5 and 8.19.23
Steps to reproduce:
1. Create the json ingest pipeline and the json index using the "json pipeline" and "json mapping" sections of database/elasticsearch/elasticsearch.map. Add "default_pipeline": "json" to the index settings: without it, the pipeline is never applied, because Zabbix server does not specify a pipeline when date-based indices are not used.
2. Configure Zabbix server:
HistoryProvider=elasticsearch;value_types="json",url=http://localhost:9200
3. Create a trapper item of the JSON type and send the following values with zabbix_sender:
{"a":1}
[{"a":1},{"b":2}]
[1,2,3]
"plain string"
42
true
[0,1,2,...,7999] (array of 46,890 characters)
{"blob":"xxx...x"} (object with a 40,000-character string)
Result:
Only the first two values (the object and the array of objects) are stored. The other six values are rejected by Elasticsearch and lost, and each rejection stalls the history syncer for 10 seconds:
cannot send data to Elasticsearch: index:json status:400 type:document_parsing_exception reason:[1:33] failed to parse field [value_parsed] of type [flattened] in document with id '...'. Preview of field's value: '1' Elasticsearch database is down: reconnecting in 10 seconds Elasticsearch database connection re-established
- Scalars and arrays of scalars ([1,2,3], "plain string", 42, true, the long array): the json processor stores them in value_parsed, which a flattened field cannot hold. The set processor handles only value_parsed == null.
- Long values: Elasticsearch cannot index terms longer than 32766 bytes. value_parsed has no ignore_above limit, and "ignore_above": 1000000 on value_raw is above that limit, so long non-object values are rejected as well (seen with the date-based json-pipeline from the documentation, which stores them in value_raw).
Expected:
All valid JSON values are stored: objects in value_parsed, other values (arrays, strings, numbers, booleans, null) as strings in value_raw. Long values are stored in _source and returned to Zabbix, but are not indexed.
Suggested fix (tested with Zabbix 8.0.0rc2 on Elasticsearch 8.19.23 and 9.5.5, with single-index and date-based setups: all values above are stored and returned by history.get):
- In the "json mapping" section, set "ignore_above": 8191 for both value_parsed and value_raw (32766 bytes / 4 bytes per UTF-8 character).
- Replace the "json pipeline" section with the same logic as json-pipeline in the documentation:
{ "processors": [ { "json": { "field": "value", "target_field": "value_parsed", "ignore_failure": true } }, { "script": { "source": "if (ctx.value_parsed == null || !(ctx.value_parsed instanceof Map)) { ctx.value_raw = ctx.value; ctx.remove(\"value_parsed\"); }" } } ] }