[ZBXNEXT-4831] Real-time export protocol missing types Created: 2018 Oct 09  Updated: 2024 Apr 10  Resolved: 2020 Jan 16

Status: Closed
Project: ZABBIX FEATURE REQUESTS
Component/s: Server (S)
Affects Version/s: 4.0.0
Fix Version/s: 4.4.0rc1, 4.4 (plan)

Type: New Feature Request Priority: Major
Reporter: Sebastian YEPES Assignee: Andrejs Kozlovs
Resolution: Fixed Votes: 12
Labels: export, real-time
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

Any

Issue Links:
Sub-task
part of ZBXNEXT-5433 Include host name into real-time expo... Closed
Team: Team C
Sprint: Sprint 56 (Sep 2019), Sprint 57 (Oct 2019), Sprint 58 (Nov 2019), Sprint 59 (Dec 2019), Sprint 60 (Jan 2020)
Story Points: 0.125

 Description   

Hello Zabbix Team,

I have started looking at building a pipeline to process and ingest the history, trends and problems NDJSON files into a more optimized time-series DB.
After building a small POC I encountered one major issue with the current history and trends real-time export protocol.

There exists no type field, so the content of the value field can be anything from a unsigned number to a string.
This is not really optimal as storing unsigned numbers, floats and strings is not the same, on the same note at query time not knowing the type beforehand is also not very optimal.

Would you guys please consider adding a field type to the current history and trends export that will indicate the type of the value.

Type Possible values:

  • 0 - numeric float;
  • 1 - character;
  • 2 - log;
  • 3 - numeric unsigned;
  • 4 - text.

Regards and thanks for the hard work on the 4.0 release,
Sebastian

 Comments   
Comment by Sebastian YEPES [ 2018 Oct 09 ]

The use case: well let’s say I would like to store / forward the history/trend values in some external DB like (InfluxDB, CrateDB, ScyllaDB...) when defining the schema for these tables its best if the value field has the precise type. If this is not done we will not be able to execute aggregate or number specific operations on these records.

Comment by Vladislavs Sokurenko [ 2018 Oct 09 ]

Could you use something like this to understand type ?

<?php
	$json = '{"host":"Host B","groups":["Group X","Group Y","Group Z"],"applications":["Zabbix Agent","Availability"],"itemid":3,"name":"Agent availability","clock":1519304285,"ns":123456789,"value":1}';
	var_dump(json_decode($json));
?>

output:

object(stdClass)#1 (8) {
  ["host"]=>
  string(6) "Host B"
  ["groups"]=>
  array(3) {
    [0]=>
    string(7) "Group X"
    [1]=>
    string(7) "Group Y"
    [2]=>
    string(7) "Group Z"
  }
  ["applications"]=>
  array(2) {
    [0]=>
    string(12) "Zabbix Agent"
    [1]=>
    string(12) "Availability"
  }
  ["itemid"]=>
  int(3)
  ["name"]=>
  string(18) "Agent availability"
  ["clock"]=>
  int(1519304285)
  ["ns"]=>
  int(123456789)
  ["value"]=>
  int(1)
}

Or:

<?php
	$json = '{"host":"Host B","groups":["Group X","Group Y","Group Z"],"applications":["Zabbix Agent","Availability"],"itemid":3,"name":"Agent availability","clock":1519304285,"ns":123456789,"value":"1"}';
	var_dump(json_decode($json));
?>

Output:

object(stdClass)#1 (8) {
  ["host"]=>
  string(6) "Host B"
  ["groups"]=>
  array(3) {
    [0]=>
    string(7) "Group X"
    [1]=>
    string(7) "Group Y"
    [2]=>
    string(7) "Group Z"
  }
  ["applications"]=>
  array(2) {
    [0]=>
    string(12) "Zabbix Agent"
    [1]=>
    string(12) "Availability"
  }
  ["itemid"]=>
  int(3)
  ["name"]=>
  string(18) "Agent availability"
  ["clock"]=>
  int(1519304285)
  ["ns"]=>
  int(123456789)
  ["value"]=>
  string(1) "1"
}
Comment by Sebastian YEPES [ 2018 Oct 09 ]

Yes, having numbers in the JSON actually represented as numbers without the double quotes would be a good start. but we would still have the issue of distinguishing between types floats and unsigned.

The final solution would be to have the type, in this way we could actually optimize the storage in the best format for each type.

Is this complicated or have some additional backed requirements?

Comment by Sebastian YEPES [ 2018 Oct 17 ]

vso, Do you have any final status or decision if this kind of change to the json could be made?

Comment by Edgars Melveris [ 2018 Oct 22 ]

Hello Sebastian!

Was this intended for the feature request?

Comment by Sebastian YEPES [ 2018 Oct 28 ]

Edgars Melveris, will yes it would be great if it could be added to the 4.0.x branch. It would really make this functionality useful.

Con you convert nit to a feature request or should I create a new issue?

Comment by Edgars Melveris [ 2018 Oct 29 ]

No problem, mowed to feature request.

Comment by Sebastian YEPES [ 2018 Nov 21 ]

vso | zux, Is there any news on the implementation of this feature / fix?

This would be really essential to be able to fully take advantage of this functionality.

Comment by Vladislavs Sokurenko [ 2018 Nov 28 ]

You mean something like this ?

Index: src/libs/zbxdbcache/dbcache.c
===================================================================
--- src/libs/zbxdbcache/dbcache.c	(revision 87412)
+++ src/libs/zbxdbcache/dbcache.c	(working copy)
@@ -1143,6 +1143,7 @@
 				THIS_SHOULD_NEVER_HAPPEN;
 		}
 
+		zbx_json_adduint64(&json, ZBX_PROTO_TAG_TYPE, trend->value_type);
 		zbx_trends_export_write(json.buffer, json.buffer_size);
 	}
 
@@ -1245,6 +1246,7 @@
 				THIS_SHOULD_NEVER_HAPPEN;
 		}
 
+		zbx_json_adduint64(&json, ZBX_PROTO_TAG_TYPE, h->value_type);
 		zbx_history_export_write(json.buffer, json.buffer_size);
 	}
Comment by Sebastian YEPES [ 2018 Nov 28 ]

 Hello vso,

First of all many thanks for spending some time on this issue.

I have done some testing and this is exactly what was needed, now anyone that wants to use the records can know beforehand what types they contain:.

  history-history-syncer-1.ndjson

{"host":"zbx-agt","groups":["Linux servers"],"applications":["Zabbix agent"],"itemid":132,"name":"Agent ping","clock":1543422372,"ns":487668918,"value":1,"type":3}
{"host":"zbx-agt","groups":["Linux servers"],"applications":["Zabbix agent"],"itemid":131,"name":"Host name of zabbix_agentd running","clock":1543422791,"ns":595095166,"value":"zbx-agt","type":1}
{"host":"Zabbix server","groups":["Zabbix servers"],"applications":["Zabbix server"],"itemid":130,"name":"Number of processed values per second","clock":1543422850,"ns":608909149,"value":0.616517,"type":0}

trends-history-syncer-1.ndjson

{"host":"Zabbix server","groups":["Zabbix servers"],"applications":["Zabbix server"],"itemid":118,"name":"Zabbix value cache misses","clock":1543420800,"count":35,"min":0.000000,"avg":0.002380,"max":0.049981,"type":0}
{"host":"Zabbix server","groups":["Zabbix servers"],"applications":["Zabbix server"],"itemid":112,"name":"Zabbix queue over 10 minutes","clock":1543420800,"count":3,"min":0,"avg":0,"max":0,"type":3}

 

Now the hard part, do you know if and when this could be integrated into the 4.0.X or 4.x branch?

 

Regards,
Seb

Comment by Sebastian YEPES [ 2019 Sep 17 ]

Have we considered integrating this minor modification into the export protocol?

Comment by Sebastian YEPES [ 2019 Sep 25 ]

Hello @Andrejs Kozlovs,

I see your working on this issue, to you have any insights if this is going to get implemented?
If yes, do you know if it's still going to be using the same tag (ZBX_PROTO_TAG_TYPE)?

vso yes syepes it will be the same patch and same tag but we are also looking into some other changes

Comment by Andrejs Kozlovs [ 2019 Oct 01 ]

Fixed in:

  • 4.4.0rc1 (master) 64d5130f51e
Comment by Sebastian YEPES [ 2019 Oct 04 ]

Hello Andrejs Kozlovs,

If you are going to be in the this years zabbix summit 2019 please let me buy you a beer! I have been waiting for this one a while and its going to be a game changer!

Thanks again!

ak Thank You, syepes! This is team work. Welcome in Riga.

Comment by Alexander Vladishev [ 2020 Jan 16 ]

Updated documentation:

Generated at Tue Apr 23 15:40:21 EEST 2024 using Jira 9.12.4#9120004-sha1:625303b708afdb767e17cb2838290c41888e9ff0.