[ZBXNEXT-1846] do not duplicate identical history values Created: 2013 Jul 31  Updated: 2019 May 03  Resolved: 2019 May 03

Status: Closed
Project: ZABBIX FEATURE REQUESTS
Component/s: Server (S)
Affects Version/s: 2.0.6
Fix Version/s: None

Type: New Feature Request Priority: Minor
Reporter: Stefan Assignee: Unassigned
Resolution: Done Votes: 18
Labels: duplicates, history
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

Debian Wheezy, Postgresql 9.1


Attachments: PDF File Advanced preprocessing 1.0rc1.pdf    
Issue Links:
Causes
causes ZBXNEXT-4724 Advanced preprocessing: validation an... Closed
Duplicate
is duplicated by ZBXNEXT-2967 items only save in case of a changed ... Closed
is duplicated by ZBXNEXT-3555 Optimise log file monitoring for non-... Closed
is duplicated by ZBXNEXT-2436 New data storage method Closed
is duplicated by ZBXNEXT-2865 Deduplication of identical data in hi... Closed

 Description   

at the moment zabbix write after every check the values in the database, but if the last value 1 and the new values is 1, too i have duplicate values that means it increase the database size.
i think a better solution is zabbix check the last value with the new value if it the same dont store the new value only refresh the date of "last check"

i hope you know what i mean and sorry for my bad english



 Comments   
Comment by richlv [ 2013 Aug 05 ]

i believe this wouldn't really work for how zabbix is designed - we would need timestamps of all those values anyway to know at what time they were collected...

a very similar feature request asks for skipping sending of identical values for active agents : ZBXNEXT-113

Comment by Stefan [ 2016 Apr 20 ]

i think here is a good solution for this, to reduce data: http://blog.endpoint.com/2014/04/filling-gaps-in-cumulative-sum-in.html

Comment by Timofey [ 2016 May 13 ]

Hi, i have implement something like by sripts, and i think it can be easy implemented in zabbix itself.
In my realization:
For ping, we have 2 items:
1. ping_trap sender
2. ping_trap reciver

ping_trap sender - run every 30-60s and ping remote host, save this value somewhere (i use files in tmpfs)
if new value is different to latest, then send it, to ping_trap reciver
As addition, i think about add periodic heartbeats, i.e. send value, without check, to ping reciver if it not send any data in 20m & etc

It's can be easy implemented as type of check and can reduce storage needs/DB performance
I.E. on WebUI add new type of checks: ~zabbix agent (active trapper)
With 2 time keys:
Frequency (30s)
Max time between forced send (20m) (barrier)
On zabbix agent side:
1. Load items from ServerActive
2. For items with that type:
2.1 Check Last value not NULL
2.2 save in memory with timestamp
2.3 send it to server

And in loop renew value, if it changed -> send it to server, if it get time barrier -> send to server

On WebUI side for Nice graps, logic for trapped graphs can be used.

For nice trapped graphs (i.e. with ping example, host will be fall in graph up to time barrier value)
Just send 2 values if it changed: old with timestamp = (timestamp - Frequency/2) + new value.

This logic can be also used for simple checks on server side

P.S. i just think what it can fix this issue for some time without big code changes in zabbix
And i don't want maintain own version of code for this feature
P.S.S.
By using this method i can increase frequency of all checks and decrease size of history tables(indexes) by 2-10x

Comment by Marc [ 2016 May 13 ]

I think this could also easily achieved by ZBXNEXT-1836 resp. ZBXNEXT-714 (e.g. by using a module for a proper time series database).

Comment by Oleksii Zagorskyi [ 2016 Nov 14 ]

ZBXNEXT-3544 is similar

Comment by Glebs Ivanovskis (Inactive) [ 2017 Aug 03 ]

ZBXNEXT-4016 hints about implementing this as a new preprocessing option.

Comment by Alexei Vladishev [ 2018 Jul 13 ]

It could also be implemented as a preprocessing option that would compare new value with the last one. If they are equal then the new value will be ignored.

Comment by Alexei Vladishev [ 2018 Aug 20 ]

Hey all! Please have a look at the attached acceptance criteria document that describes proposed enhancements for preprocessing that will cover this functionality as well as ZBXNEXT-4016 and ZBXNEXT-3544. It has all chances to be implemented in Zabbix 4.2. Let me know what you think. Is there anything to improve? I appreciate any feedback.

Comment by Daniel Suen [ 2018 Sep 04 ]

I recall that sometimes, when putting in some trigger conditions, there is no support in negative matching, i.e. !~, and I have to set it up in the global regex section and use it in the trigger. The thing is, if I am not mistaken, those global stuff are not easily exportable and kept consistent across multiple zabbix instances. Is it possible to add this feature?

 

Daniel.

Comment by Nicolas Bataille [ 2019 May 03 ]

I read the document, and it's really a good way to reduce the database size.

Comment by Alexei Vladishev [ 2019 May 03 ]

Support of throttling has been implemented under https://support.zabbix.com/browse/ZBXNEXT-4724, I am closing this ZBXNEXT.

Generated at Sat Apr 27 07:53:08 EEST 2024 using Jira 9.12.4#9120004-sha1:625303b708afdb767e17cb2838290c41888e9ff0.