[ZBXNEXT-2463] use trends for trigger and calculated item functions Created: 2014 Sep 16  Updated: 2021 Mar 30  Resolved: 2021 Mar 30

Status: Closed
Project: ZABBIX FEATURE REQUESTS
Component/s: Server (S)
Affects Version/s: 2.0.12
Fix Version/s: None

Type: Change Request Priority: Minor
Reporter: Jiann-Ming Su Assignee: Unassigned
Resolution: Duplicate Votes: 11
Labels: calculateditems, history, trends, triggers
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

RH el6


Issue Links:
Duplicate
is duplicated by ZBXNEXT-6140 New set of trigger functions for long... Closed

 Description   

I am trying to generate some calculated items from an external check item. For at least the "avg" and "max" functions, I know for certain that using a time offset returns obviously wrong data as compared to what is shown on a graph over the same time period.

The external check item from which the others are being calculated, is polled every 5 minutes. Detailed data is stored for 1 day, and history is stored for 365 days.

The calculated items are looking for min, max, and avg over (7d,1d) and (1d,1h). The returned items with the 1d offset are obviously wrong. When collecting without the offset, the "max" seems to return only the last 1d max, and not the specified period of 7d. The (1d,1h) seems to return correct data.

Please let me know if I need to supply any more info.



 Comments   
Comment by Aleksandrs Saveljevs [ 2014 Sep 16 ]

As far as I understand, you store history for the external item for 1 day and trends for 365 days.

Trigger and calculated item functions only work with history data and do not look into trends. So the maximum amount of data that a function can process in your case is 1 day. Thus, a function like "max(7d)" is not expected to process data for the last 7 days - only 1 day which is stored in history.

Comment by Jiann-Ming Su [ 2014 Sep 16 ]

Thanks for the clarification. Should I submit a feature request? My understanding is that the trend data is the digested daily min, max, & avg. So a function like "max(7d)" should still be valid for items with only 1d of detailed history.

BTW, I'm using the min, max, and avg functions similarly in some triggers, and they seem to work correctly. I can't be for certain since I can't verify what values are being used. But, the triggers functions appear to evaluate correctly.

Comment by richlv [ 2014 Sep 16 ]

such a subset of functionality would be valid, but it is not implemented currently (and might be somewhat confusing for users)

Comment by Jiann-Ming Su [ 2014 Sep 16 ]

Honestly, the reason I'm even "abusing" the min, max, and avg functions is to calculate statistical variance. Maybe the real feature request is for Zabbix to support some basic statistical functions?

Comment by Aleksandrs Saveljevs [ 2014 Sep 17 ]

Adding the ability to raise a number to a power is covered by ZBXNEXT-552. Perhaps that could help.

Although that still would not work if you rely on trend data. If you need functions to use trend data, we could move this ticket to ZBXNEXT project.

Comment by Jiann-Ming Su [ 2014 Sep 17 ]

Moving this ticket to ZBXNEXT would be great. I would have thought statistical analysis would be relatively easy to implement as most databases already have those functions built in. Trend data would be useful when using sample data rather than the full population.

Comment by Aleksandrs Saveljevs [ 2014 Sep 18 ]

Moved the ticket to ZBXNEXT project, updated issue description.

Comment by Glebs Ivanovskis (Inactive) [ 2015 Oct 16 ]

Some users may have years and years of trends but all they can do is looking at at them on graphs and being proud of themselves. In many situations there are weekly, monthly, yearly patterns in data and it would be great to be able to extract them somehow. Maybe a generalization of trigger functions to look at both history and trends is a tough challenge, but implementing trend-only siblings of history-only functions seems not that difficult. To distinguish them from each other trend-only functions may be preceded by 't' or underscore or exclamation mark or whatever... t_avg(), t_delta(), t_min(), t_max() can be calculated easily for trends. t_count() with limited functionality is also possible. t_forecast() and t_timeleft() are very intriguing.

I think t_functions should also support sampling to account for weekly, monthly, yearly cycles. (If we had weather records in the database we could ask a question: "Is the weather today much different from the weather on the same date averaged for the last century?" )

Benefits:

  • saving space in the database if all you are keeping history for is averaging and minimum/maximum finding;
  • more robust anomaly detection (the load today may be twice as high as the load one week ago just because that day was a holiday);
  • long-term forecasting and capacity planning.
Comment by David Lang [ 2016 Jan 05 ]

rrdtool implements the Holt-Winters Time Series Forecasting Algorithm for this purpose.

It lets you define additional lines on your graph for expected value and expected value +- X standard deviations. It then lets you do comparisons of the actual value with these calculated expected values and take action based on the result

This lets you do something like "Alert if the actual value is > 2SD away from the expected value" which can alert for 'traffic too high' at 3am on sunday for the same traffic level that generates a 'traffic too low' at 9am on monday.

It 'learns' the patterns. As I understand it, after ~10 cycles of the pattern it will be pretty close to accurate. so after a couple weeks you can rely on it's daily pattern, after a couple months it will notice weekends with great accuracy, etc.

some useful links for this

the original usenix paper and slides
http://usenix.org/legacy/publications/library/proceedings/lisa2000/full_papers/brutlag/brutlag_html/index.html
http://www.hpl.hp.com/news/events/csc/2005/jake_slides.pdf

info in it's implementation in rrdtool (also under GPLv2 so code can be copied directly)

http://cricket.sourceforge.net/aberrant/rrd_hw.htm
https://oss.oetiker.ch/rrdtool/doc/rrdcreate.en.html

Comment by David Lang [ 2016 Jan 05 ]

I think that there should be multiple trend functions available, let the admin define what functions they want to have calculated for a particular item, and then have the results of those functions available for alert calculations.

The brute force method would be to track the result of these trend functions just like any other data item, and I think that this is a reasonable default to start with. If the database (or other available code to call) can calculate this on the fly, that may be an option, but it can end up being a tradeoff between the overhead of calculating the trend data each time a graph is viewed vs the space to store the data. I expect that for anything more than a trivial trend function, it's going to be cheaper to calculate it once and store it than to calculate it each time it's displayed.

In displaying trends on the graph, we may want to have more options than just 'display items' and 'display items stacked', you may want to display your main data items stacked, but with the trend data as lines overlayed with the data, with the trend line being relative to the data it's associated with.

Comment by Marcel Jäpel [ 2016 Feb 28 ]

Hi,

I agree to Glebs Ivanovskis.
If I use large time frames in triggers for historical data than only functions like min/max/avg are important. Not each single values.

I have several items where I have 2 triggers: one for current situation (needs history for the last day) and one for longtime situation (needs history for the last 30/90/365 days). So trend data for longtime triggers would be great.
Currently I have to fill my hard drives with history data altough the same information is existing in trends and take much less capacity.

Please make trend data available for trigger expressions. Never mind if there would be separat trigger functions for this or you want to extend the existing functions.

Regards,
Marcel

Comment by richlv [ 2016 Feb 29 ]

Marcel, there's a way to keep less information in the history tables - you could create calculated items that do something like avg(1w), then only keep history data for the referenced items for one week - enough to fill in the calculated item.
the calculated item would then be used in long-term triggers.

it would add extra items, so that definitely does not invalidate the need to use trend data as per this feature request.

Comment by dimir [ 2020 Aug 25 ]

Looks like something similar is being implemented here: ZBXNEXT-6140

Comment by Alexei Vladishev [ 2021 Mar 30 ]

@dimir, Correct, I close this ticket.

Generated at Sat Apr 20 06:13:46 EEST 2024 using Jira 9.12.4#9120004-sha1:625303b708afdb767e17cb2838290c41888e9ff0.