[ZBXNEXT-4016] Extend item preprocessing with subsampling function Created: 2017 Aug 03  Updated: 2023 Oct 25  Resolved: 2023 Oct 25

Status: Closed
Project: ZABBIX FEATURE REQUESTS
Component/s: Server (S)
Affects Version/s: 3.4.0beta1
Fix Version/s: None

Type: New Feature Request Priority: Major
Reporter: Glebs Ivanovskis (Inactive) Assignee: Unassigned
Resolution: Duplicate Votes: 6
Labels: preprocessing
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: PDF File Advanced preprocessing 1.0rc1.pdf     PDF File Advanced preprocessing 1.2.pdf    
Issue Links:
Duplicate
duplicates ZBXNEXT-4724 Advanced preprocessing: validation an... Closed

 Description   

As far as I can tell from #zabbix IRC channel, users are concerned about the current implementation of ZBXNEXT-3006. The problem is that all dependent values will get a value every time master item gets a new value. This is not ideal in the case when bulk request result consist of rapidly changing metrics and slowly changing metrics. Naturally one would want to poll rapidly changing metrics more often while having new value of slowly changing metric less frequently to keep database size (and graph performance, and other things) under control.

My suggestion is to provide a preprocessing option that would only pass through every other, or every third or every fourth... value.

Or subsampling can be based on minimal time since previous value.

Or (but this is a bit different thing, kinda solution for ZBXNEXT-1846) subsampling can be based on whether value changed from last time.



 Comments   
Comment by Marc [ 2017 Aug 07 ]

How about making this configurable on dependent item level? This seems more intuitive to me.
There the master item's update interval should also be displayed for information. A valid update interval must be a multiple of the master item's interval.

PS: A constraint to the master item's update interval may become problematic with the use of user macros.

PPS: Same to flexible/scheduled intervals. Possibly there should be no constraint at all but a smart algorithm for interpreting the update interval.

Comment by Marc [ 2017 Aug 14 ]

I fear that whatever is done, it only leads to even more dis-improving the whole thing. I stop thinking about this for now and looking forward to the Eichhoernchen proejct - if ever implemented

Edit:
Anyway, you got my vote!

Comment by richlv [ 2017 Aug 23 ]

i'm voting on this one because of the excitement when reading "...subsampling can be based on whether value changed from last time"

Comment by Alexei Vladishev [ 2018 Jul 13 ]

I think it might be implemented as a new type of preprocessing rule that would ignore new incoming value if previous value exists in a range of N seconds. Therefore it would guarantee that this item will not be collected more often than once per N seconds.

Comment by Alexei Vladishev [ 2018 Aug 20 ]

Hey all! Please have a look at the attached acceptance criteria document that describes proposed enhancements for preprocessing that will cover this functionality as well as ZBXNEXT-1846 and ZBXNEXT-3544. It has all chances to be implemented in Zabbix 4.2. Let me know what you think. Is there anything to improve? I appreciate any feedback.

Comment by Glebs Ivanovskis [ 2018 Sep 05 ]

Dear alexei, consider adding support for a macro ({STEP.ERROR}?) in Set error action which would resolve to an error from Zabbix server. This would give an opportunity to combine the best of both worlds (user-friendly description + technical details from server).

Comment by Dimitri Bellini [ 2018 Sep 11 ]

And maybe add also this feature ZBXNEXT-4713, is very useful for log monitoring.

Comment by Alexei Vladishev [ 2018 Sep 21 ]

I attached new v1.2 of the document with a few important enhancements added.

Gleb, such macros would probably require control flow operators. Let's keep things simple for now. See v1.2 of the document, we do support ability to extract errors from incoming data.

Dimitri, I think ZBXNEXT-4713 issue is quite unrelated to what we are discussing here.

It is probably better to move discussion to ZBXNEXT-4724, which is an umbrella ticket for preprocessing related improvements.

Comment by Glebs Ivanovskis [ 2018 Sep 23 ]

such macros would probably require control flow operators

Dear alexei, I don't see where my suggestion would require changes in flow control. To ensure we are on the same page I will elaborate. I was referring to this part of specification (present in both versions):

Zabbix user may specify optional "On fail" action for each rule, which will be executed if this rule fails:

  1. Discard value: preprocessing will stop immediately and Zabbix will ignore the value as it never existed
  2. Set value to + "<value>": Zabbix will set result of preprocessing to <value> (can be empty) and preprocessing will continue normally
  3. Set error to + "<error message>": preprocessing will fail and error message will be set to provided non empty "<error message>"
  4. If no action is specified then Set error is executed with internal error message set by Zabbix Server

Here is how I imagine the algorithm:

on_fail(on_fail_action, internal_error_message, value_to_set, error_to_set)
{
    switch (on_fail_action)
    {
        case DISCARD_VALUE:
            return {.next_action = STOP_PREPROCESSING, .error = NULL, .value = NULL};
        case SET_VALUE_TO:
            return {.next_action = CONTINUE_PREPROCESSING, .error = NULL, .value = value_to_set};
        case SET_ERROR_TO:
            return {.next_action = STOP_PREPROCESSING, .error = error_to_set, .value = NULL};
        default:
            return {.next_action = STOP_PREPROCESSING, .error = internal_error_message, .value = NULL};
    }
}

My suggestion is to add macro substitution in case SET_ERROR_TO:, like so:

on_fail(on_fail_action, internal_error_message, value_to_set, error_to_set)
{
    switch (on_fail_action)
    {
        case DISCARD_VALUE:
            return {.next_action = STOP_PREPROCESSING, .error = NULL, .value = NULL};
        case SET_VALUE_TO:
            return {.next_action = CONTINUE_PREPROCESSING, .error = NULL, .value = value_to_set};
        case SET_ERROR_TO:
            return {.next_action = STOP_PREPROCESSING, .error = replace(error_to_set, "{STEP.ERROR}", internal_error_message), .value = NULL};
        default:
            return {.next_action = STOP_PREPROCESSING, .error = internal_error_message, .value = NULL};
    }
}

Current specification allows user to get either human readable error message (e.g. "We tried to extract some useful data, but failed.") or very technical message from Zabbix (e.g. "Invalid XPath syntax."). That

replace(error_to_set, "{STEP.ERROR}", internal_error_message)

will allow to get the best of both worlds (e.g. "We tried to extract some useful data, but failed. Here is why: {STEP.ERROR}" would be transformed into "We tried to extract some useful data, but failed. Here is why: Invalid XPath syntax.").

Comment by Dimitri Bellini [ 2018 Sep 24 ]

@Alexei: Sorry you are right

Generated at Wed Apr 24 05:26:50 EEST 2024 using Jira 9.12.4#9120004-sha1:625303b708afdb767e17cb2838290c41888e9ff0.