[ZBX-13424] Pre-processing doesn't work in calculated item Created: 2018 Feb 01  Updated: 2024 Apr 10  Resolved: 2018 Mar 17

Status: Closed
Project: ZABBIX BUGS AND ISSUES
Component/s: API (A), Templates (T)
Affects Version/s: 3.4.6
Fix Version/s: 3.4.8rc1, 4.0.0alpha5, 4.0 (plan)

Type: Problem report Priority: Minor
Reporter: Vadims Kurmis Assignee: Ivo Kurzemnieks
Resolution: Fixed Votes: 0
Labels: items, templates, webmonitoring
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Team: Team B
Team: Team B
Sprint: Sprint 27, Sprint 28, Sprint 29
Story Points: 1

 Description   

Steps to reproduce:

Have Web Scenario, which generates several items. Those cannot be edited directly, but can be referenced in Triggers and newly created items - just fine.

One of them is: web.test.error[Get-Status], which contains text string of last error.
Error text string is similar to: "Could not resolve host: Could not resolve host: '127.0.0.1/TestValue=84'."

Then I create item:
type: Calculated Item
key: LevelValue
value type: Numeric or float

Formula:
tried both, this:

last("web.test.error[Get-Status]")

and even this:

last(\"web.test.error[Get-Status]\")

Pre-Processing:

I have tried both:

Reg-exp:  ^.+([0-9]+$
Value: \1

OR

Trim Left: 69
Trim Right: 2

Result:

Item not supported.
Error:
Cannot evaluate expression: expected numeric token at "Could not resolve host: Could not resolve host: '127.0.0.1/TestValue=84'.".

Expected:

Expected that this calculated item would strip-out numeric value from another item, a have it stored as numeric item.

Could anyone explain me, WHY there are pre-processing options like "regexp" and "trim" in "calculated item" type, if those simply do not work?
As the value of "last("web.test.error[...]") is evaluated against Numeric data type - BEFORE Pre-Processing is performed?
What the point of pre-processing then?
If it doesn't work before that numeric data type evaluation.



 Comments   
Comment by Glebs Ivanovskis (Inactive) [ 2018 Feb 01 ]

Instead of calculated item try dependent item. I think it should work.

Comment by Vadims Kurmis [ 2018 Feb 01 ]

No, unfortunately Dependent Item - Doesn't see "webscenario items"

Comment by richlv [ 2018 Feb 02 ]

is your regexp missing closing parenthesis, and incorrectly anchoring to the end of the string ?

Comment by Vadims Kurmis [ 2018 Feb 02 ]

No, it was a typo. Regex: ^.([0-9])+$
and I always check first with online regex validators.
Besides, regex alternative, Trim Left/Right, should have worked anyway.

Comment by Vadims Kurmis [ 2018 Feb 02 ]

To me, the logic in this case looks the following:

1)
Regular expression with capture group support and Trim functions - are Exactly made for stripping out the required part of Strings.
Regular expressions and Trim - are by idea designed for string manipulation.

2)
If these functions are available - then you should be able to use them. They should work.

3)
If pre-processing block is available then it should be working as its name says: PRE - means - before - means processing Before.
Before final evaluation against Numeric or Float data type.

4) Evaluation against Numeric or Float data type should be initiated only when required.

It it required:
1) before any MATH operation in "Calculated item" body ( last(ItemName) - this function is not a Math operation ! )
2) before any MATH operation in Pre-processing block ( Regex capture groups and Trim - are not Math operations ! )
3) and finally evaluation against numeric - should be At The END - just before actual item creation, so this makes possibility for zabbix users to fulfill Regex/Trim manipulations BEFORE -
to get actual needed numeric part from the string.

Comment by Vadims Kurmis [ 2018 Feb 02 ]

Here, I found a comment that confirms my thoughts:

In Ticket: ZBX-13230

https://support.zabbix.com/browse/ZBX-13230?focusedCommentId=255558&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-255558

Comment by Vadims Kurmis [ 2018 Feb 02 ]

From documentation: https://www.zabbix.com/documentation/3.4/manual/config/items/itemtypes/calculated

The correct syntax of a simple formula is:
func(<key>|<hostname:key>,<parameter1>,<parameter2>,...)
, Where:
func One of the functions supported in trigger expressions: last, min, max, avg, count, etc

now if we check functions link:
https://www.zabbix.com/documentation/3.4/manual/appendix/triggers/functions

we find this definition:
last (sec|#num,<time_shift>)
Supported value types: float, int, str, text, log

and at the bottom of page we see:
1) All functions return numeric values only (Whaat ??!). Comparison to strings is not supported, for example
P.S: if so, then Trigger functions would never work for any String item and .last() function.

This is confusing.
Looks like some things are messed up.

Comment by Glebs Ivanovskis (Inactive) [ 2018 Feb 02 ]

Let me do some explanation... First of all, I strongly recommend you read a beautiful piece of documentation describing operation of item preprocessing.

1) I agree, applying regexps and trimming numbers does not make much sense. Usually. But since Zabbix really lacks mathematical operations, regexp can save you someday.
2) They work, but they are applied to the end result of calculation.
3) Pre- means "before". Before saving the value into DB and putting it into triggers. But from another perspective it's postprocessing since it's done after data gathering (which in case of calculated items is formula calculation).
4) Not sure what you mean by that.

You completely miss the fact that there could be many items in calculated item formula and it would be strange to apply same preprocessing to all of them.

As I said, for your use case you should create a Dependent item. I know it is not possible at the moment, but maybe it's a bug. I'll try to figure out why such decision was made and will get back to you later. Stay tuned.

Comment by Glebs Ivanovskis (Inactive) [ 2018 Feb 02 ]

P.S: if so, then Trigger functions would never work for any String item and .last() function.

They only work if your Character, Text or Log item for some reasons stores numbers, see ZBX-10449.

Comment by Vadims Kurmis [ 2018 Feb 02 ]

"But from another perspective it's postprocessing"

That's it.
The name of this feature is confusing.

Pre - processing - is logically - apply some specific action, that happens over data - Before actual data processing.
Actual data processing is - Formula.

But this really is - Post-processing. - means - apply some specific action - AFTER we have acquired and made manipulations on data.

"You completely miss the fact that there could be many items in calculated item formula and it would be strange to apply same preprocessing to all of them."

I agree, this makes sense.

But this name "pre-processing" - is incorrect then.

Comment by Vadims Kurmis [ 2018 Feb 02 ]

what about last() function which in documentation said: supports - num, float, str, text types
and it fails because of string value in this Formula block.
however what it does - just gets the last value from the item - nothing else.
Its not a math function, not an avg, not min. max, not any calculation.

Comment by Glebs Ivanovskis (Inactive) [ 2018 Feb 02 ]

"Processing" is triggers, for all item types. Formula is a kind of "data gathering" for calculated items. "Preprocessing" means "before processing triggers". I hope this starts to make sense if you think of it this way.

Comment by Vadims Kurmis [ 2018 Feb 02 ]

this ZBX-10449 doesn't throw a light on this behavior.

as documentation says: Supported value types: float, int, str, text, log

why it then evaluates on numeric and fails?
this is a bug?

Comment by Vadims Kurmis [ 2018 Feb 02 ]

"I hope this starts to make sense if you think of it this way."
But this is a hack.

Because pre-processing of data - literally means - apply some changes - before actual data manipulation.
In triggers - you do not manipulate this data, you do not process it, as the data - remains unchanged.
Triggers, graphs - is a reaction or display of data.

I have worked in DSP (Digital Signal Processing) laboratory so I exactly understand the term of "processing"

Comment by Vadims Kurmis [ 2018 Feb 02 ]

and that " a beautiful piece of documentation "
was made, I suppose, exactly because "Preprocessing" in Zabbix Items - was (and possibly still is) - confusing for many people

Comment by Vadims Kurmis [ 2018 Feb 05 ]

Hello,

any updates on why "web monitoring" items are not available for "dependent items" ?
maybe developers just forgot about them

Comment by Vadims Kurmis [ 2018 Feb 05 ]

I've noticed this thing is controlled by an URL, which is popped when I press "select" dependent item.

http:/zabbix.server/popup.php?srctbl=items&srcfld1=itemid&dstfld1=master_itemid&dstfrm=itemForm&srcfld2=master_itemname&dstfld2=master_itemname&only_hostid=32&with_webitems=0&excludeids[]=

as you can see, the argument value "with_webitems=0" possibly is the key to how web items were hidden.

Comment by Vadims Kurmis [ 2018 Feb 05 ]

Yes, I confirm, that opening this url with with_webitems=1 - does Unhide web items.
However, the opened page has no reference to parent page, where it should pass selected item.
Maybe it is possible to modify somewhere in zabbix sources to make this parameter "with_webitems=1" or use CURL to create this item, but CURL would be hard and will take a lot of time, since there are needed many actions and clicks to add one item.

Comment by Glebs Ivanovskis (Inactive) [ 2018 Feb 06 ]

Discussed and the conclusion is that Web scenario items should be available for selection as master item.

Comment by Vadims Kurmis [ 2018 Feb 06 ]

I tried to trick the system, changed hardcoded setting with_webitems to "1", then changed passed HTTP headers, but Zabbix system designed smarter.
It said: I have no permissions to this object

Thanks, that may help some day to those who want to reflect web scenario failures or web page load speed on a graph, for example, or those who want to get and process error of web scenarios.

I just wanted to read Ink levels on a web page from HTTP enabled printer, which does not have SNMP. And create numeric items from obtained ink levels on a printer web page.
I can successfully strip out Ink Level, say 83% for example, in my web scenario, using these nice Regexp Capture groups. But there was no option to return acquired and processed value.
So I found out a way to return captured numeric value - through the web.error item.
But, it unfortunately appended it with text, so there are no way to turn it into new numeric item and by coincidence web items were not available in "dependent items".

Comment by Ivo Kurzemnieks [ 2018 Feb 23 ]

(1) translation string changes

Strings added:

  • Item "%1$s" has master item and cannot be copied.

Strings deleted:

  • Item "%1$s" have master item and cannot be copied.

sasha RESOLVED

iivs Thanks! CLOSED

Comment by Ivo Kurzemnieks [ 2018 Feb 23 ]

RESOLVED in svn://svn.zabbix.com/branches/dev/ZBX-13424

Comment by Vadims Kurmis [ 2018 Feb 27 ]

Hi,

Do you add these web items to be visible in "Calculated items" as well?
Because there was at least one numeric item, I guess site loading latency/speed, something, that could be used in Calculated items via last(web.test.time) or last(web.test.in)

Comment by Ivo Kurzemnieks [ 2018 Mar 15 ]

Fixed in:

  • pre-3.4.8rc1 r78686 (corrected ChangeLog r78687)
  • pre-4.0.0alpha5 r78688
Comment by Ivo Kurzemnieks [ 2018 Mar 15 ]

(5) [D] API documentation updated.

sasha CLOSED

Generated at Sat Apr 20 08:08:18 EEST 2024 using Jira 9.12.4#9120004-sha1:625303b708afdb767e17cb2838290c41888e9ff0.