[ZBXNEXT-3205] Add support for wildcards and / or global regular expressions in context macros Created: 2016 Mar 21  Updated: 2024 Apr 10  Resolved: 2020 Jun 11

Status: Closed
Project: ZABBIX FEATURE REQUESTS
Component/s: Frontend (F)
Affects Version/s: 3.0.1, 4.0.14, 4.4.1, 5.0.0alpha1
Fix Version/s: 5.0.2rc1, 5.2.0alpha1, 5.2 (plan)

Type: Change Request Priority: Major
Reporter: Ingus Vilnis Assignee: Andris Zeila
Resolution: Fixed Votes: 21
Labels: context, macro, macrocontext, regexps
Σ Remaining Estimate: Not Specified Remaining Estimate: Not Specified
Σ Time Spent: Not Specified Time Spent: Not Specified
Σ Original Estimate: Not Specified Original Estimate: Not Specified

Attachments: PNG File macros.png    
Issue Links:
Duplicate
is duplicated by ZBXNEXT-4000 Regular expression support in context... Closed
Sub-task
depends on ZBXNEXT-4004 offload LLD data processing from trap... Closed
Sub-Tasks:
Key
Summary
Type
Status
Assignee
ZBXNEXT-5938 Server side changes to add support fo... Specification change (Sub-task) Closed Roberts Lataria  
Team: Team A
Sprint: Sprint 13, Sprint 64 (May 2020), Sprint 65 (Jun 2020)
Story Points: 2

 Description   

Context macros now work great if the context is a static string.

It has some limitations though, e.g. if one would like to set a different trigger threshold for all LLD disk mounts with string home in it.

It is impractical to have a hardcoded context macro for every possible mount point which may happen in large organizations.

Here are few possible examples for this request:

{$LOW_SPACE_LIMIT:/*/home*}

or even better:

{$LOW_SPACE_LIMIT:@Global regexp}


 Comments   
Comment by richlv [ 2016 Mar 21 ]

good idea. for consistency, should probably support regexps + global regexps, thus the first example would look like this :

{$LOW_SPACE_LIMIT:/.*/home/.*}

using curly braces likely would require quoting :

{$LOW_SPACE_LIMIT:"/home/[a-z]{3}"}
Comment by Glebs Ivanovskis (Inactive) [ 2017 Jan 30 ]

I suspect the main source of such variable contexts is LLD. The syntax then could look like {$LOW_SPACE_LIMIT:"{{#FSNAME}.regsub(\"(boot|home|root)\", \1)}"}. The remaining part is defining {$LOW_SPACE_LIMIT:boot}, {$LOW_SPACE_LIMIT:home}, {$LOW_SPACE_LIMIT:root} and default {$LOW_SPACE_LIMIT}. This uses existing syntactic features - macro functions - and goes in line with other requests that could be implemented in a form of macro functions for LLD macro. But it has few problems. From user perspective quoting and double quote escaping will very soon get totally unbearable and from developer's perspective parsing such "Russian doll macros" gets exponentially harder with every new nested layer.

Comment by Glebs Ivanovskis (Inactive) [ 2017 Jan 30 ]

With regexp directly in macro context it is a bit ambiguous, suppose we have:

{$M:[a-z]} => 0
{$M:[0-9a-f]} => 1

What would be the value of {$M:a}? Is "a" a letter or a hexadecimal digit?

Comment by Glebs Ivanovskis (Inactive) [ 2017 Jul 28 ]

Today I wouldn't suggest to implement this feature as I proposed 7 months ago. Today we have item preprocessing (ZBXNEXT-1443) and very much loved IPC solution with master-workers pattern first used in ZBXNEXT-3386.

As we discussed, after offloading LLD processing from data gathering processes (ZBX-11782) to a separate master-workers pipeline it will be easy to implement preprocessing for LLD rules as well (currently they stay aside from ZBXNEXT-1443 and ZBXNEXT-3006 party). Given that there are preprocessing abilities for LLD rules we can then implement the feature in question as a clever preprocessing rule. I have no syntax suggestions, but it should do the following: "for every row in LLD JSON take {#THIS} macro, apply regsub(...) and put result in {#THAT} macro". Since same regular expression is applied to all LLD JSON rows, regular expression can be compiled only once, which is nice for performance.

Existing LLD filters can be naturally implemented as another LLD JSON preprocessing (like custom multipliers and deltas became item preprocessing options in ZNXNEXT-1443): "for every row in LLD JSON take {#THIS} macro and if it (not) matches @regexp delete row". This is nice because it reduces complexity of Zabbix.

However, such implementation path affects frontend, API, XML import, server and requires DB upgrade. And, as said, it depends on ZBX-11782 and preprocessing for LLD rules.

Comment by Glebs Ivanovskis (Inactive) [ 2017 Jul 28 ]

If by "old proposal" you mean issue description or this comment - it is simply out of question for a simple reason: ambiguity. Let's say, I've created two macros:

{$M:foo} => 0
{$M:bar} => 1

Here "foo" and "bar" are regular expressions, not just strings. Now Zabbix encounters macro {$M:foobar}. It's context matches both "foo" and "bar", which value should Zabbix choose? Additionally, there is a cost of regexp compilation and matching times the number of different contexts defined for {$M}, context macro substitution will be really slow in this case.

If by "old proposal" you mean my initial suggestion I'm not sure it would be cheap either. It does not require changing frontend, API or XML import, it does not require patching DB, but server changes are very substantial. Due to complicated syntax (for example: {$LOW_SPACE_LIMIT:"{{#FSNAME}.regsub(\"(boot|home|root)\", \1)}"}) user friendliness is almost zero. And performance wouldn't be ideal either.

Comment by Dimitri Bellini [ 2017 Jul 31 ]

Hi Glebs,
I discussed your new implementation with Ingus but please keep in mind this points:
1- Provide the Macro context regexp with the old proposal - If its not so huge amount of work - Solve my problem and provide a better implementation of "Macro context" - Catch some money from me for the development now
2- Find all the money needed (not so easy) and wait for a very good implementation like your suggestion (maybe one year but is not garanteed).
From the point of view of a customer (but also the zabbix community), we actually have a good feature but it lack of the "potentially" of the daily usability, so my suggestion is to go with option 1 (try to develop with the old proposal) if is not a huge work and when will be possible substitute it with your better implementation.
I hope you could understand my point of view.
Thanks very much

Comment by Glebs Ivanovskis (Inactive) [ 2017 Jul 31 ]

Dear dimitri.bellini, please check my comment above.

Comment by Dimitri Bellini [ 2017 Jul 31 ]

Hi Glen,
Sorry, it was richlv to propose the "macro context regexp".
Ok i could understand but there are no other easy way to accomplish my need?
At beginning of my request i have proposed a new "item" on trigger expression, something like {{#FSNAME}.regexp(@MyExcludePath)} but seems too complicate. Do you think is too complex to develop like your last proposition?
example Trigger expression:

{ISP - Template VCS OS Linux:vfs.fs.size[{#FSNAME},pfree].last(0)}<{$LOW_SPACE_LIMIT:"{#FSNAME}"} and {{#FSNAME}.regexp(@MyExcludePath)}=0

Sorry for so many question but i think "Macro context" is very useful but it lack of flexibility requested for big installations.
Thanks very much

Comment by Glebs Ivanovskis (Inactive) [ 2017 Jul 31 ]

I don't get the idea of this trigger expression. How is it supposed to work?

Comment by Dimitri Bellini [ 2017 Jul 31 ]

Sorry for the bad explaination, i will try to clarify my idea:

{#FSNAME} LLD macro of a mountpoint

{{#FSNAME}

.regexp(MyRegExp)} => An Item that explode the content of the last value of

{#FSNAME}

and try to mach with a desired RegExp. 0 not matched | 1 matched

I hope you could understand my intention.
Thank you!

Comment by Glebs Ivanovskis (Inactive) [ 2017 Jul 31 ]

In my understanding if you have trigger prototype

{ISP - Template VCS OS Linux:vfs.fs.size[{#FSNAME},pfree].last(0)}<{$LOW_SPACE_LIMIT:"{#FSNAME}"} and {{#FSNAME}.regexp(@MyExcludePath)}=0

you will end up with two types of discovered triggers, either

{ISP - Template VCS OS Linux:vfs.fs.size[<somemountpoint>,pfree].last(0)}<{$LOW_SPACE_LIMIT:"<somemountpoint>"} and 0=0

or

{ISP - Template VCS OS Linux:vfs.fs.size[<somemountpoint>,pfree].last(0)}<{$LOW_SPACE_LIMIT:"<somemountpoint>"} and 1=0

depending on whether "<somemountpoint>" matching @MyExcludePath or not. And the latter trigger does not make much sense because it will never fire.

So how is this approach better than having two discovery rules with different filters?

Comment by Dimitri Bellini [ 2017 Jul 31 ]

Ok, i will try simplify
My intention is to filtering some Oracle mountpoint that is not need to be monitored "/oradata/PLUTO/database/datafile/datb".
So my prototype trigger must have something to say:
"for all mountpoint check free space < 5% but exclude the mountpoint with something like "%datafile%""

I also explore the simple idea of two "Filesystem Discovery" but i could not have same prototype keys on the same host.
Have you a better idea? Maybe i misunderstood your suggestion.

Comment by Glebs Ivanovskis (Inactive) [ 2017 Jul 31 ]

That was exactly my suggestion. There is an ugly workaround of putting spaces in prototype key that won't change the meaning of the key, but will make keys different.

It is doable in current Zabbix, but it is messy because you will end up with multiple discovery rules and multiple prototype sets to manage.

Comment by Dimitri Bellini [ 2017 Jul 31 ]

Ok, no other ways with not so huge development work?

Comment by Glebs Ivanovskis (Inactive) [ 2017 Aug 01 ]

Just to clarify. For all mount points you want an item and a trigger, but for those like "%datafile%" you just want an item (or maybe a trigger that never fires), right?

If you insist on writing triggers like this one

{ISP - Template VCS OS Linux:vfs.fs.size[{#FSNAME},pfree].last(0)}<{$LOW_SPACE_LIMIT:"{#FSNAME}"} and {{#FSNAME}.regexp(@MyExcludePath)}=0

it looks more like a request to apply trigger functions to non-item objects in trigger expression. Or like a request to save text {#FSNAME} in "calculated" item to use it then in trigger expression.

For your specific case I can suggest a custom vfs.fs.discovery (maybe implemented as a module) which would enrich LLD JSON with custom LLD macros based on existing LLD macro values and regexps. Like so:

vfs.fs.discovery[{#FSCATEGORY}={#FSNAME}.regsub(.*(datafile).*, \1)]

It is basically the same idea as LLD preprocessing, filtering and enrichment, but on agent level instead of server and for individual key.

Or one more option is making the "Create enabled/Create disabled" checkbox of every prototype dependent on a condition...

Comment by Dimitri Bellini [ 2017 Aug 01 ]

Hi Glebs,
yes you are right, i have wrote an example of "non working trigger" only to show what i want to do, the main purpose is to do not fire the trigger when the "mountpoint" is like "%datafile%".
I have also thought to write a custom LLD but is to complex on our customer's scenario, we have installed the Zabbix agent on 4000 clients and we have created a very slim/standard agent configuration for different OS.
I think your suggestion below seems interesting:
1- Concept of non-item objects in trigger expression
2- Calculated item based on MACRO result or item manipulations

I did not like so much the item proliferation (option 2) because in huge Zabbix installations can impact on performance but the option 1 seems good also.
Do you think is more easy to develop your suggestions than "macro context with regexp support" this case (ZBXNEXT-3205).
Sorry for so many questions.

Comment by Glebs Ivanovskis (Inactive) [ 2017 Aug 01 ]

I think we need to discuss possible solutions internally. There are so many options... I understand that the best solution in my opinion is a "step too big", but small step in wrong direction is not ideal either.

Comment by Dimitri Bellini [ 2017 Aug 01 ]

Yes i understand your position and i deal with you, i will wait for a final possible solution.
Thanks very much

Comment by Glebs Ivanovskis (Inactive) [ 2017 Aug 07 ]

Just letting you know, that it seems that discussion and decision regarding this development is postponed.

Comment by Dimitri Bellini [ 2017 Aug 07 ]

No to so good news
Thanks for your feedback.

Comment by Christoph Loesch [ 2017 Oct 01 ]

slightly offtopic but this comment got my attention:

Glebs Ivanovskis added a comment - 2017 Aug 01 08:53
> "Or one more option is making the "Create enabled/Create disabled" checkbox of every prototype dependent on a condition..."

it would be awesome if this could be implemented!

i have set up prototype items and triggers for network interface discovery to get notified when there is a higher throughput and for this i get many items and triggers created for every single interface per device, because the monitored devices, in my case routers, have many interfaces. i "handled" that by limiting global regular expressions, but that is not optimal this way.
same is, at least what i read on the forums, when people want that analogy for filesystem discovery.

hope this example gives an idea as to why it would be great to have an option for conditions here.

Comment by Евланов Александр Викторович [ 2017 Dec 11 ]

Syntax like this will be enough for solving problem of context part multi-matching:
{$FS_PFREE_AVG} instead of ambiguous, useless and absolutely unclear {$FS_PFREE_AVG:"#FSNAME"} - in common template
{$FS_PFREE_AVG:regexp(/db/vol[0-9]+)}=1
or
{$FS_PFREE_AVG:eq(/boot)}=1 - in host macros
{tmpl.os.linux:vfs.fs.pfree("{#FSNAME}").last} < {$FS_PFREE_AVG} - in trigger expression

If one regexp matches several macro values - pretend that user knows what she do and use last or first match (this behaviour must be documented of course).

{$FS_PFREE_AVG:regexp(/db/vol[0-9]+)}=1

  • is ergonomic and clear syntax. If the user wants to shoot her leg using this syntax - don't bother user, simply do what she want - i'm sure this is rightly and adequate solution.
Comment by Vladislavs Sokurenko [ 2020 Apr 23 ]

Please also have a look at regsub it could be used to solve the issue but probably not so user friendly
Value:

customername_1

regsub to extract what is needed

{$MACRO:"{{#IFALIAS}.regsub(\"(.*)_([0-9]+)\", \1)}"} 

Context macro with required info

{$MACRO:"customername"}
Comment by Roberts Lataria (Inactive) [ 2020 Jun 11 ]

Implemented in:

Updated documentation:

Generated at Sat Apr 27 03:31:54 EEST 2024 using Jira 9.12.4#9120004-sha1:625303b708afdb767e17cb2838290c41888e9ff0.