[ZBXNEXT-2617] Triggers on multiple items with the same clock should be evaluated at the same time Created: 2014 Nov 27  Updated: 2016 Jun 02

Status: Open
Project: ZABBIX FEATURE REQUESTS
Component/s: Server (S)
Affects Version/s: 2.4.2
Fix Version/s: None

Type: Change Request Priority: Major
Reporter: Strahinja Kustudic Assignee: Unassigned
Resolution: Unresolved Votes: 2
Labels: dbsyncer, triggers
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: PNG File items.png    

 Description   

Currently in Zabbix if you create a trigger on two (or more) items, the trigger is evaluated when each item last value is changed, even if both items change their value at the exact same time (using zabbix_sender), which makes it impossible to compare values of more items in triggers.

I uploaded an example. In the graph you can see two items which have a trigger on them which checks if the green item is greater or equal to the red one, if it is, it will send a notification. These items were updated in Zabbix at the exact same time which you can see from the database:

zabbix=> SELECT clock,value from history_uint where itemid = 98532 and clock > 1417042800 and value = 70;
   clock    | value                                                                                                                                                                                                
------------+-------                                                                                                                                                                                               
 1417060938 |    70
(1 row)

zabbix=> SELECT clock,value from history_uint where itemid = 98522 and clock > 1417042800 and value = 65;
   clock    | value 
------------+-------
 1417060938 |    65
(1 row)

As you can see clocks are the same, but Zabbix evaluated this item 2 times in 6 seconds:

  • the first time it evaluated when the value of the green item changed to 65 (which triggered a notification in Problem state, since it didn't use a value 70 from the red item), but the old value, which was 64,
  • after 6 seconds it evaluated the trigger again for value 70 of the red item, which then sent a notification with OK state.

What Zabbix should do is it should evaluate this trigger only one time for all items which receive a new value at the same time.



 Comments   
Comment by Constantin Oshmyan [ 2015 Feb 25 ]

Hello,

I could confirm that we also had the same problem.

As a workaround we use a bit more complicated trigger expression.
In your example, instead of initial trigger expression

{host:green.last()}>{host:red.last()}

it is possible to use the following:

{host:green.count(30)}>0 & {host:red.count(30)}>0 & {host:green.last()}>{host:red.last()}

I.e. initial trigger expression (comparing green and red items) will be used only if both these items have a values updated during the last 30 seconds.
The timeout (30 seconds in this case) should be smaller than an update interval for both items, but big enough to have Zabbix-server chance to correctly process both new values even on a busy server.

Comment by Strahinja Kustudic [ 2015 Feb 25 ]

That is a really cool solution, but the problem is that when that trigger becomes true, on next item fetch it will become false because of the count expressions. For that trigger to work, you would need to use TRIGGER.VALUE, something like this (I didn't test it and I use the 2.4 syntax):

({TRIGGER.VALUE}=1 and not ({host:green.count(30)}>0 and {host:red.count(30)}>0))
or
({host:green.count(30)}>0 and {host:red.count(30)}>0 and {host:green.last()}>{host:red.last()})

But all of this is still a workaround which makes trigger expressions harder to read and understand. Zabbix should support this out of the box for items with the same timestamp.

P.S. In your example, if your items are refreshed every 30 seconds, shouldn't you use count(29)>0?

Comment by Constantin Oshmyan [ 2015 Feb 25 ]

Good point, thanks.

We use a long enough refresh interval for items (about 20 minutes), so 30 seconds for the count() function is a good timeout. It is just to be sure that item was refrshed "recently" (i.e. both item's values are "fresh").

Really, we need to check a trigger state and describe a different conditions for correct trigger opening/closing. It is possible; however, a very simple trigger expression is transformed to a difficult for understanding (and error-prone) hysteresis-like expression. Of course, it could be much better if we can simple and clear trigger expressions "out of the box" (without these tricks).

Comment by Oleksii Zagorskyi [ 2015 Oct 10 ]

Issue description doesn't mention several important points.
Many values may come to zabbix server during single second and that's ok. Fact also is that zabbix uses nanoseconds for time precision.
It's guaranteed (on agent or server side) that two values of an item cannot get identical unix timestamp and nanosecond when they comes into write cache.
Write cache (including triggers evaluation) then processed by history syncers.

I don't think it can be implemented at all, as we even cannot resolve ZBX-9201

Comment by Strahinja Kustudic [ 2015 Oct 10 ]

Wouldn't it be possible to delay trigger evaluations on triggers with multiple items (for a second or something like that). If zabbix_sender sends two items with the exact same timestamp and those items are used in one trigger, when the first item value is received in the cache, check if the other item's value is within a second, if not, wait a second and evaluate the trigger again. Of course I have no idea if this is possible to do, since I haven't read the Zabbix trigger evaluation code.

If above is not possible, could we maybe add a new function which would make these triggers easier to read, or for example add an additional parameter to last(), e.g.:

{host:green.last(0,3s)}>{host:red.last(0,3s)})

3s means that both green and red have a fresh value within the last 3 seconds.

Comment by Oleksii Zagorskyi [ 2015 Oct 10 ]

1. I'm not developer but I suppose it would not be possible, IMO. Remember please about systems with very high NVPS. I could not imagine there any delays.
2. Didn't really get the example

Comment by Glebs Ivanovskis (Inactive) [ 2016 Jun 01 ]

I would translate suggested fictional

{host:green.last(0,3s)}>{host:red.last(0,3s)}

into supported

{host:green.fuzzytime(3s)}=1 and {host:red.fuzzytime(3s)}=1 and ({host:green.last()}>{host:red.last()})

Or more sophisticated variant which preserves existing trigger state if at least one of item values is not "fresh":

({host:green.fuzzytime(3s)}=1 and {host:red.fuzzytime(3s)}=1) * ({host:green.last()}>{host:red.last()}) +
({host:green.fuzzytime(3s)}=0 or {host:red.fuzzytime(3s)}=0) * {TRIGGER.VALUE}
Comment by Strahinja Kustudic [ 2016 Jun 01 ]

I don't understand how would fuzztyime(3s) help in this situation, since it only checks if the difference between current time of the host and Zabbix server is more than 3 seconds.

Comment by Glebs Ivanovskis (Inactive) [ 2016 Jun 02 ]

Not between host time and server time, but between item value timestamp and current server time.

Comment by Strahinja Kustudic [ 2016 Jun 02 ]

Looking at the docs I would say fuzzytime only works with items which value is the timestamp, if it's not, it cannot be used. If items red and green are not timestamps, then fuzzytime won't do anything.

Comment by Glebs Ivanovskis (Inactive) [ 2016 Jun 02 ]

Yes, indeed. Sorry for being misleading.

Comment by Oleksii Zagorskyi [ 2016 Jun 02 ]

in ZBX-10360 we improved the fuzzytime function description in documentation and item key helper (for 3.0 only).
The doc currently says "item timestamp value", which looks a bit more correct than Gleb's "item value timestamp"

Comment by richlv [ 2016 Jun 02 ]

and it sounds like you might want to vote on ZBXNEXT-2045

Generated at Mon May 12 06:38:22 EEST 2025 using Jira 9.12.4#9120004-sha1:625303b708afdb767e17cb2838290c41888e9ff0.