Trigger with "Multiple PROBLEM events generation" in combination with timer-related functions: nodata(), date(), dayofmonth(), dayofweek(), time(), now()

Example 1: Using "nodata()" function to close trigger automatically by time-out

Task

It is necessary to have a trigger for Windows Application Log: if a new message with the severity "Error" or "Critical" appeared, it should be forwarded the administrator by e-mail with a delay maximum 1 minute.

Item (in the appropriate Template)

Key: eventlog[Application,,Error|Critical",,,100,]
Type: "Zabbix agent (active)"
Type of information: "Log"
Update interval (sec) = 30

(Trigger in the same Template)

Name: New error message in Windows Application log
Expression:

({Template OS Windows:eventlog[Application,,"Error|Critical",,,100,].logseverity(0)}=4 | {Template OS Windows:eventlog[Application,,"Error|Critical",,,100,].logseverity(0)}=9 ) & {Template OS Windows:eventlog[Application,,"Error|Critical",,,100,].nodata(30)}#1

Multiple PROBLEM events generation: ON
Note1: the "nodata(30)}#1" condition is used to close this trigger automatically (the 30 seconds is a minimum value for "nodata()" function).
Note2: the "Multiple PROBLEM events generation" option is necessary: otherwise it is possible to miss some events if they goes one-after-another.

Action

Action has the following logic:
IF
Trigger value = "PROBLEM"
and
Trigger name like "message in Windows"
THEN
Send message to users: Admin1, Admin2, ..., AdminN
ENDIF

Results

If several error messages appeared in the given interval (30 seconds), then all messages are delivered successfully. However, the last message is delivered twice: the 1-st time upon new datas receiving from the Agent, and the 2-nd time - generated by the timer process. If the timeout for nodata() function is longer, then the last message is repeated every 30 seconds: for example, for 10 minutes (to have possibility for operator to see it on the Web-console) it repeates 21 times.

The first and second item references in your trigger seem to be identical. You might wish to simplify that.

asaveljevs, thank you! I've fixed this example (logseverity are different: ERROR and CRITICAL).

Example 2: using "time()" function as an additional condition clause

Initial Task

It is necessary to monitor a log.file of some application for error messages (lines containing "ERROR" string) for notification of the appropriate administrator.

Item

log[/var/log/myApp/myApp.log,ERROR,,,skip,]

Trigger

{Host:log[/var/log/myApp/myApp.log,ERROR,,,skip,].str(ERROR)}=1

As in example 1, the "Multiple PROBLEM events generation" should be enabled to avoid missing of some messages.

Result

It works OK. However, every night (between midnight and 02:00) the database performs an offline backup, it cause to some error messages in the log that could be ignored.
So, the administrator wants to exclude any error messages at this period.

Modified Task

It is necessary to monitor this log.file for error messages only after 02:00 AM.

Modified Trigger

{Host:log[/var/log/myApp/myApp.log,ERROR,,,skip,].str(ERROR)}=1 & {Host:log[/var/log/myApp/myApp.log,ERROR,,,skip,].time(0)}>020000

Result

Despite of minimal changes, the result will very differ. If some error occurs in this log file after 02:00 AM, then the event "Trigger goes to PROBLEM state" will be generated every 30 seconds by the timer process; in this example - the rest of day up to midnight...

Looks like a regex to find A and not B on a line could be a workaround in the example 2. Of course, it will not help in the first example.

Looks like a regex to find A and not B on a line could be a workaround in the example 2. Of course, it will not help in the first example.

I agree that in some cases it's possible. If the monitored log file includes the clearly formatted timestamp, for example log of Zabbix-server:

  7294:20141227:100121.813 SNMP agent item "ifNumber" on host "CiscoV-SW1" failed: first network error, wait for 15 seconds

then a trigger expression could be re-formulated to use a regexp() instead of time() function, something like the following:

{Host:log[/tmp/zabbix_server.log,error,,,skip,].str(error)}=1 & {Host:log[/tmp/zabbix_server.log,error,,,skip,].regexp([0-9]*:[0-9]{8}:0[01][0-9]{4}\.[0-9]{3})}#1

I.e. "if the timestamp in this record of log file has the hour==00 or hour==01, then ignore".

However, in other cases it is difficult or impossible to use just regexp. For example, in the Windows Event logs the timestamp is a separate field; many Java applications have a multi-line error messages (where the timestamp and the message text are on different lines), some applications could have their timestamps in the same format that could occurs in the message text also, etc. After all, using the time() function is just more understandable.

ZBXNEXT-1604 is related a bit

Unfortunately, all new versions still have this trouble.
Add, please, the 3.0.x, 3.2.x and 3.4.x to "Affects Version/s:" header (I'm reporter for this issue, but anyway I have no permissions for that).
Thanks in advance.

Agree, this is very useful feature! Voted.
If it will never be implemented, please provide proper way to alarm on every occurrences of consecutive lines in a log file (with no duplicates).

Just a reminder as this problem is still actual. And it still exists in v4.0.x also (and, probably, 4.2 and 4.4 as well).

Just once more reminder about this problem importance. It still does exist in versions 5.0 (LTS) and, probably, 5.2 and 5.4 also.
As well, this problem regularly occurs for novice users and discussed on Zabbix forum (at least, Russian-language: link1, link2, link3, link4).

Reminder: this problem is still actual.
New LTS version 6.0.x is still affected.

+1 Voted as well! For example, we would simply like to have the successful and failed logins (from each login) as a separate info event in Zabbix. Therefore only multiple event generation is practicable to not miss any events. But the event should close automatically after X minutes. Due to the mentioned issue this can't be implemented, which makes eventlog, log, logrt and snmptrap quite useless (at least for simple info events that don't send a resolved notification). With this workaround there is probably a way, but Zabbix should offer something more user friendly.

FYI, 6.2.x is still affected.

Hi DevTeam,
as already mention in the thread, we still suffer of this "problem" on Zabbix 6.2.x.
There are some possible idea to fix this kind of scenario for the next 6.4 or 7.0?
Thanks so much

alexei, just reminder, as discussed at Zabbix Summit 2023
This problem is one of the most annoying and long-lived, unfortunately

Possible fix to explore as a starting point (not released yet):
ZBXNEXT-2452-6.0.diff
ZBXNEXT-2452-6.4.diff
ZBXNEXT-2452-6.5.diff

vso, is your possible fix for which version of Zabbix, please?

constantin.oshmyan attached both for 6.4 and 7.0, the fix is as per your request in description.

vso, great, thanks! Can this patch be applied also to the v6.0 (the current LTS version)?

It's only a quick research thing to try out, if you have a chance. For a proper thing this needs to be code-reviewed, tested, documented and so on.

Another thing that needs to be considered here is a possible regression for those that need an every 30-second alarm. Maybe a checkbox should be added. Anyway, we wanted to quickly check how complicated the fix might be here and if you can try this maybe become aware of some possible side effects/regressions early.

constantin.oshmyan sure, added patch for 6.0 ZBXNEXT-2452-6.0.diff. It is safe to use unless as dimir mentioned someone will need also old behaviour for some triggers.

Thank you, guys!
I'll try it just after finding a workaround for the ZBXNEXT-8014 that is high priority for me at the moment...

vso, dimir I've tried this patch in the test environment with the current v6.0.22 release; it works great!
Thank you very much!

Another thing that needs to be considered here is a possible regression for those that need an every 30-second alarm.

Yes, I understand your doubts.
However, this regression is probably theoretical only.
I could not imagine the real situation where such behaviour could be useful, especially taking into account that this interval (30 seconds) is not configurable by a user.
Additionally, I remember that the combination ("Multiple PROBLEM events generation" mode + time-based trigger functions) has never been supported; however, I can not find the confirmation in the documentation now (it's possible, this was in our discussions with Zabbix tech. support).
In any case, such modification should be documented in Release Notes, of course.

Thanks constantin.oshmyan! What you're saying makes sense. We'll discuss it internally and let you know.

We'll discuss it internally and let you know.

dimir, what are the results of these discussions? Are there any chances this feature could be implemented in the v7.0 (nearest LTS version), or will it be postponed for a few more years?

what are the results of these discussions? Are there any chances this feature could be implemented in the v7.0 (nearest LTS version), or will it be postponed for a few more years?

Oh, we can celebrate the 10-years anniversary of this ticket!
As I understand from the proposed patches, technically this issue could be solved simple enough (several lines of code).
I'm still surprised why it takes so long time...

What is the solution for this? I am facing same problem, once the alert is resolved, it sends multiple actions. version 7