[ZBX-11402] Mail escalations wrongly sent on SNMP dependent triggers Created: 2016 Oct 27  Updated: 2024 Apr 10  Resolved: 2017 Feb 17

Status: Closed
Project: ZABBIX BUGS AND ISSUES
Component/s: Server (S)
Affects Version/s: 3.0.4
Fix Version/s: 3.0.8rc1, 3.2.4rc1, 3.4.0alpha1

Type: Incident report Priority: Major
Reporter: Piotr Ciechoński Assignee: Unassigned
Resolution: Fixed Votes: 2
Labels: triggers
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: PNG File actions.png     PNG File host_icmp_unavailable.png     PNG File triggerA.png     PNG File triggerB.png     PNG File triggerB_eventDetail.png    
Team: Team A
Sprint: Sprint 1
Story Points: 0.5

 Description   

There are two SNMP triggers on different SNMP items. Trigger B is dependent on Trigger A. Escalations are set to be sent immediately.
The situation is as follows:
21-10-2016 11:38:00 - Trigger B is triggered - Escalation is not sent initially, it is sent 22-10-2016 09:58:53 (which is the error behaviour)
21-10-2016 11:38:01 - Trigger A is triggered - Escalation is sent at 21-10-2016 11:38:03
22-10-2016 09:59:02 - Host is unreachable

In log there is an entry at 22-10-2016 09:58:51 that states: "temporarily disabling SNMP agent check on host".
It looks like the escalation of Trigger B was somehow stuck, because the Trigger A was triggered 1 second after and it was sent when the SNMP agent check on that host was disabled.

 Comments   
Comment by Aleksandrs Saveljevs [ 2016 Oct 31 ]

Could you please show some screenshots from the frontend (e.g., "Monitoring" -> "Events", event details, action configuration, and action log)?

Comment by Piotr Ciechoński [ 2016 Nov 10 ]

I have attached screenshots

Comment by Aleksandrs Saveljevs [ 2016 Dec 05 ]

Looking at process_escalations() and check_escalation_trigger() functions in escalator.c, it seems that escalations for dependent triggers are delayed. That is, if trigger B in the discussion above depends on trigger A and trigger A is in PROBLEM state, then escalation for trigger B is delayed. That would explain why notification for trigger B was not sent immediately, because even though it triggered before trigger A, escalations are processed in friendly bunches later (after events are generated and after escalations are initially inserted into the database), and at that time both triggers were PROBLEM.

The next question is why did it suddenly send a notification at 09:58:53 on October 22? My guess would be that trigger A came into an UNKNOWN state, because its host was unreachable. Since it was in an UNKNOWN state, it was not seen as "blocking" trigger B's escalation (see DCconfig_check_trigger_dependencies_rec() function in dbconfig.c), so trigger B's escalation was finally processed.

Comment by Aleksandrs Saveljevs [ 2016 Dec 05 ]

The above comment seems to explain the behavior, so changing issue state to "Open".

Comment by Viktors Tjarve [ 2017 Jan 12 ]

Fixed in development branch svn://svn.zabbix.com/branches/dev/ZBX-11402

Notifications will no longer be sent from dependent trigger (Trigger B) when it depends on another trigger (Trigger A) which was in PROBLEM state and the host of this trigger (Trigger A) is unreachable.

Comment by Alexander Vladishev [ 2017 Jan 13 ]

(1) Documentation

viktors.tjarve RESOLVED
Added discription to:

All check by martins-v

sasha

  1. changes from "Upgrade notes 3.0.0" should be moved into "Upgrade notes 3.0.8" and "Upgrade notes 3.2.4"
  2. "unreachable triggers" must be reworded into "triggers in UNKNOWN state"

REOPENED

viktors.tjarve RESOLVED please review VSI

  • Fixed terminology and moved comment in upgrade notes from 3.0.0 to 3.0.8 and 3.2.4
  • Fixed terminology in 'Trigger dependencies' 3.0, 3.2 and 3.4

VSI Looks like all requested changes are done.

CLOSED

Comment by Sergejs Paskevics [ 2017 Jan 19 ]

Successfully tested

Comment by Viktors Tjarve [ 2017 Jan 23 ]

Released in:

  • 3.0.8rc1 r65223
  • 3.2.4rc1 r65224
  • 3.3.0 r65225
Comment by Rostislav Palivoda [ 2017 Feb 13 ]

VSI please review documentation and mark as done or assign to me.

Generated at Fri Apr 26 01:16:59 EEST 2024 using Jira 9.12.4#9120004-sha1:625303b708afdb767e17cb2838290c41888e9ff0.