-
Incident report
-
Resolution: Duplicate
-
Minor
-
None
-
2.0.2
-
RHEL 5.6 x64
Percona MySQL 5.5
We have an interesting situation where notification emails are being sent for a trigger that was acknowledged even though the action is defined to only send notifications for non-acknowledged triggers (see attached screenshot). We know 100% for sure that this action is good, because we've been using it exactly as it is configured for the last 3 months. Here's the chain of events that happened.
1. A maintenance window was defined for a 12 hour window.
2. During that window, four triggers went off (as was expected).
3. The triggers were then acknowledged by a user.
4. Before the scheduled maintenance window had expired, the work that had necessitated the maintenance window was completed.
5. Due to #4, the decision was made to delete the maintenance window before it expired (since all the work had been done).
6. Once the maintenance window was deleted, the four triggers in #2 started sending out notifications that they had problems. This made no sense, because they had already been acknowledged in #3.
7. Those notifications have continued to be sent every hour (as specified in the interval for the action) even though the triggers were acknowledged.
As a part of trying to figure out what was going on, I did some queries in the DB (MySQL) and found something weird. First, I ran this query:
SELECT triggers.`description`, escalations.* FROM `triggers` INNER JOIN escalations ON escalations.`triggerid`=triggers.`triggerid` WHERE triggers.`triggerid` IN (SELECT triggerid FROM escalations);
That query listed all the current escalations and the trigger names to go along with them. This was done to verify that the escalations were in fact the ones we thought. After that, I ran:
SELECT * FROM acknowledges a INNER JOIN escalations e ON a.eventid=e.`eventid`;
That returned all the acknowledgments for the running escalations (all of which had acknowledgments on the Zabbix dashboard). The really weird thing about the returned rows was that the acknowledgments for the triggers sending notifications did NOT show up at all in the results. I then found through other queries that the acknowledgments we were seeing on the dashboard were associated with a different event.
Perhaps I'm misunderstanding how the system should work, but this sounds like a bug to me. Is it possible that a trigger being acknowledged during a maintenance window that is later deleted causes actions to not see the acknowledgments?
- duplicates
-
ZBX-5523 Acknowledgement doesn't stop actions from being processed after maintenance ended
- Closed