-
Type:
Problem report
-
Resolution: Unresolved
-
Priority:
Major
-
Affects Version/s: 7.0.31
-
Component/s: Server (S)
-
None
-
Environment:Zabbix server 7.0.23, MySQL 8.0.45, Ubuntu 22.04, StartEscalators=1 (default)
-
Sprint candidates
When a single problem event matches multiple actions, where one action has no recovery operations and another has recovery operations, and the event recovers shortly after it occurred, a deadlock can occur between the escalator and history syncer on the escalations table. As a result, the escalator transaction is rolled back and the same escalation step is executed again in the next cycle, which causes duplicate notifications.
Root cause:
Escalator performs all UPDATEs before all DELETEs within a single transaction:
UPDATE escalations ... WHERE escalationid=55560; -- higher id, updated DELETE FROM escalations WHERE escalationid=55559; -- lower id, deleted
Meanwhile history syncer issues r_eventid updates in ascending escalationid order:
UPDATE escalations SET r_eventid=... WHERE escalationid=55559; UPDATE escalations SET r_eventid=... WHERE escalationid=55560;
When the to-be-deleted id (55559) is smaller than the to-be-updated id (55560), lock acquisition order is reversed between the two processes, causing a deadlock.
history syncer: LOCK 55559 → waiting for 55560 escalator: LOCK 55560 → waiting for 55559 ← deadlock
This condition arises when a single event matches two actions where one has no recovery operations (escalation completed → deleted) and one has recovery operations, and the event recovers within a few seconds.
Evidence:
Zabbix server log:
3455214:20260928:010835.091 [Z3005] query failed: [1213] Deadlock found when trying to get lock; try restarting transaction [delete from escalations where escalationid=53201;
3455214:20261008:112611.352 [Z3005] query failed: [1213] Deadlock found when trying to get lock; try restarting transaction [delete from escalations where escalationid=55559;
MySQL SHOW ENGINE INNODB STATUS (2026-10-08 11:26:11):
*** (1) TRANSACTION (history syncer): update escalations set r_eventid=45442,nextcheck=0 where escalationid=55560 HOLDS LOCK on escalationid=55559, WAITING FOR escalationid=55560 *** (2) TRANSACTION (escalator): delete from escalations where escalationid=55559 HOLDS LOCK on escalationid=55560, WAITING FOR escalationid=55559 *** WE ROLL BACK TRANSACTION (2)
Relation to ZBX-14830:
ZBX-14830 addressed a similar deadlock in history syncer.
The present issue is on the escalator side: the UPDATE-before-DELETE ordering within one transaction produces the reverse lock order and reintroduces the deadlock in the described scenario.