ExportXMLWordPrintable

    • Type: Problem report
    • Resolution: Unresolved
    • Priority: Major
    • Affects Version/s: 7.0.31
    • Component/s: Server (S)
    • None
    • Environment:
      Zabbix server 7.0.23, MySQL 8.0.45, Ubuntu 22.04, StartEscalators=1 (default)
    • Sprint candidates

      When a single problem event matches multiple actions, where one action has no recovery operations and another has recovery operations, and the event recovers shortly after it occurred, a deadlock can occur between the escalator and history syncer on the escalations table. As a result, the escalator transaction is rolled back and the same escalation step is executed again in the next cycle, which causes duplicate notifications.

      Root cause:
      Escalator performs all UPDATEs before all DELETEs within a single transaction:

      UPDATE escalations ... WHERE escalationid=55560;  -- higher id, updated
      DELETE FROM escalations WHERE escalationid=55559; -- lower id, deleted
      

      Meanwhile history syncer issues r_eventid updates in ascending escalationid order:

      UPDATE escalations SET r_eventid=... WHERE escalationid=55559;
      UPDATE escalations SET r_eventid=... WHERE escalationid=55560;
      

      When the to-be-deleted id (55559) is smaller than the to-be-updated id (55560), lock acquisition order is reversed between the two processes, causing a deadlock.

      history syncer:   LOCK 55559  →  waiting for 55560
      escalator:        LOCK 55560  →  waiting for 55559   ← deadlock
      

      This condition arises when a single event matches two actions where one has no recovery operations (escalation completed → deleted) and one has recovery operations, and the event recovers within a few seconds.

      Evidence:

      Zabbix server log:

      3455214:20260928:010835.091 [Z3005] query failed: [1213] Deadlock found when trying to get lock; try restarting transaction [delete from escalations where escalationid=53201;
      3455214:20261008:112611.352 [Z3005] query failed: [1213] Deadlock found when trying to get lock; try restarting transaction [delete from escalations where escalationid=55559;

      MySQL SHOW ENGINE INNODB STATUS (2026-10-08 11:26:11):

      *** (1) TRANSACTION (history syncer):
      update escalations set r_eventid=45442,nextcheck=0 where escalationid=55560
      HOLDS LOCK on escalationid=55559, WAITING FOR escalationid=55560
      
      *** (2) TRANSACTION (escalator):
      delete from escalations where escalationid=55559
      HOLDS LOCK on escalationid=55560, WAITING FOR escalationid=55559
      
      *** WE ROLL BACK TRANSACTION (2)
      

      Relation to ZBX-14830:
      ZBX-14830 addressed a similar deadlock in history syncer.
      The present issue is on the escalator side: the UPDATE-before-DELETE ordering within one transaction produces the reverse lock order and reintroduces the deadlock in the described scenario.

            Assignee:
            Zabbix Development Team
            Reporter:
            Syusei YIn
            Team A
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

              Created:
              Updated: