Uploaded image for project: 'ZABBIX BUGS AND ISSUES'
  1. ZABBIX BUGS AND ISSUES
  2. ZBX-19880

MariaDB deadlock on closing problems

    XMLWordPrintable

Details

    • Problem report
    • Status: Open
    • Trivial
    • Resolution: Unresolved
    • None
    • None
    • None
    • None
    • CentOS 8, MariaDB 10.3, Zabbix 5.0.14

    Description

      I am using zabbix-server 5.0.14 on CentOS 8 with MariaDB.
      In my environment, zabbix-server works with other systems and I made some tools to close problems or to change the severity.

      I sometimes find that some problems are closing state for more than 1day.  Please see attached image.
      And the following logs are written in zabbix_server.log.

      4502:20210827:163439.404 [Z3005] query failed: [1213] Deadlock found when trying to get lock; try restarting transaction [update problem set r_eventid=232854,r_clock=1630049678,r_ns=94954861,userid=1 where eventid=172849;]
       4502:20210827:163439.724 [Z3005] query failed: [1213] Deadlock found when trying to get lock; try restarting transaction [insert into event_recovery (eventid,r_eventid,correlationid,c_eventid,userid) values (172850,232855,null,null,1);]
      

      It seems that DB deadlock occurs and it causes problem close failure.
      I attached the result of "show engine innodb status" in innodb.log. Please see "LATEST DETECTED DEADLOCK" section.

      Steps to reproduce:
      This is how I checked to reproduce the issue.
      1. create many problems (~30k problems)
      2. close all problems by API
      3. change problem severity by API repeatedly

      I used a python script for 2. and 3. Please see attached file close_problem.py.
      Repeating this procedure several times caused the problem reproduction.

      Result:
      DB deadlock occurs and some problems remains in closing state.

      Expected:
      All problems are closed.

      Attachments

        1. close_problem.py
          2 kB
        2. close_problem2.py
          2 kB
        3. closing_problem.png
          closing_problem.png
          12 kB
        4. innodb.log
          129 kB
        5. retry_closing_problem.patch
          2 kB

        Activity

          People

            asincovs Antons Sincovs
            kento.takahashi Kento Takahashi
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated: