[ZBX-20419] A deadlock occurred in lld worker (ids) Created: 2022 Jan 11  Updated: 2024 Apr 10  Resolved: 2022 Jul 28

Status: Closed
Project: ZABBIX BUGS AND ISSUES
Component/s: Server (S)
Affects Version/s: 5.0.18
Fix Version/s: 5.0.27rc1, 6.0.8rc1, 6.2.2rc1, 6.4.0alpha1, 6.4 (plan)

Type: Problem report Priority: Trivial
Reporter: Kim Jongkwon Assignee: Andrejs Kozlovs
Resolution: Fixed Votes: 1
Labels: deadlock
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: Text File LATEST_DETECTED_DEADLOCK.txt     Text File zabbix_server.log    
Issue Links:
Duplicate
Sub-task
part of ZBX-21218 A deadlock occurred in lld worker (ids) Closed
Team: Team C
Sprint: Sprint 84 (Jan 2022), Sprint 85 (Feb 2022), Sprint 86 (Mar 2022), Sprint 87 (Apr 2022), Sprint 88 (May 2022), Sprint 89 (Jun 2022), Sprint 90 (Jul 2022)
Story Points: 1

 Description   

Report for investigation:
Deadlock output below from the LLD Worker. (Zabbix 5.0.16)

   701:20220107:143417.284 [Z3005] query failed: [1213] Deadlock found when trying to get lock; try restarting transaction [update ids set nextid=nextid+2 where table_name='triggers' and field_name='triggerid']
   700:20220107:143417.298 [Z3005] query failed: [1213] Deadlock found when trying to get lock; try restarting transaction [update ids set nextid=nextid+2 where table_name='triggers' and field_name='triggerid']
   704:20220107:143417.301 [Z3005] query failed: [1213] Deadlock found when trying to get lock; try restarting transaction [update ids set nextid=nextid+4 where table_name='triggers' and field_name='triggerid']
   703:20220107:143417.305 [Z3005] query failed: [1213] Deadlock found when trying to get lock; try restarting transaction [update ids set nextid=nextid+4 where table_name='triggers' and field_name='triggerid']
   702:20220107:143417.362 [Z3005] query failed: [1213] Deadlock found when trying to get lock; try restarting transaction [update ids set nextid=nextid+1 where table_name='trigger_depends' and field_name='triggerdepid']

   702:20220107:143417.362 [Z3005] query failed: [1213] Deadlock found when trying to get lock; try restarting transaction [update ids set nextid=nextid+1 where table_name='trigger_depends' and field_name='triggerdepid']
   702:20220107:143417.362 slow query: 45.143978 sec, "update ids set nextid=nextid+1 where table_name='trigger_depends' and field_name='triggerdepid'"
zabbix_server [702]: ERROR [file and function: <db.c,DBget_nextid>, revision:422e00c66a, line:824] Something impossible has just happened.
   702:20220107:143417.365 === Backtrace: ===
   702:20220107:143417.366 13: /usr/sbin/zabbix_server: lld worker #3 [processed 31 LLD rules, idle 4.703067 sec during 5.107193 sec](zbx_backtrace+0x52) [0x563150702809]
   702:20220107:143417.366 12: /usr/sbin/zabbix_server: lld worker #3 [processed 31 LLD rules, idle 4.703067 sec during 5.107193 sec](+0x26ceab) [0x563150777eab]
   702:20220107:143417.366 11: /usr/sbin/zabbix_server: lld worker #3 [processed 31 LLD rules, idle 4.703067 sec during 5.107193 sec](DBget_maxid_num+0x112) [0x56315077804f]
   702:20220107:143417.366 10: /usr/sbin/zabbix_server: lld worker #3 [processed 31 LLD rules, idle 4.703067 sec during 5.107193 sec](+0x16124a) [0x56315066c24a]
   702:20220107:143417.366 9: /usr/sbin/zabbix_server: lld worker #3 [processed 31 LLD rules, idle 4.703067 sec during 5.107193 sec](lld_update_triggers+0x2e9) [0x56315066effe]
   702:20220107:143417.366 8: /usr/sbin/zabbix_server: lld worker #3 [processed 31 LLD rules, idle 4.703067 sec during 5.107193 sec](lld_process_discovery_rule+0x5a1) [0x563150646574]
   702:20220107:143417.366 7: /usr/sbin/zabbix_server: lld worker #3 [processed 31 LLD rules, idle 4.703067 sec during 5.107193 sec](+0x137823) [0x563150642823]
   702:20220107:143417.366 6: /usr/sbin/zabbix_server: lld worker #3 [processed 31 LLD rules, idle 4.703067 sec during 5.107193 sec](lld_worker_thread+0x36c) [0x563150642f74]
   702:20220107:143417.366 5: /usr/sbin/zabbix_server: lld worker #3 [processed 31 LLD rules, idle 4.703067 sec during 5.107193 sec](zbx_thread_start+0x3b) [0x563150712dd7]
   702:20220107:143417.366 4: /usr/sbin/zabbix_server: lld worker #3 [processed 31 LLD rules, idle 4.703067 sec during 5.107193 sec](MAIN_ZABBIX_ENTRY+0xfef) [0x563150556939]
   702:20220107:143417.366 3: /usr/sbin/zabbix_server: lld worker #3 [processed 31 LLD rules, idle 4.703067 sec during 5.107193 sec](daemon_start+0x303) [0x56315070234b]
   702:20220107:143417.366 2: /usr/sbin/zabbix_server: lld worker #3 [processed 31 LLD rules, idle 4.703067 sec during 5.107193 sec](main+0x309) [0x5631505558e7]
   702:20220107:143417.366 1: /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf3) [0x7f566fa700b3]
   702:20220107:143417.366 0: /usr/sbin/zabbix_server: lld worker #3 [processed 31 LLD rules, idle 4.703067 sec during 5.107193 sec](_start+0x2e) [0x56315055488e]

+Add information
For about 30 minutes before the deadlock occurred, there was some UI task of adding triggers and trigger prototypes.

More details are in the attachment files.

 Comments   
Comment by Andris Mednis [ 2022 Jan 26 ]

LATEST_DETECTED_DEADLOCK.txt looks very informative:

  • Process 1 has locked some records in 'triggers' table and tries to lock 'ids' table,
  • Process 2 has locked 'ids' table and tries to lock primary key in 'triggers' table.

Classic deadlock when locking is attempted in different order.

Thanks to vso for pointing out that this is a deadlock between server's lld_triggers_save() and frontend!

SQLs from frontend come like this:

2022-01-28 18:28:25.488 EET [282775] zabbix50@zabbix50 LOG:  statement: SELECT nextid FROM ids WHERE table_name='trigger_depends' AND field_name='triggerdepid' FOR UPDATE
2022-01-28 18:28:25.488 EET [282775] zabbix50@zabbix50 LOG:  statement: UPDATE ids SET nextid=31498 WHERE table_name='trigger_depends' AND field_name='triggerdepid'
2022-01-28 18:28:25.488 EET [282775] zabbix50@zabbix50 LOG:  statement: INSERT INTO trigger_depends (triggerid_down,triggerid_up,triggerdepid) VALUES ('20088','20097','31498')
2022-01-28 18:28:25.489 EET [282775] zabbix50@zabbix50 LOG:  statement: SELECT t.triggerid FROM triggers t WHERE t.templateid=20088
2022-01-28 18:28:25.489 EET [282775] zabbix50@zabbix50 LOG:  statement: SELECT nextid FROM ids WHERE table_name='trigger_depends' AND field_name='triggerdepid' FOR UPDATE
2022-01-28 18:28:25.489 EET [282775] zabbix50@zabbix50 LOG:  statement: UPDATE ids SET nextid=31499 WHERE table_name='trigger_depends' AND field_name='triggerdepid'
2022-01-28 18:28:25.489 EET [282775] zabbix50@zabbix50 LOG:  statement: INSERT INTO trigger_depends (triggerid_down,triggerid_up,triggerdepid) VALUES ('20092','20093','31499')
Comment by Andrejs Kozlovs [ 2022 Jul 28 ]

Available in:

Generated at Tue Jan 07 15:42:00 EET 2025 using Jira 9.12.4#9120004-sha1:625303b708afdb767e17cb2838290c41888e9ff0.