[ZBX-14802] Creating a trigger is causing a deadlock Created: 2018 Sep 03  Updated: 2019 Feb 25  Resolved: 2018 Nov 12

Status: Closed
Project: ZABBIX BUGS AND ISSUES
Component/s: Server (S)
Affects Version/s: 3.4.12
Fix Version/s: None

Type: Incident report Priority: Major
Reporter: itamar levy Assignee: Unassigned
Resolution: Cannot Reproduce Votes: 0
Labels: crash, triggers
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: PNG File Screenshot_20190225_171852.png     Text File image-2018-09-03-18-07-04-011.png     Text File zabbix_server.log    

 Description   

Steps to reproduce:

I've created the following trigger:

({System test template:proc_info[xxxx.exe,wkset,avg].avg(5)}) +
({System test template:proc_info[xxxx.exe,wkset,avg].avg(5)}) + ({System test template:proc_info[xxxx.exe,wkset,avg].avg(5)}) + ({System test template:proc_info[xxxx.exewkset,avg].avg(5)}) > 409600 and {System test template:system.run[wmic datafile where name="C:\\Program Files\\xxxx\\xxxx.exe" get Version /value].str()}=0

which caused a deadlock. 

**Result:

The following error details appeared on the UI:

Cannot update trigger
Details * pg_query(): Query failed: ERROR: deadlock detected

  • DETAIL: Process 20868 waits for ShareLock on transaction 117755409; blocked by process 7265.
  • Process 7265 waits for ShareLock on transaction 117755327; blocked by process 20868.
  • HINT: See server log for query details. [triggers.php:352 → CFrontendApiWrapper->update() → CApiWrapper->__call() → CFrontendApiWrapper->callMethod() → CApiWrapper->callMethod() → CFrontendApiWrapper->callClientMethod() → CLocalApiClient->callMethod() → call_user_func_array() → CTrigger->update() → CTriggerGeneral->inherit() → CTriggerGeneral->inheritOnHost() → CTriggerGeneral->updateReal() → DB::update() → DBexecute() → pg_query() in include/db.inc.php:483]
  • Error in query [UPDATE triggers SET expression='({93316}) +
  • ({93317}) + ({93318}) + ({93319}) > 409600 and {93320}=0',recovery_expression='' WHERE triggerid='47440'] [ERROR: deadlock detected
  • DETAIL: Process 20868 waits for ShareLock on transaction 117755409; blocked by process 7265.
  • Process 7265 waits for ShareLock on transaction 117755327; blocked by process 20868.
  • HINT: See server log for query details.]
  • SQL statement execution has failed "UPDATE triggers SET expression='({93316}) +
  • ({93317}) + ({93318}) + ({93319}) > 409600 and {93320}=0',recovery_expression='' WHERE triggerid='47440'".
     

Zabbix log:

**Please let me know if you need any additional information.

Expected:
See screenshot....
See attached patch file...



 Comments   
Comment by itamar levy [ 2018 Sep 03 ]

I've just realized that I might be missing brackets after the AND, I added the brackets but I still get the same error:

 

({System test template:proc_info[cytray.exe,wkset,avg].avg(5)}) +
({System test template:proc_info[tlaservice.exe,wkset,avg].avg(5)}) + ({System test template:proc_info[cyserver.exe,wkset,avg].avg(5)}) + ({System test template:proc_info[CyveraService.exe,wkset,avg].avg(5)}) > 409600 and ({System test template:system.run[wmic datafile where name="C:\\Program Files\\Palo Alto Networks\\Traps\\tlaservice.exe" get Version /value].str()}=0*)*

Comment by Vladislavs Sokurenko [ 2018 Sep 03 ]

Thank you for your report, it will be fixed under ZBX-14614

Comment by itamar levy [ 2018 Sep 16 ]

Hi Vladislavs,

I've looked at defect ZBX 14614 and it seems to have been resolved and pushed into 3.4.14

I upgraded my environment to 3.4.14 and the deadlock still exists, here is the error message from 3.4.14:

Cannot update trigger
Details * pg_query(): Query failed: ERROR: deadlock detected

  • DETAIL: Process 64685 waits for ShareLock on transaction 2929522; blocked by process 32414.
  • Process 32414 waits for ShareLock on transaction 2929338; blocked by process 64685.
  • HINT: See server log for query details.
  • CONTEXT: while updating tuple (384,5) in relation "triggers" [triggers.php:352 → CFrontendApiWrapper->update() → CApiWrapper->__call() → CFrontendApiWrapper->callMethod() → CApiWrapper->callMethod() → CFrontendApiWrapper->callClientMethod() → CLocalApiClient->callMethod() → call_user_func_array() → CTrigger->update() → CTriggerGeneral->inherit() → CTriggerGeneral->inherit() → CTriggerGeneral->inheritOnHost() → CTriggerGeneral->updateReal() → DB::update() → DBexecute() → pg_query() in include/db.inc.php:483]
  • Error in query [UPDATE triggers SET expression='({95973}) +
  • ({95974}) + ({95975}) + ({95976}) > 409600 and {95977}=0',recovery_expression='' WHERE triggerid='52851'] [ERROR: deadlock detected
  • DETAIL: Process 64685 waits for ShareLock on transaction 2929522; blocked by process 32414.
  • Process 32414 waits for ShareLock on transaction 2929338; blocked by process 64685.
  • HINT: See server log for query details.
  • CONTEXT: while updating tuple (384,5) in relation "triggers"]
  • SQL statement execution has failed "UPDATE triggers SET expression='({95973}) +
  • ({95974}) + ({95975}) + ({95976}) > 409600 and {95977}=0',recovery_expression='' WHERE triggerid='52851'".
Comment by Vladislavs Sokurenko [ 2018 Sep 16 ]

Could you please be so kind and provide Zabbix server log from new version once again ? This is very interesting

Comment by itamar levy [ 2018 Sep 16 ]

Log attached as requested, I see no correlation to the deadlock that appeared in the UI (I've added a comment in the log before reproducing the issue). Let me know if you want me to reproduce again with higher log level. 

Comment by Vladislavs Sokurenko [ 2018 Sep 17 ]

Deadlock from Zabbix server log:

4473:20180916:144222.829 [Z3005] query failed: [0] PGRES_FATAL_ERROR:ERROR:  deadlock detected
DETAIL:  Process 32413 waits for ShareLock on transaction 3125909; blocked by process 85304.
Process 85304 waits for ShareLock on transaction 3126135; blocked by process 32413.
HINT:  See server log for query details.
CONTEXT:  while updating tuple (579,45) in relation "triggers"
 [update triggers set state=0,error='' where triggerid=49616;
update triggers set state=0,error='' where triggerid=50126;
update triggers set state=0,error='' where triggerid=53441;
update triggers set state=0,error='' where triggerid=56761;
update triggers set state=0,error='' where triggerid=60194;
update triggers set state=1,error='Cannot evaluate expression: "Cannot evaluate function "A19891S20082:proc_info[cyserver.exe,wkset,avg].avg(5)".".' where triggerid=60517;
update triggers set state=1,error='Cannot evaluate expression: "Cannot evaluate function "A09973WN7X64:proc_info[cyserver.exe,wkset,avg].avg(5)".".' where triggerid=60857;
update triggers set state=0,error='' where triggerid=60874;
update triggers set state=0,error='' where triggerid=61214;
update triggers set state=0,error='' where triggerid=61554;
]

Is there anything insteresting in PostgreSQL log ?

HINT:  See server log for query details

Did you do any backups during those deadlocks ? Or is it deadlock between Zabbix server and Zabbix frontend ?

Comment by Edgars Melveris [ 2018 Oct 22 ]

Hello itamar!

Was this problem fixed for you? If this was not a Zabbix bug, we would appreciate info that it's not.

If it is, can you please provide the PostgreSQL log, that Vladislavs asked for?

Comment by itamar levy [ 2018 Oct 25 ]

HI @zux

Yes, the issue still reproduces. I will try to get the logs this or next week as I currently have no access to this DB. 

 

Comment by Edgars Melveris [ 2018 Nov 12 ]

Hello itamar!
Did you manage to get those logs? If not, we will have to close this issue.

Comment by itamar levy [ 2018 Nov 12 ]

Hi and sorry for the late reply, 

I am happy to say that the issue stopped reproducing!

I have found a workaround while it was still reproducing that is working well, I basically rotated the condition and got the string name at the beginning of the statement while summing up process memory after. I now tried to reproduce it to get the needed logs and the trigger was created successfully with no issues. 

Thanks for your help!!!

Comment by Edgars Melveris [ 2018 Nov 12 ]

Good to hear.

Comment by Dzianis Bazhok [ 2019 Feb 25 ]

We have the same problem but on updating existing trigger.

Zabbix Server: 4.0.4

PostgreSQL: 9.6

 

Part of PostgreSQL log:

2019-02-25 14:13:30 UTC [642]: [10-1] db=zabbix,user=postgres,app=[unknown],client=127.0.0.1 LOG: process 642 detected deadlock while waiting for ShareLock on transaction 1176 853916 after 1000.092 ms 2019-02-25 14:13:30 UTC [642]: [11-1] db=zabbix,user=postgres,app=[unknown],client=127.0.0.1 DETAIL: Process holding the lock: 773. Wait queue: . 2019-02-25 14:13:30 UTC [642]: [12-1] db=zabbix,user=postgres,app=[unknown],client=127.0.0.1 CONTEXT: while updating tuple (524,35) in relation "triggers" 2019-02-25 14:13:30 UTC [642]: [13-1] db=zabbix,user=postgres,app=[unknown],client=127.0.0.1 STATEMENT: UPDATE triggers SET expression='{615165} > {$SYSTEM_CPU_LOAD_PERCPU_AVG 5} and {615166}>0',recovery_expression='{615167} < {$SYSTEM_CPU_LOAD_PERCPU_AVG5}',recovery_mode='1' WHERE triggerid='161786' 2019-02-25 14:13:30 UTC [642]: [14-1] db=zabbix,user=postgres,app=[unknown],client=127.0.0.1 ERROR: deadlock detected 2019-02-25 14:13:30 UTC [642]: [15-1] db=zabbix,user=postgres,app=[unknown],client=127.0.0.1 DETAIL: Process 642 waits for ShareLock on transaction 1176853916; blocked by proc ess 773. Process 773 waits for ShareLock on transaction 1176853831; blocked by process 642. Process 642: UPDATE triggers SET expression='{615165} > {$SYSTEM_CPU_LOAD_PERCPU_AVG5} and {615166}>0',recovery_expression='{615167} < {$SYSTEM_CPU_LOAD_PERCPU_AVG5}',r ecovery_mode='1' WHERE triggerid='161786' Process 773: update triggers set state=0,error='' where triggerid=154586;

Screenshot from Zabbix Frontend:

Generated at Mon Jun 08 04:21:08 EEST 2026 using Jira 10.3.18#10030018-sha1:5642e4ad348b6c2a83ebdba689d04763a2393cab.