[ZBX-14802] Creating a trigger is causing a deadlock Created: 2018 Sep 03 Updated: 2019 Feb 25 Resolved: 2018 Nov 12 |
|
| Status: | Closed |
| Project: | ZABBIX BUGS AND ISSUES |
| Component/s: | Server (S) |
| Affects Version/s: | 3.4.12 |
| Fix Version/s: | None |
| Type: | Incident report | Priority: | Major |
| Reporter: | itamar levy | Assignee: | Unassigned |
| Resolution: | Cannot Reproduce | Votes: | 0 |
| Labels: | crash, triggers | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Attachments: |
|
| Description |
|
Steps to reproduce: I've created the following trigger: ({System test template:proc_info[xxxx.exe,wkset,avg].avg(5)}) + which caused a deadlock. **Result: The following error details appeared on the UI: Cannot update trigger
Zabbix log:
**Please let me know if you need any additional information. Expected: |
| Comments |
| Comment by itamar levy [ 2018 Sep 03 ] |
|
I've just realized that I might be missing brackets after the AND, I added the brackets but I still get the same error:
({System test template:proc_info[cytray.exe,wkset,avg].avg(5)}) + |
| Comment by Vladislavs Sokurenko [ 2018 Sep 03 ] |
|
Thank you for your report, it will be fixed under |
| Comment by itamar levy [ 2018 Sep 16 ] |
|
Hi Vladislavs, I've looked at defect ZBX 14614 and it seems to have been resolved and pushed into 3.4.14 I upgraded my environment to 3.4.14 and the deadlock still exists, here is the error message from 3.4.14: Cannot update trigger
|
| Comment by Vladislavs Sokurenko [ 2018 Sep 16 ] |
|
Could you please be so kind and provide Zabbix server log from new version once again ? This is very interesting |
| Comment by itamar levy [ 2018 Sep 16 ] |
|
Log attached as requested, I see no correlation to the deadlock that appeared in the UI (I've added a comment in the log before reproducing the issue). Let me know if you want me to reproduce again with higher log level. |
| Comment by Vladislavs Sokurenko [ 2018 Sep 17 ] |
|
Deadlock from Zabbix server log: 4473:20180916:144222.829 [Z3005] query failed: [0] PGRES_FATAL_ERROR:ERROR: deadlock detected DETAIL: Process 32413 waits for ShareLock on transaction 3125909; blocked by process 85304. Process 85304 waits for ShareLock on transaction 3126135; blocked by process 32413. HINT: See server log for query details. CONTEXT: while updating tuple (579,45) in relation "triggers" [update triggers set state=0,error='' where triggerid=49616; update triggers set state=0,error='' where triggerid=50126; update triggers set state=0,error='' where triggerid=53441; update triggers set state=0,error='' where triggerid=56761; update triggers set state=0,error='' where triggerid=60194; update triggers set state=1,error='Cannot evaluate expression: "Cannot evaluate function "A19891S20082:proc_info[cyserver.exe,wkset,avg].avg(5)".".' where triggerid=60517; update triggers set state=1,error='Cannot evaluate expression: "Cannot evaluate function "A09973WN7X64:proc_info[cyserver.exe,wkset,avg].avg(5)".".' where triggerid=60857; update triggers set state=0,error='' where triggerid=60874; update triggers set state=0,error='' where triggerid=61214; update triggers set state=0,error='' where triggerid=61554; ] Is there anything insteresting in PostgreSQL log ?
HINT: See server log for query details
Did you do any backups during those deadlocks ? Or is it deadlock between Zabbix server and Zabbix frontend ? |
| Comment by Edgars Melveris [ 2018 Oct 22 ] |
|
Hello itamar! Was this problem fixed for you? If this was not a Zabbix bug, we would appreciate info that it's not. If it is, can you please provide the PostgreSQL log, that Vladislavs asked for? |
| Comment by itamar levy [ 2018 Oct 25 ] |
|
HI @zux Yes, the issue still reproduces. I will try to get the logs this or next week as I currently have no access to this DB.
|
| Comment by Edgars Melveris [ 2018 Nov 12 ] |
|
Hello itamar! |
| Comment by itamar levy [ 2018 Nov 12 ] |
|
Hi and sorry for the late reply, I am happy to say that the issue stopped reproducing! I have found a workaround while it was still reproducing that is working well, I basically rotated the condition and got the string name at the beginning of the statement while summing up process memory after. I now tried to reproduce it to get the needed logs and the trigger was created successfully with no issues. Thanks for your help!!! |
| Comment by Edgars Melveris [ 2018 Nov 12 ] |
|
Good to hear. |
| Comment by Dzianis Bazhok [ 2019 Feb 25 ] |
|
We have the same problem but on updating existing trigger. Zabbix Server: 4.0.4 PostgreSQL: 9.6
Part of PostgreSQL log: 2019-02-25 14:13:30 UTC [642]: [10-1] db=zabbix,user=postgres,app=[unknown],client=127.0.0.1 LOG: process 642 detected deadlock while waiting for ShareLock on transaction 1176 853916 after 1000.092 ms 2019-02-25 14:13:30 UTC [642]: [11-1] db=zabbix,user=postgres,app=[unknown],client=127.0.0.1 DETAIL: Process holding the lock: 773. Wait queue: . 2019-02-25 14:13:30 UTC [642]: [12-1] db=zabbix,user=postgres,app=[unknown],client=127.0.0.1 CONTEXT: while updating tuple (524,35) in relation "triggers" 2019-02-25 14:13:30 UTC [642]: [13-1] db=zabbix,user=postgres,app=[unknown],client=127.0.0.1 STATEMENT: UPDATE triggers SET expression='{615165} > {$SYSTEM_CPU_LOAD_PERCPU_AVG 5} and {615166}>0',recovery_expression='{615167} < {$SYSTEM_CPU_LOAD_PERCPU_AVG5}',recovery_mode='1' WHERE triggerid='161786' 2019-02-25 14:13:30 UTC [642]: [14-1] db=zabbix,user=postgres,app=[unknown],client=127.0.0.1 ERROR: deadlock detected 2019-02-25 14:13:30 UTC [642]: [15-1] db=zabbix,user=postgres,app=[unknown],client=127.0.0.1 DETAIL: Process 642 waits for ShareLock on transaction 1176853916; blocked by proc ess 773. Process 773 waits for ShareLock on transaction 1176853831; blocked by process 642. Process 642: UPDATE triggers SET expression='{615165} > {$SYSTEM_CPU_LOAD_PERCPU_AVG5} and {615166}>0',recovery_expression='{615167} < {$SYSTEM_CPU_LOAD_PERCPU_AVG5}',r ecovery_mode='1' WHERE triggerid='161786' Process 773: update triggers set state=0,error='' where triggerid=154586; Screenshot from Zabbix Frontend:
|