-
Incident report
-
Resolution: Unresolved
-
Trivial
-
None
-
5.2.6
-
Debian 10, PostgreSQL(12.6)+TimescaleDB (2.1.0)
Steps to reproduce:
- After server cold restart.
Result:
Errors about lld worker and duplicate key value (unique constraint "trigger_depends_1")
From zabbix_server.log:
zabbix_server [767]: ERROR [file and function: <lld_trigger.c,lld_trigger_changed>, revision:798506596c] Something impossible has just happened. 767:20210330:180810.836 === Backtrace: === 767:20210330:180810.836 12: /usr/sbin/zabbix_server: lld worker #2 [processed 8 LLD rules, idle 5.975842 sec during 6.964571 sec](zbx_backtrace+0x3f) [0x55f863484db9] 767:20210330:180810.836 11: /usr/sbin/zabbix_server: lld worker #2 [processed 8 LLD rules, idle 5.975842 sec during 6.964571 sec](+0x154a00) [0x55f8633f5a00] 767:20210330:180810.836 10: /usr/sbin/zabbix_server: lld worker #2 [processed 8 LLD rules, idle 5.975842 sec during 6.964571 sec](+0x154e40) [0x55f8633f5e40] 767:20210330:180810.836 9: /usr/sbin/zabbix_server: lld worker #2 [processed 8 LLD rules, idle 5.975842 sec during 6.964571 sec](lld_update_triggers+0x200) [0x55f8633f9cb3] 767:20210330:180810.836 8: /usr/sbin/zabbix_server: lld worker #2 [processed 8 LLD rules, idle 5.975842 sec during 6.964571 sec](lld_process_discovery_rule+0x4f8) [0x55f8633d1657] 767:20210330:180810.836 7: /usr/sbin/zabbix_server: lld worker #2 [processed 8 LLD rules, idle 5.975842 sec during 6.964571 sec](+0x12d2a2) [0x55f8633ce2a2] 767:20210330:180810.836 6: /usr/sbin/zabbix_server: lld worker #2 [processed 8 LLD rules, idle 5.975842 sec during 6.964571 sec](lld_worker_thread+0x2f6) [0x55f8633ce8fd] 767:20210330:180810.836 5: /usr/sbin/zabbix_server: lld worker #2 [processed 8 LLD rules, idle 5.975842 sec during 6.964571 sec](zbx_thread_start+0x37) [0x55f863494151] 767:20210330:180810.836 4: /usr/sbin/zabbix_server: lld worker #2 [processed 8 LLD rules, idle 5.975842 sec during 6.964571 sec](MAIN_ZABBIX_ENTRY+0x1098) [0x55f8632eed78] 767:20210330:180810.836 3: /usr/sbin/zabbix_server: lld worker #2 [processed 8 LLD rules, idle 5.975842 sec during 6.964571 sec](daemon_start+0x2ff) [0x55f8634849c5] 767:20210330:180810.836 2: /usr/sbin/zabbix_server: lld worker #2 [processed 8 LLD rules, idle 5.975842 sec during 6.964571 sec](main+0x2f5) [0x55f8632edc95] 767:20210330:180810.836 1: /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xeb) [0x7f902ad6809b] 767:20210330:180810.836 0: /usr/sbin/zabbix_server: lld worker #2 [processed 8 LLD rules, idle 5.975842 sec during 6.964571 sec](_start+0x2a) [0x55f8632eccca] zabbix_server [767]: ERROR [file and function: <lld_trigger.c,lld_trigger_changed>, revision:798506596c] Something impossible has just happened. 767:20210330:180810.836 === Backtrace: === 767:20210330:180810.837 12: /usr/sbin/zabbix_server: lld worker #2 [processed 8 LLD rules, idle 5.975842 sec during 6.964571 sec](zbx_backtrace+0x3f) [0x55f863484db9] 767:20210330:180810.837 11: /usr/sbin/zabbix_server: lld worker #2 [processed 8 LLD rules, idle 5.975842 sec during 6.964571 sec](+0x154a00) [0x55f8633f5a00] 767:20210330:180810.837 10: /usr/sbin/zabbix_server: lld worker #2 [processed 8 LLD rules, idle 5.975842 sec during 6.964571 sec](+0x154e40) [0x55f8633f5e40] 767:20210330:180810.837 9: /usr/sbin/zabbix_server: lld worker #2 [processed 8 LLD rules, idle 5.975842 sec during 6.964571 sec](lld_update_triggers+0x200) [0x55f8633f9cb3] 767:20210330:180810.837 8: /usr/sbin/zabbix_server: lld worker #2 [processed 8 LLD rules, idle 5.975842 sec during 6.964571 sec](lld_process_discovery_rule+0x4f8) [0x55f8633d1657] 767:20210330:180810.837 7: /usr/sbin/zabbix_server: lld worker #2 [processed 8 LLD rules, idle 5.975842 sec during 6.964571 sec](+0x12d2a2) [0x55f8633ce2a2] 767:20210330:180810.837 6: /usr/sbin/zabbix_server: lld worker #2 [processed 8 LLD rules, idle 5.975842 sec during 6.964571 sec](lld_worker_thread+0x2f6) [0x55f8633ce8fd] 767:20210330:180810.837 5: /usr/sbin/zabbix_server: lld worker #2 [processed 8 LLD rules, idle 5.975842 sec during 6.964571 sec](zbx_thread_start+0x37) [0x55f863494151] 767:20210330:180810.837 4: /usr/sbin/zabbix_server: lld worker #2 [processed 8 LLD rules, idle 5.975842 sec during 6.964571 sec](MAIN_ZABBIX_ENTRY+0x1098) [0x55f8632eed78] 767:20210330:180810.837 3: /usr/sbin/zabbix_server: lld worker #2 [processed 8 LLD rules, idle 5.975842 sec during 6.964571 sec](daemon_start+0x2ff) [0x55f8634849c5] 767:20210330:180810.837 2: /usr/sbin/zabbix_server: lld worker #2 [processed 8 LLD rules, idle 5.975842 sec during 6.964571 sec](main+0x2f5) [0x55f8632edc95] 767:20210330:180810.837 1: /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xeb) [0x7f902ad6809b] 767:20210330:180810.837 0: /usr/sbin/zabbix_server: lld worker #2 [processed 8 LLD rules, idle 5.975842 sec during 6.964571 sec](_start+0x2a) [0x55f8632eccca] zabbix_server [767]: ERROR [file and function: <lld_trigger.c,lld_trigger_changed>, revision:798506596c] Something impossible has just happened. 767:20210330:180810.837 === Backtrace: === 767:20210330:180810.837 12: /usr/sbin/zabbix_server: lld worker #2 [processed 8 LLD rules, idle 5.975842 sec during 6.964571 sec](zbx_backtrace+0x3f) [0x55f863484db9] 767:20210330:180810.837 11: /usr/sbin/zabbix_server: lld worker #2 [processed 8 LLD rules, idle 5.975842 sec during 6.964571 sec](+0x154a00) [0x55f8633f5a00] 767:20210330:180810.837 10: /usr/sbin/zabbix_server: lld worker #2 [processed 8 LLD rules, idle 5.975842 sec during 6.964571 sec](+0x154e40) [0x55f8633f5e40] 767:20210330:180810.837 9: /usr/sbin/zabbix_server: lld worker #2 [processed 8 LLD rules, idle 5.975842 sec during 6.964571 sec](lld_update_triggers+0x200) [0x55f8633f9cb3] 767:20210330:180810.837 8: /usr/sbin/zabbix_server: lld worker #2 [processed 8 LLD rules, idle 5.975842 sec during 6.964571 sec](lld_process_discovery_rule+0x4f8) [0x55f8633d1657] 767:20210330:180810.837 7: /usr/sbin/zabbix_server: lld worker #2 [processed 8 LLD rules, idle 5.975842 sec during 6.964571 sec](+0x12d2a2) [0x55f8633ce2a2] 767:20210330:180810.837 6: /usr/sbin/zabbix_server: lld worker #2 [processed 8 LLD rules, idle 5.975842 sec during 6.964571 sec](lld_worker_thread+0x2f6) [0x55f8633ce8fd] 767:20210330:180810.837 5: /usr/sbin/zabbix_server: lld worker #2 [processed 8 LLD rules, idle 5.975842 sec during 6.964571 sec](zbx_thread_start+0x37) [0x55f863494151] 767:20210330:180810.837 4: /usr/sbin/zabbix_server: lld worker #2 [processed 8 LLD rules, idle 5.975842 sec during 6.964571 sec](MAIN_ZABBIX_ENTRY+0x1098) [0x55f8632eed78] 767:20210330:180810.837 3: /usr/sbin/zabbix_server: lld worker #2 [processed 8 LLD rules, idle 5.975842 sec during 6.964571 sec](daemon_start+0x2ff) [0x55f8634849c5] 767:20210330:180810.837 2: /usr/sbin/zabbix_server: lld worker #2 [processed 8 LLD rules, idle 5.975842 sec during 6.964571 sec](main+0x2f5) [0x55f8632edc95] 767:20210330:180810.837 1: /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xeb) [0x7f902ad6809b] 767:20210330:180810.837 0: /usr/sbin/zabbix_server: lld worker #2 [processed 8 LLD rules, idle 5.975842 sec during 6.964571 sec](_start+0x2a) [0x55f8632eccca] zabbix_server [767]: ERROR [file and function: <lld_trigger.c,lld_trigger_changed>, revision:798506596c] Something impossible has just happened. 767:20210330:180810.837 === Backtrace: === 767:20210330:180810.838 12: /usr/sbin/zabbix_server: lld worker #2 [processed 8 LLD rules, idle 5.975842 sec during 6.964571 sec](zbx_backtrace+0x3f) [0x55f863484db9] 767:20210330:180810.838 11: /usr/sbin/zabbix_server: lld worker #2 [processed 8 LLD rules, idle 5.975842 sec during 6.964571 sec](+0x154a00) [0x55f8633f5a00] 767:20210330:180810.838 10: /usr/sbin/zabbix_server: lld worker #2 [processed 8 LLD rules, idle 5.975842 sec during 6.964571 sec](+0x154e40) [0x55f8633f5e40] 767:20210330:180810.838 9: /usr/sbin/zabbix_server: lld worker #2 [processed 8 LLD rules, idle 5.975842 sec during 6.964571 sec](lld_update_triggers+0x200) [0x55f8633f9cb3] 767:20210330:180810.838 8: /usr/sbin/zabbix_server: lld worker #2 [processed 8 LLD rules, idle 5.975842 sec during 6.964571 sec](lld_process_discovery_rule+0x4f8) [0x55f8633d1657] 767:20210330:180810.838 7: /usr/sbin/zabbix_server: lld worker #2 [processed 8 LLD rules, idle 5.975842 sec during 6.964571 sec](+0x12d2a2) [0x55f8633ce2a2] 767:20210330:180810.838 6: /usr/sbin/zabbix_server: lld worker #2 [processed 8 LLD rules, idle 5.975842 sec during 6.964571 sec](lld_worker_thread+0x2f6) [0x55f8633ce8fd] 767:20210330:180810.838 5: /usr/sbin/zabbix_server: lld worker #2 [processed 8 LLD rules, idle 5.975842 sec during 6.964571 sec](zbx_thread_start+0x37) [0x55f863494151] 767:20210330:180810.838 4: /usr/sbin/zabbix_server: lld worker #2 [processed 8 LLD rules, idle 5.975842 sec during 6.964571 sec](MAIN_ZABBIX_ENTRY+0x1098) [0x55f8632eed78] 767:20210330:180810.838 3: /usr/sbin/zabbix_server: lld worker #2 [processed 8 LLD rules, idle 5.975842 sec during 6.964571 sec](daemon_start+0x2ff) [0x55f8634849c5] 767:20210330:180810.838 2: /usr/sbin/zabbix_server: lld worker #2 [processed 8 LLD rules, idle 5.975842 sec during 6.964571 sec](main+0x2f5) [0x55f8632edc95] 767:20210330:180810.838 1: /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xeb) [0x7f902ad6809b] 767:20210330:180810.838 0: /usr/sbin/zabbix_server: lld worker #2 [processed 8 LLD rules, idle 5.975842 sec during 6.964571 sec](_start+0x2a) [0x55f8632eccca] 767:20210330:180810.847 [Z3005] query failed: [0] PGRES_FATAL_ERROR:ERROR: duplicate key value violates unique constraint "trigger_depends_1"DETAIL: Key (triggerid_down, triggerid_up)=(38731, 38623) already exists. [insert into trigger_depends (triggerdepid,triggerid_down,triggerid_up) values (36359,38731,38623),(36360,38731,38623),(36361,38732,38623),(36362,38733,38623),(36363,38734,38623);] 766:20210330:180810.872 [Z3005] query failed: [0] PGRES_FATAL_ERROR:ERROR: duplicate key value violates unique constraint "trigger_depends_1"DETAIL: Key (triggerid_down, triggerid_up)=(47330, 47336) already exists. [insert into trigger_depends (triggerdepid,triggerid_down,triggerid_up) values (36359,47330,47336),(36360,47330,47336),(36361,47331,47337),(36362,47332,47338),(36363,47336,40678),(36364,47337,40679),(36365,47338,40680);
From postgresql-12-main.log:
2021-03-30 18:08:10.847 UTC [775] zabbix@zabbix ERROR: duplicate key value violates unique constraint "trigger_depends_1" 2021-03-30 18:08:10.847 UTC [775] zabbix@zabbix DETAIL: Key (triggerid_down, triggerid_up)=(38731, 38623) already exists. 2021-03-30 18:08:10.847 UTC [775] zabbix@zabbix STATEMENT: insert into trigger_depends (triggerdepid,triggerid_down,triggerid_up) values (36359,38731,38623),(36360,38731,38623),(36361,38732,38623),(36362,38733,38623),(36363,38734,38623); 2021-03-30 18:08:10.872 UTC [776] zabbix@zabbix ERROR: duplicate key value violates unique constraint "trigger_depends_1" 2021-03-30 18:08:10.872 UTC [776] zabbix@zabbix DETAIL: Key (triggerid_down, triggerid_up)=(47330, 47336) already exists. 2021-03-30 18:08:10.872 UTC [776] zabbix@zabbix STATEMENT: insert into trigger_depends (triggerdepid,triggerid_down,triggerid_up) values (36359,47330,47336),(36360,47330,47336),(36361,47331,47337),(36362,47332,47338),(36363,47336,40678),(36364,47337,40679),(36365,47338,40680);