-
Problem report
-
Resolution: Won't fix
-
Blocker
-
None
-
5.0.7
-
None
-
Team A
-
Sprint 72 (Jan 2021)
MySQL cluster with mysql router is used in production, it works fine.
There is identical development/testing installation where something wrong happens -configuration syncer may "hang" in unknown/busy state.
On graphs it's marked as busy, when we connect by strace - it's silent.
Something happens after "query failed" message and for next SQL performed the config syncer "hangs".
581414:20201224:144531.194 [Z3005] query failed: [2000] Unknown MySQL error [select distinct t.triggerid,t.description,t.expression,t.error,t.priority,t.type,t.value,t.state,t.lastchange,t.status,t.recovery_mode,t.recovery_expression,t.correlation_mode,t.correlation_tag,opdata from hosts h,items i,functions f,triggers t where h.hostid=i.hostid and i.itemid=f.itemid and f.triggerid=t.triggerid and h.status in (0,1) and t.flags<>2] zabbix_server [581414]: ERROR [file:dbconfig.c,line:5713] Something impossible has just happened. 581414:20201224:144531.195 === Backtrace: === 581414:20201224:144531.197 8: /usr/sbin/zabbix_server: configuration syncer [synced configuration in 0.493910 sec, syncing configuration](zbx_backtrace+0x3f) [0x5612890aa016] 581414:20201224:144531.198 7: /usr/sbin/zabbix_server: configuration syncer [synced configuration in 0.493910 sec, syncing configuration](DCsync_configuration+0x2c42) [0x56128907a7f4] 581414:20201224:144531.198 6: /usr/sbin/zabbix_server: configuration syncer [synced configuration in 0.493910 sec, syncing configuration](dbconfig_thread+0x1b5) [0x561288f31539] 581414:20201224:144531.199 5: /usr/sbin/zabbix_server: configuration syncer [synced configuration in 0.493910 sec, syncing configuration](zbx_thread_start+0x37) [0x5612890b8d6d] 581414:20201224:144531.199 4: /usr/sbin/zabbix_server: configuration syncer [synced configuration in 0.493910 sec, syncing configuration](MAIN_ZABBIX_ENTRY+0x9cb) [0x561288f21cc1] 581414:20201224:144531.200 3: /usr/sbin/zabbix_server: configuration syncer [synced configuration in 0.493910 sec, syncing configuration](daemon_start+0x2ff) [0x5612890a9c22] 581414:20201224:144531.201 2: /usr/sbin/zabbix_server: configuration syncer [synced configuration in 0.493910 sec, syncing configuration](main+0x2f5) [0x561288f212ab] 581414:20201224:144531.201 1: /lib64/libc.so.6(__libc_start_main+0xf3) [0x7f47850e67b3] 581414:20201224:144531.202 0: /usr/sbin/zabbix_server: configuration syncer [synced configuration in 0.493910 sec, syncing configuration](_start+0x2e) [0x561288f2035e]
It may stay in such state many hour and than suddenly may un-hang itself (visible drop from 100% to 0%) for short period, until it again will not try to perform conf sync and hangs again on SQL.
Some grep (Starting|query failed|configuration syncer #):
581414:20210103:050330.674 [Z3005] query failed: [2013] Lost connection to MySQL server during query [select distinct t.triggerid,t.description,t.expression,t.error,t.priority,t.type,> 581414:20210103:050632.396 [Z3005] query failed: [2000] Unknown MySQL error [select distinct t.triggerid,t.description,t.expression,t.error,t.priority,t.type,t.value,t.state,t.lastcha> 581414:20210103:130732.497 [Z3005] query failed: [2013] Lost connection to MySQL server during query [select refresh_unsupported,discovery_groupid,snmptrap_logging,severity_name_0,sev> 581414:20210103:211034.585 [Z3005] query failed: [2013] Lost connection to MySQL server during query [select distinct tt.triggertagid,tt.triggerid,tt.tag,tt.value from trigger_tag tt,> 581414:20210104:051537.467 [Z3005] query failed: [2013] Lost connection to MySQL server during query [select distinct tt.triggertagid,tt.triggerid,tt.tag,tt.value from trigger_tag tt,> 581414:20210104:051839.037 [Z3005] query failed: [2000] Unknown MySQL error [select distinct t.triggerid,t.description,t.expression,t.error,t.priority,t.type,t.value,t.state,t.lastcha> 791745:20210104:114503.129 Starting Zabbix Server. Zabbix 5.0.7 (revision 1b3db7f980). 791746:20210104:114503.285 server #1 started [configuration syncer #1] 791746:20210104:201016.663 [Z3005] query failed: [2013] Lost connection to MySQL server during query [select distinct t.triggerid,t.description,t.expression,t.error,t.priority,t.type,> 791746:20210104:201016.902 [Z3005] query failed: [2000] Unknown MySQL error [select distinct tt.triggertagid,tt.triggerid,tt.tag,tt.value from trigger_tag tt,triggers t,hosts h,items > 791789:20210105:010028.463 [Z3005] query failed: [2000] Unknown MySQL error [select td.parent_triggerid,t.triggerid,t.description,t.expression,t.type,t.priority,t.comments,t.url,t.rec> 791746:20210105:041117.010 [Z3005] query failed: [2013] Lost connection to MySQL server during query [select refresh_unsupported,discovery_groupid,snmptrap_logging,severity_name_0,sev> 791789:20210105:090028.532 [Z3005] query failed: [2013] Lost connection to MySQL server during query [select functionid,triggerid,itemid,name,parameter from functions where triggerid > 791746:20210105:121318.590 [Z3005] query failed: [2013] Lost connection to MySQL server during query [select distinct t.triggerid,t.description,t.expression,t.error,t.priority,t.type,> 1296244:20210105:160143.816 Starting Zabbix Server. Zabbix 5.0.7 (revision 1b3db7f980). 1296245:20210105:160143.847 server #1 started [configuration syncer #1] 1297721:20210105:160536.880 Starting Zabbix Server. Zabbix 5.0.7 (revision 1b3db7f980). 1297722:20210105:160536.916 server #1 started [configuration syncer #1] 1297722:20210105:160738.423 [Z3005] query failed: [2000] Unknown MySQL error [select distinct t.triggerid,t.description,t.expression,t.error,t.priority,t.type,t.value,t.state,t.lastch> 1300712:20210105:161434.908 Starting Zabbix Server. Zabbix 5.0.7 (revision 1b3db7f980). 1300713:20210105:161434.943 server #1 started [configuration syncer #1] 1300713:20210105:161837.727 [Z3005] query failed: [2000] Unknown MySQL error [select distinct tt.triggertagid,tt.triggerid,tt.tag,tt.value from trigger_tag tt,triggers t,hosts h,items> 1304458:20210105:162602.910 Starting Zabbix Server. Zabbix 5.0.7 (revision 1b3db7f980). 1304459:20210105:162602.939 server #1 started [configuration syncer #1] 1304459:20210106:003825.754 [Z3005] query failed: [2013] Lost connection to MySQL server during query [select distinct tt.triggertagid,tt.triggerid,tt.tag,tt.value from trigger_tag tt> 1304459:20210106:004730.776 [Z3005] query failed: [2000] Unknown MySQL error [select distinct t.triggerid,t.description,t.expression,t.error,t.priority,t.type,t.value,t.state,t.lastch> 1304499:20210106:083028.919 [Z3005] query failed: [2013] Lost connection to MySQL server during query [select td.parent_triggerid,t.triggerid,t.description,t.expression,t.type,t.prior> 1304459:20210106:084830.867 [Z3005] query failed: [2013] Lost connection to MySQL server during query [select refresh_unsupported,discovery_groupid,snmptrap_logging,severity_name_0,se> 1304459:20210106:091044.015 [Z3005] query failed: [2000] Unknown MySQL error [select distinct t.triggerid,t.description,t.expression,t.error,t.priority,t.ty
Related graphs:
Then we restarted zabbix_server, quickly (after initial conf cync of course) increased log level for conf syncer to debug and attached by strace.
zabbix_server.log
conf-syncer.out
Not sure what happens and does mysql router work well on test installation, but I'm sure that config syncer should not "hang" in unknown state and do not perform syncs at all.
