-
Problem report
-
Resolution: Fixed
-
Major
-
3.0.2
-
Team A
-
Sprint 50 (Mar 2019), Sprint 51 (Apr 2019)
-
1
This log file is part from a production, when MySQL server got a WSREP (Write Set REPlication) error. Zabbix server termination not on start but on run time:
11126:20160505:030001.537 [Z3005] query failed: [1213] Deadlock found when trying to get lock; try restarting transaction [commit;] 11125:20160505:030001.572 [Z3005] query failed: [1213] Deadlock found when trying to get lock; try restarting transaction [insert into history_uint (itemid,clock,ns,value) values (28561,1462435201,190482764,1462435201),(114841,1462435201,207893842,0),(253081,1462435201,209905590,0), trimmed ..... ); ] 11126:20160505:030002.054 [Z3005] query failed: [2013] Lost connection to MySQL server during query [commit;] 11125:20160505:030002.054 [Z3005] query failed: [2013] Lost connection to MySQL server during query [commit;] 11126:20160505:030002.059 [Z3001] connection to database 'zabbix' failed: [2013] Lost connection to MySQL server at 'reading initial communication packet', system error: 104 11125:20160505:030002.059 [Z3001] connection to database 'zabbix' failed: [2013] Lost connection to MySQL server at 'reading initial communication packet', system error: 104 11126:20160505:030002.059 database is down: reconnecting in 10 seconds 11125:20160505:030002.059 database is down: reconnecting in 10 seconds ... 11126:20160505:030022.062 [Z3001] connection to database 'zabbix' failed: [2013] Lost connection to MySQL server at 'reading initial communication packet', system error: 104 11126:20160505:030022.062 database is down: reconnecting in 10 seconds 11128:20160505:030022.193 [Z3001] connection to database 'zabbix' failed: [1047] WSREP has not yet prepared node for application use 11128:20160505:030022.193 Cannot connect to the database. Exiting... 10964:20160505:030022.265 One child process died (PID:11128,exitcode/signal:1). Exiting ... 10964:20160505:030024.270 [Z3001] connection to database 'zabbix' failed: [1047] WSREP has not yet prepared node for application use 10964:20160505:030024.270 Cannot connect to the database. Exiting... 30929:20160505:030034.378 Starting Zabbix Server. Zabbix 3.0.2 (revision 59540). 30929:20160505:030034.378 ****** Enabled features ****** 30929:20160505:030034.379 SNMP monitoring: YES 30929:20160505:030034.379 IPMI monitoring: YES 30929:20160505:030034.379 Web monitoring: YES 30929:20160505:030034.379 VMware monitoring: YES 30929:20160505:030034.379 SMTP authentication: YES 30929:20160505:030034.379 Jabber notifications: YES 30929:20160505:030034.379 Ez Texting notifications: YES 30929:20160505:030034.379 ODBC: YES 30929:20160505:030034.379 SSH2 support: YES 30929:20160505:030034.379 IPv6 support: YES 30929:20160505:030034.379 TLS support: YES 30929:20160505:030034.379 ****************************** 30929:20160505:030034.379 using configuration file: /etc/zabbix/zabbix_server.conf 30929:20160505:030034.394 [Z3001] connection to database 'zabbix' failed: [1047] WSREP has not yet prepared node for application use 30929:20160505:030034.394 Cannot connect to the database. Exiting... 30936:20160505:030044.619 Starting Zabbix Server. Zabbix 3.0.2 (revision 59540). 30936:20160505:030044.619 ****** Enabled features ****** 30936:20160505:030044.619 SNMP monitoring: YES 30936:20160505:030044.619 IPMI monitoring: YES 30936:20160505:030044.619 Web monitoring: YES 30936:20160505:030044.619 VMware monitoring: YES 30936:20160505:030044.619 SMTP authentication: YES 30936:20160505:030044.619 Jabber notifications: YES 30936:20160505:030044.619 Ez Texting notifications: YES 30936:20160505:030044.619 ODBC: YES 30936:20160505:030044.619 SSH2 support: YES 30936:20160505:030044.619 IPv6 support: YES 30936:20160505:030044.619 TLS support: YES 30936:20160505:030044.619 ****************************** 30936:20160505:030044.619 using configuration file: /etc/zabbix/zabbix_server.conf 30936:20160505:030044.625 [Z3001] connection to database 'zabbix' failed: [1047] WSREP has not yet prepared node for application use 30936:20160505:030044.625 Cannot connect to the database. Exiting... ... restarts repeated here
note - server daemon started again automatically by systemd (by default).
The solution for the issue might be very similar to the issue in ZBX-10753 but most likely it has to do with communication error between DB replications not stopped MySQL.
