-
Incident report
-
Resolution: Duplicate
-
Critical
-
None
-
3.0.3, 3.0.4
-
None
-
Debian
Once in a while MySQL needs restart when a security patch is installed but every time when MySQL is restarted Zabbix server terminates:
16584:20160919:132424.526 [Z3005] query failed: [2002] Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2) [select escalationid,actionid,triggerid,eventid,r_eventid,nextcheck,esc_step,status,itemid from escalations where itemid is not null order by actionid,triggerid,itemid,escalationid] 16584:20160919:132424.540 [Z3001] connection to database 'zabbix' failed: [2002] Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2) 16584:20160919:132424.572 Cannot connect to the database. Exiting... 16583:20160919:132424.592 [Z3005] query failed: [2006] MySQL server has gone away [insert into history (itemid,clock,ns,value) values (62544,1474255464,578564042,75.676449),(48084,1474255464,580184457,67.923016); ] 16583:20160919:132424.622 [Z3001] connection to database 'zabbix' failed: [2002] Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2) 16583:20160919:132424.641 Cannot connect to the database. Exiting... 16559:20160919:132424.674 [Z3005] query failed: [1053] Server shutdown in progress [select hostid,key_,state,evaltype,formula,error,lifetime from items where itemid=30264] 16559:20160919:132424.712 [Z3001] connection to database 'zabbix' failed: [2002] Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2) 16559:20160919:132424.714 Cannot connect to the database. Exiting... 16582:20160919:132424.802 item "Zabbix server:mysql.status[Bytes_sent]" became not supported: Received value [ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2)] is not suitable for value type [Numeric (float)] 16582:20160919:132424.802 [Z3005] query failed: [2006] MySQL server has gone away [update items set state=1,error='Received value [ERROR 2002 (HY000): Can\'t connect to local MySQL server through socket \'/var/run/mysqld/mysqld.sock\' (2)] is not suitable for value type [Numeric (float)]' where itemid=23784; ] 16580:20160919:132424.815 [Z3005] query failed: [2006] MySQL server has gone away [insert into history (itemid,clock,ns,value) values (57444,1474255464,812126377,9.089814); ] 16580:20160919:132424.828 [Z3001] connection to database 'zabbix' failed: [2002] Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2) 16582:20160919:132424.828 [Z3001] connection to database 'zabbix' failed: [2002] Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2) 16580:20160919:132424.828 Cannot connect to the database. Exiting... 16582:20160919:132424.828 Cannot connect to the database. Exiting... 16553:20160919:132424.935 One child process died (PID:16584,exitcode/signal:1). Exiting ... 16553:20160919:132426.974 [Z3001] connection to database 'zabbix' failed: [2002] Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2) 16553:20160919:132426.975 Cannot connect to the database. Exiting...
This problem was introduced with 3.0.x (previously 2.4 could survive restart of MySQL). Clearly Zabbix Server should keep trying to re-establish connection with database for some time, ideally with configurable timeout. At the moment Zabbix server gives up on DB connectivity too fast so any restart of database (even if it takes seconds) reliably knock down Zabbix server.
This issue may be somewhat related to ZBX-10753 (which discusses startup failure) or similar to ZBX-11004...
- duplicates
-
ZBX-10753 3.0 server on MySQL (at least) stops immediately when cannot connect to database on start
- Closed