-
Problem report
-
Resolution: Fixed
-
Blocker
-
3.0.6
This zabbix server has many deadlock messages in log file, here for ~20 hours:
# grep "Lock wait timeout exceeded" zabbix_server_12_01.log 24383:20170111:184028.660 [Z3005] query failed: [1205] Lock wait timeout exceeded; try restarting transaction [update hosts set available=2,error='Get value from agent failed: cannot connect to [[[XX.XX.XX.XX]]:10050]: [113] No route to host' where hostid=49297; 24526:20170111:184459.097 [Z3005] query failed: [1205] Lock wait timeout exceeded; try restarting transaction [update hosts set available=2,error='Get value from agent failed: cannot connect to [[[XX.XX.XX.XX]]:10050]: [113] No route to host' where hostid=49423; 23921:20170111:184537.807 [Z3005] query failed: [1205] Lock wait timeout exceeded; try restarting transaction [update hosts set available=1,error='' where hostid=51567; 24050:20170111:184556.526 [Z3005] query failed: [1205] Lock wait timeout exceeded; try restarting transaction [update hosts set available=1,error='' where hostid=52995; 24804:20170111:184930.063 [Z3005] query failed: [1205] Lock wait timeout exceeded; try restarting transaction [update hosts set available=2,error='Get value from agent failed: cannot connect to [[[XX.XX.XX.XX]]:10050]: [113] No route to host' where hostid=48267; 24149:20170111:185331.951 [Z3005] query failed: [1205] Lock wait timeout exceeded; try restarting transaction [update hosts set available=2,error='Get value from agent failed: cannot connect to [[[XX.XX.XX.XX]]:10050]: [113] No route to host' where hostid=59019; 24122:20170111:185421.810 [Z3005] query failed: [1205] Lock wait timeout exceeded; try restarting transaction [update hosts set available=2,error='Get value from agent failed: cannot connect to [[[XX.XX.XX.XX]]:10050]: [113] No route to host' where hostid=50982; 24721:20170111:185533.073 [Z3005] query failed: [1205] Lock wait timeout exceeded; try restarting transaction [update hosts set error='Get value from agent failed: cannot connect to [[[XX.XX.XX.XX]]:10050]: [113] No route to host' where hostid=63192; 24288:20170111:185539.271 [Z3005] query failed: [1205] Lock wait timeout exceeded; try restarting transaction [update hosts set available=2,error='Get value from agent failed: cannot connect to [[[XX.XX.XX.XX]]:10050]: [113] No route to host' where hostid=62481; 24797:20170111:185604.309 [Z3005] query failed: [1205] Lock wait timeout exceeded; try restarting transaction [update hosts set available=2,error='Get value from agent failed: cannot connect to [[[XX.XX.XX.XX]]:10050]: [113] No route to host' where hostid=58739; 24705:20170111:185650.512 [Z3005] query failed: [1205] Lock wait timeout exceeded; try restarting transaction [update hosts set available=2,error='Get value from agent failed: cannot connect to [[[XX.XX.XX.XX]]:10050]: [113] No route to host' where hostid=64196; 24444:20170111:185709.946 [Z3005] query failed: [1205] Lock wait timeout exceeded; try restarting transaction [update hosts set available=2,error='Get value from agent failed: cannot connect to [[[XX.XX.XX.XX]]:10050]: [113] No route to host' where hostid=52145; 23878:20170111:185714.509 [Z3005] query failed: [1205] Lock wait timeout exceeded; try restarting transaction [update hosts set available=1,error='' where hostid=65274; 23879:20170111:185917.687 [Z3005] query failed: [1205] Lock wait timeout exceeded; try restarting transaction [update hosts set available=2,error='Get value from agent failed: cannot connect to [[[XX.XX.XX.XX]]:10050]: [113] No route to host' where hostid=61593; 23756:20170111:185920.685 [Z3005] query failed: [1205] Lock wait timeout exceeded; try restarting transaction [update hosts set available=2,error='Get value from agent failed: cannot connect to [[[XX.XX.XX.XX]]:10050]: [113] No route to host' where hostid=58759; 24120:20170111:185939.689 [Z3005] query failed: [1205] Lock wait timeout exceeded; try restarting transaction [update hosts set available=2,error='Get value from agent failed: cannot connect to [[[XX.XX.XX.XX]]:10050]: [113] No route to host' where hostid=63613; 23848:20170111:185955.686 [Z3005] query failed: [1205] Lock wait timeout exceeded; try restarting transaction [update hosts set error='Get value from agent failed: cannot connect to [[[XX.XX.XX.XX]]:10050]: [111] Connection refused' where hostid=63700; 24176:20170111:190122.584 [Z3005] query failed: [1205] Lock wait timeout exceeded; try restarting transaction [update hosts set available=2,error='Get value from agent failed: cannot connect to [[[XX.XX.XX.XX]]:10050]: [113] No route to host' where hostid=55279; 23757:20170111:190142.437 [Z3005] query failed: [1205] Lock wait timeout exceeded; try restarting transaction [update hosts set available=2,error='Get value from agent failed: cannot connect to [[[XX.XX.XX.XX]]:10050]: [113] No route to host' where hostid=65976; 24099:20170111:190211.150 [Z3005] query failed: [1205] Lock wait timeout exceeded; try restarting transaction [update hosts set available=2,error='Get value from agent failed: cannot connect to [[[XX.XX.XX.XX]]:10050]: [113] No route to host' where hostid=71992; 24368:20170111:190333.953 [Z3005] query failed: [1205] Lock wait timeout exceeded; try restarting transaction [update hosts set available=2,error='Get value from agent failed: cannot connect to [[[XX.XX.XX.XX]]:10050]: [113] No route to host' where hostid=65452; 24294:20170111:190340.565 [Z3005] query failed: [1205] Lock wait timeout exceeded; try restarting transaction [update hosts set available=2,error='Get value from agent failed: cannot connect to [[[XX.XX.XX.XX]]:10050]: [113] No route to host' where hostid=56928; 24381:20170111:190435.956 [Z3005] query failed: [1205] Lock wait timeout exceeded; try restarting transaction [select null from hosts where hostid=60609 for update] 6794:20170112:102939.140 [Z3005] query failed: [1205] Lock wait timeout exceeded; try restarting transaction [update hosts set available=1,error='' where hostid=59460; 6702:20170112:102942.141 [Z3005] query failed: [1205] Lock wait timeout exceeded; try restarting transaction [update hosts set available=2,error='Get value from agent failed: cannot connect to [[[XX.XX.XX.XX]]:10050]: [113] No route to host' where hostid=48658; 7091:20170112:102953.147 [Z3005] query failed: [1205] Lock wait timeout exceeded; try restarting transaction [update hosts set error='Get value from agent failed: cannot connect to [[[XX.XX.XX.XX]]:10050]: [113] No route to host' where hostid=64896; 7072:20170112:103000.273 [Z3005] query failed: [1205] Lock wait timeout exceeded; try restarting transaction [update hosts set available=1,error='' where hostid=69385; 6549:20170112:103002.432 [Z3005] query failed: [1205] Lock wait timeout exceeded; try restarting transaction [update hosts set available=1,error='' where hostid=52009; 7018:20170112:103003.432 [Z3005] query failed: [1205] Lock wait timeout exceeded; try restarting transaction [update hosts set available=1,error='' where hostid=74091; 7052:20170112:103003.432 [Z3005] query failed: [1205] Lock wait timeout exceeded; try restarting transaction [update hosts set available=1,error='' where hostid=46770; 7064:20170112:103006.762 [Z3005] query failed: [1205] Lock wait timeout exceeded; try restarting transaction [update hosts set available=2,error='Get value from agent failed: cannot connect to [[[XX.XX.XX.XX]]:10050]: [113] No route to host' where hostid=68565; 6907:20170112:103010.728 [Z3005] query failed: [1205] Lock wait timeout exceeded; try restarting transaction [update hosts set available=1,error='' where hostid=85253; 6770:20170112:103014.729 [Z3005] query failed: [1205] Lock wait timeout exceeded; try restarting transaction [update hosts set available=1 where hostid=50438; 6817:20170112:103016.118 [Z3005] query failed: [1205] Lock wait timeout exceeded; try restarting transaction [update hosts set available=2,error='Get value from agent failed: cannot connect to [[[XX.XX.XX.XX]]:10050]: [113] No route to host' where hostid=81132; 6858:20170112:103028.262 [Z3005] query failed: [1205] Lock wait timeout exceeded; try restarting transaction [select null from hosts where hostid=109478 for update]
But critical point is that twice zabbix server stopped because of a rare condition:
24381:20170111:190435.956 [Z3005] query failed: [1205] Lock wait timeout exceeded; try restarting transaction [select null from hosts where hostid=60609 for update] 24381:20170111:190435.956 slow query: 1001.288937 sec, "select null from hosts where hostid=60609 for update" 24381:20170111:190435.978 [Z3005] query failed: [1048] Column 'itemid' cannot be null [insert into items_applications (itemappid,applicationid,itemid) values (11202011,97994,null),(11202012,97994,null); ] 24381:20170111:190435.978 ERROR: commit without transaction. Please report it to Zabbix Team. zabbix_server_mm: trapper #358 [processing data]: db.c:818: zbx_db_commit: Assertion `0' failed. 18495:20170111:190436.241 One child process died (PID:24381,exitcode/signal:6). Exiting ... 18495:20170111:190443.122 syncing history data... 18495:20170111:190538.706 syncing history data done 18495:20170111:190538.706 syncing trends data... 18495:20170111:190557.797 syncing trends data done 18495:20170111:190557.797 Zabbix Server stopped. Zabbix 3.0.6rc1 (revision 64124).
6858:20170112:103028.262 [Z3005] query failed: [1205] Lock wait timeout exceeded; try restarting transaction [select null from hosts where hostid=109478 for update] 6858:20170112:103028.262 slow query: 1000.388938 sec, "select null from hosts where hostid=109478 for update" 6858:20170112:103028.275 [Z3005] query failed: [1048] Column 'itemid' cannot be null [insert into items_applications (itemappid,applicationid,itemid) values (11228662,275523,null),(11228663,275523,null); ] 6858:20170112:103028.275 ERROR: commit without transaction. Please report it to Zabbix Team. zabbix_server_mm: trapper #384 [processing data]: db.c:818: zbx_db_commit: Assertion `0' failed. 328:20170112:103028.280 One child process died (PID:6858,exitcode/signal:6). Exiting ... 328:20170112:103030.391 syncing history data... 328:20170112:103030.405 syncing history data done 328:20170112:103030.405 syncing trends data... 328:20170112:103141.040 syncing trends data done 328:20170112:103141.040 Zabbix Server stopped. Zabbix 3.0.6rc1 (revision 64124).
This should be fixed that server will not stop.