Uploaded image for project: 'ZABBIX BUGS AND ISSUES'
  1. ZABBIX BUGS AND ISSUES
  2. ZBX-11703

zabbix server may stop after failed query - ERROR: commit without transaction

    Details

      Description

      This zabbix server has many deadlock messages in log file, here for ~20 hours:

      # grep "Lock wait timeout exceeded" zabbix_server_12_01.log 
       24383:20170111:184028.660 [Z3005] query failed: [1205] Lock wait timeout exceeded; try restarting transaction [update hosts set available=2,error='Get value from agent failed: cannot connect to [[[XX.XX.XX.XX]]:10050]: [113] No route to host' where hostid=49297;
       24526:20170111:184459.097 [Z3005] query failed: [1205] Lock wait timeout exceeded; try restarting transaction [update hosts set available=2,error='Get value from agent failed: cannot connect to [[[XX.XX.XX.XX]]:10050]: [113] No route to host' where hostid=49423;
       23921:20170111:184537.807 [Z3005] query failed: [1205] Lock wait timeout exceeded; try restarting transaction [update hosts set available=1,error='' where hostid=51567;
       24050:20170111:184556.526 [Z3005] query failed: [1205] Lock wait timeout exceeded; try restarting transaction [update hosts set available=1,error='' where hostid=52995;
       24804:20170111:184930.063 [Z3005] query failed: [1205] Lock wait timeout exceeded; try restarting transaction [update hosts set available=2,error='Get value from agent failed: cannot connect to [[[XX.XX.XX.XX]]:10050]: [113] No route to host' where hostid=48267;
       24149:20170111:185331.951 [Z3005] query failed: [1205] Lock wait timeout exceeded; try restarting transaction [update hosts set available=2,error='Get value from agent failed: cannot connect to [[[XX.XX.XX.XX]]:10050]: [113] No route to host' where hostid=59019;
       24122:20170111:185421.810 [Z3005] query failed: [1205] Lock wait timeout exceeded; try restarting transaction [update hosts set available=2,error='Get value from agent failed: cannot connect to [[[XX.XX.XX.XX]]:10050]: [113] No route to host' where hostid=50982;
       24721:20170111:185533.073 [Z3005] query failed: [1205] Lock wait timeout exceeded; try restarting transaction [update hosts set error='Get value from agent failed: cannot connect to [[[XX.XX.XX.XX]]:10050]: [113] No route to host' where hostid=63192;
       24288:20170111:185539.271 [Z3005] query failed: [1205] Lock wait timeout exceeded; try restarting transaction [update hosts set available=2,error='Get value from agent failed: cannot connect to [[[XX.XX.XX.XX]]:10050]: [113] No route to host' where hostid=62481;
       24797:20170111:185604.309 [Z3005] query failed: [1205] Lock wait timeout exceeded; try restarting transaction [update hosts set available=2,error='Get value from agent failed: cannot connect to [[[XX.XX.XX.XX]]:10050]: [113] No route to host' where hostid=58739;
       24705:20170111:185650.512 [Z3005] query failed: [1205] Lock wait timeout exceeded; try restarting transaction [update hosts set available=2,error='Get value from agent failed: cannot connect to [[[XX.XX.XX.XX]]:10050]: [113] No route to host' where hostid=64196;
       24444:20170111:185709.946 [Z3005] query failed: [1205] Lock wait timeout exceeded; try restarting transaction [update hosts set available=2,error='Get value from agent failed: cannot connect to [[[XX.XX.XX.XX]]:10050]: [113] No route to host' where hostid=52145;
       23878:20170111:185714.509 [Z3005] query failed: [1205] Lock wait timeout exceeded; try restarting transaction [update hosts set available=1,error='' where hostid=65274;
       23879:20170111:185917.687 [Z3005] query failed: [1205] Lock wait timeout exceeded; try restarting transaction [update hosts set available=2,error='Get value from agent failed: cannot connect to [[[XX.XX.XX.XX]]:10050]: [113] No route to host' where hostid=61593;
       23756:20170111:185920.685 [Z3005] query failed: [1205] Lock wait timeout exceeded; try restarting transaction [update hosts set available=2,error='Get value from agent failed: cannot connect to [[[XX.XX.XX.XX]]:10050]: [113] No route to host' where hostid=58759;
       24120:20170111:185939.689 [Z3005] query failed: [1205] Lock wait timeout exceeded; try restarting transaction [update hosts set available=2,error='Get value from agent failed: cannot connect to [[[XX.XX.XX.XX]]:10050]: [113] No route to host' where hostid=63613;
       23848:20170111:185955.686 [Z3005] query failed: [1205] Lock wait timeout exceeded; try restarting transaction [update hosts set error='Get value from agent failed: cannot connect to [[[XX.XX.XX.XX]]:10050]: [111] Connection refused' where hostid=63700;
       24176:20170111:190122.584 [Z3005] query failed: [1205] Lock wait timeout exceeded; try restarting transaction [update hosts set available=2,error='Get value from agent failed: cannot connect to [[[XX.XX.XX.XX]]:10050]: [113] No route to host' where hostid=55279;
       23757:20170111:190142.437 [Z3005] query failed: [1205] Lock wait timeout exceeded; try restarting transaction [update hosts set available=2,error='Get value from agent failed: cannot connect to [[[XX.XX.XX.XX]]:10050]: [113] No route to host' where hostid=65976;
       24099:20170111:190211.150 [Z3005] query failed: [1205] Lock wait timeout exceeded; try restarting transaction [update hosts set available=2,error='Get value from agent failed: cannot connect to [[[XX.XX.XX.XX]]:10050]: [113] No route to host' where hostid=71992;
       24368:20170111:190333.953 [Z3005] query failed: [1205] Lock wait timeout exceeded; try restarting transaction [update hosts set available=2,error='Get value from agent failed: cannot connect to [[[XX.XX.XX.XX]]:10050]: [113] No route to host' where hostid=65452;
       24294:20170111:190340.565 [Z3005] query failed: [1205] Lock wait timeout exceeded; try restarting transaction [update hosts set available=2,error='Get value from agent failed: cannot connect to [[[XX.XX.XX.XX]]:10050]: [113] No route to host' where hostid=56928;
       24381:20170111:190435.956 [Z3005] query failed: [1205] Lock wait timeout exceeded; try restarting transaction [select null from hosts where hostid=60609 for update]
        6794:20170112:102939.140 [Z3005] query failed: [1205] Lock wait timeout exceeded; try restarting transaction [update hosts set available=1,error='' where hostid=59460;
        6702:20170112:102942.141 [Z3005] query failed: [1205] Lock wait timeout exceeded; try restarting transaction [update hosts set available=2,error='Get value from agent failed: cannot connect to [[[XX.XX.XX.XX]]:10050]: [113] No route to host' where hostid=48658;
        7091:20170112:102953.147 [Z3005] query failed: [1205] Lock wait timeout exceeded; try restarting transaction [update hosts set error='Get value from agent failed: cannot connect to [[[XX.XX.XX.XX]]:10050]: [113] No route to host' where hostid=64896;
        7072:20170112:103000.273 [Z3005] query failed: [1205] Lock wait timeout exceeded; try restarting transaction [update hosts set available=1,error='' where hostid=69385;
        6549:20170112:103002.432 [Z3005] query failed: [1205] Lock wait timeout exceeded; try restarting transaction [update hosts set available=1,error='' where hostid=52009;
        7018:20170112:103003.432 [Z3005] query failed: [1205] Lock wait timeout exceeded; try restarting transaction [update hosts set available=1,error='' where hostid=74091;
        7052:20170112:103003.432 [Z3005] query failed: [1205] Lock wait timeout exceeded; try restarting transaction [update hosts set available=1,error='' where hostid=46770;
        7064:20170112:103006.762 [Z3005] query failed: [1205] Lock wait timeout exceeded; try restarting transaction [update hosts set available=2,error='Get value from agent failed: cannot connect to [[[XX.XX.XX.XX]]:10050]: [113] No route to host' where hostid=68565;
        6907:20170112:103010.728 [Z3005] query failed: [1205] Lock wait timeout exceeded; try restarting transaction [update hosts set available=1,error='' where hostid=85253;
        6770:20170112:103014.729 [Z3005] query failed: [1205] Lock wait timeout exceeded; try restarting transaction [update hosts set available=1 where hostid=50438;
        6817:20170112:103016.118 [Z3005] query failed: [1205] Lock wait timeout exceeded; try restarting transaction [update hosts set available=2,error='Get value from agent failed: cannot connect to [[[XX.XX.XX.XX]]:10050]: [113] No route to host' where hostid=81132;
        6858:20170112:103028.262 [Z3005] query failed: [1205] Lock wait timeout exceeded; try restarting transaction [select null from hosts where hostid=109478 for update]
      

      But critical point is that twice zabbix server stopped because of a rare condition:

       24381:20170111:190435.956 [Z3005] query failed: [1205] Lock wait timeout exceeded; try restarting transaction [select null from hosts where hostid=60609 for update]
       24381:20170111:190435.956 slow query: 1001.288937 sec, "select null from hosts where hostid=60609 for update"
       24381:20170111:190435.978 [Z3005] query failed: [1048] Column 'itemid' cannot be null [insert into items_applications (itemappid,applicationid,itemid) values (11202011,97994,null),(11202012,97994,null);
      ]
       24381:20170111:190435.978 ERROR: commit without transaction. Please report it to Zabbix Team.
      zabbix_server_mm: trapper #358 [processing data]: db.c:818: zbx_db_commit: Assertion `0' failed.
       18495:20170111:190436.241 One child process died (PID:24381,exitcode/signal:6). Exiting ...
       18495:20170111:190443.122 syncing history data...
       18495:20170111:190538.706 syncing history data done
       18495:20170111:190538.706 syncing trends data...
       18495:20170111:190557.797 syncing trends data done
       18495:20170111:190557.797 Zabbix Server stopped. Zabbix 3.0.6rc1 (revision 64124).
      
        6858:20170112:103028.262 [Z3005] query failed: [1205] Lock wait timeout exceeded; try restarting transaction [select null from hosts where hostid=109478 for update]
        6858:20170112:103028.262 slow query: 1000.388938 sec, "select null from hosts where hostid=109478 for update"
        6858:20170112:103028.275 [Z3005] query failed: [1048] Column 'itemid' cannot be null [insert into items_applications (itemappid,applicationid,itemid) values (11228662,275523,null),(11228663,275523,null);
      ]
        6858:20170112:103028.275 ERROR: commit without transaction. Please report it to Zabbix Team.
      zabbix_server_mm: trapper #384 [processing data]: db.c:818: zbx_db_commit: Assertion `0' failed.
         328:20170112:103028.280 One child process died (PID:6858,exitcode/signal:6). Exiting ...
         328:20170112:103030.391 syncing history data...
         328:20170112:103030.405 syncing history data done
         328:20170112:103030.405 syncing trends data...
         328:20170112:103141.040 syncing trends data done
         328:20170112:103141.040 Zabbix Server stopped. Zabbix 3.0.6rc1 (revision 64124).
      

      This should be fixed that server will not stop.

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              zalex_ua Oleksiy Zagorskyi
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: