zabbix server crashes after Syncing trends data

XMLWordPrintable

    • Type: Incident report
    • Resolution: Fixed
    • Priority: Critical
    • None
    • Affects Version/s: 1.9.0 (alpha)
    • Component/s: Server (S)
    • None
    • Environment:
      ZABBIX 1.7.1 (revision 8119) on SunOS 5.10 x86 and Oracle 10g

      This may or may not be a duplicate of ZBX-1095. Nearly all hosts are auto-discovered.

      Every 10 minutes the zabbix server is crashing, maybe even voluntarily shutting down (all processes have exited), after the "Syncing trends data" stage.

      12743:20091101:114323.489 In evaluate_simple(0=1)
      12743:20091101:114323.489 In evaluate_simple(0)
      12743:20091101:114323.489 In evaluate_simple(1)
      12743:20091101:114323.489 End evaluate(result:0.000000)
      12743:20091101:114323.489 evaluate_expression() result:0
      12743:20091101:114323.489 End of evaluate_expression():SUCCEED
      12743:20091101:114323.489 In update_trigger_value(triggerid:24406,old:0,new:0,1257093794)
      12743:20091101:114323.489 End update_trigger_value()
      12743:20091101:114323.490 In DCmass_update_trends()
      12743:20091101:114323.491 Syncing history data...done.
      12743:20091101:114323.491 In DCsync_trends(trends_num: 6825)
      12743:20091101:114323.491 Syncing trends data...

      Line 25,428 begins:

      12743:20091101:114323.491 Query [txnlev:0] [select num,value_min,value_avg,value_max from trends_uint where itemid=41837 and clock=1257091200]
      12743:20091101:114323.492 Query [txnlev:0] [select num,value_min,value_avg,value_max from trends_uint where itemid=41838 and clock=1257091200]
      12743:20091101:114323.492 Query [txnlev:0] [select num,value_min,value_avg,value_max from trends_uint where itemid=41839 and clock=1257091200]
      12743:20091101:114323.493 Query [txnlev:0] [select num,value_min,value_avg,value_max from trends_uint where itemid=41840 and clock=1257091200]
      12743:20091101:114323.493 Query [txnlev:0] [select num,value_min,value_avg,value_max from trends where itemid=41841 and clock=1257091200]

      Skipping to line 32,247...

      12743:20091101:114325.994 Query [txnlev:0] [select num,value_min,value_avg,value_max from trends_uint where itemid=54533 and clock=1257091200]
      12743:20091101:114325.995 Query [txnlev:0] [select num,value_min,value_avg,value_max from trends_uint where itemid=54536 and clock=1257091200]
      12743:20091101:114325.995 Query [txnlev:0] [select num,value_min,value_avg,value_max from trends_uint where itemid=54538 and clock=1257091200]
      12743:20091101:114325.998 Query [txnlev:1] [begin
      insert into trends_uint (itemid,clock,num,value_min,value_avg,value_max) values (41837,1257091200,10,29,29,29);
      insert into trends_uint (itemid,clock,num,value_min,value_avg,value_max) values (41838,1257091200,9,24,24,25);
      insert into trends_uint (itemid,clock,num,value_min,value_avg,value_max) values (41839,1257091200,11,22,22,23);
      insert into trends_uint (itemid,clock,num,value_min,value_avg,value_max) values (41840,1257091200,9,21,21,21);
      insert into trends (itemid,clock,num,value_min,value_avg,value_max) values (41841,1257091200,10,39540.296978,120433.758562,152046.868852);

      Skipping to line 39,074...

      insert into trends (itemid,clock,num,value_min,value_avg,value_max) values (54531,1257091200,5,1.132662,1.132663,1.132663);
      insert into trends_uint (itemid,clock,num,value_min,value_avg,value_max) values (54532,1257091200,5,357014774784,357014786661,357014834176);
      insert into trends_uint (itemid,clock,num,value_min,value_avg,value_max) values (54533,1257091200,5,4043772928,4043773337,4043773952);
      insert into trends_uint (itemid,clock,num,value_min,value_avg,value_max) values (54536,1257091200,8,2436820992,2559283199,2746306560);
      insert into trends_uint (itemid,clock,num,value_min,value_avg,value_max) values (54538,1257091200,1,17179869184,17179869184,17179869184);
      end;
      ]
      12743:20091101:114356.867 Syncing trends data...done.
      12743:20091101:114356.867 End of DCsync_trends()
      12743:20091101:114356.867 End of DCsync_all()
      12743:20091101:114356.867 End of free_database_cache()
      12743:20091101:114356.867 In free_configuration_cache()
      12743:20091101:114356.867 End of free_configuration_cache()
      12743:20091101:114356.870 ZABBIX Server stopped. ZABBIX 1.7.1 (revision 8119).

      $ ldd zabbix_server
      libcurl.so.4 => /usr/local/lib/libcurl.so.4
      libssl.so.0.9.7 => /usr/local/lib/libssl.so.0.9.7
      libcrypto.so.0.9.7 => /usr/local/lib/libcrypto.so.0.9.7
      librt.so.1 => /lib/librt.so.1
      libsocket.so.1 => /lib/libsocket.so.1
      libnsl.so.1 => /lib/libnsl.so.1
      libdl.so.1 => /lib/libdl.so.1
      libz.so.1 => /usr/lib/libz.so.1
      libnetsnmp.so.5 => /usr/sfw/lib/libnetsnmp.so.5
      libgen.so.1 => /lib/libgen.so.1
      libpkcs11.so.1 => /usr/lib/libpkcs11.so.1
      libkstat.so.1 => /lib/libkstat.so.1
      libelf.so.1 => /lib/libelf.so.1
      libm.so.2 => /lib/libm.so.2
      libadm.so.1 => /lib/libadm.so.1
      libldap-2.3.so.0 => /usr/local/lib/libldap-2.3.so.0
      libclntsh.so.10.1 => /applications/oracle/10.2.0/lib/libclntsh.so.10.1
      libnnz10.so => /applications/oracle/10.2.0/lib/libnnz10.so
      libkvm.so.1 => /usr/lib/libkvm.so.1
      libresolv.so.2 => /lib/libresolv.so.2
      libc.so.1 => /lib/libc.so.1
      libgcc_s.so.1 => /usr/local/lib/libgcc_s.so.1
      libaio.so.1 => /lib/libaio.so.1
      libmd.so.1 => /lib/libmd.so.1
      libmp.so.2 => /lib/libmp.so.2
      libscf.so.1 => /lib/libscf.so.1
      libcryptoutil.so.1 => /usr/lib/libcryptoutil.so.1
      libdoor.so.1 => /lib/libdoor.so.1
      liblber-2.3.so.0 => /usr/local/lib/liblber-2.3.so.0
      libsasl2.so.2 => /usr/local/lib/libsasl2.so.2
      libsched.so.1 => /usr/lib/libsched.so.1
      libuutil.so.1 => /lib/libuutil.so.1

            Assignee:
            Unassigned
            Reporter:
            Jeff Quast
            Votes:
            1 Vote for this issue
            Watchers:
            1 Start watching this issue

              Created:
              Updated:
              Resolved: