-
Incident report
-
Resolution: Fixed
-
Critical
-
2.1.9
zabbix server was started. db was unavailable initially, then became available. it started db upgrade process, got sigterm. after running the upgrade for ~25 minutes, it crashed.
21450:20131105:073223.044 Database is down. Reconnecting in 10 seconds.
21450:20131105:073233.100 current database version (mandatory/optional): 02010190/02010190
21450:20131105:073233.100 required mandatory version: 02010195
21450:20131105:073233.100 starting automatic database upgrade
21450:20131105:073741.606 Got signal [signal:15(SIGTERM),sender_pid:15435,sender_uid:0,reason:0]. Exiting ...
21450:20131105:075641.797 Got signal [signal:11(SIGSEGV),reason:1,refaddr:0x6c]. Crashing ...
21450:20131105:075642.110 ====== Fatal information: ======
21450:20131105:075642.110 Program counter: 0x808cacf
21450:20131105:075642.110 === Registers: ===
21450:20131105:075642.110 gs = 33 = 51 = 51
21450:20131105:075642.110 fs = 0 = 0 = 0
21450:20131105:075642.110 es = 7b = 123 = 123
21450:20131105:075642.166 ds = 7b = 123 = 123
21450:20131105:075642.166 edi = 4 = 4 = 4
21450:20131105:075642.166 esi = 8dfe463 = 148890723 = 148890723
21450:20131105:075642.166 ebp = bfbe42e8 = 3216917224 = -1078050072
21450:20131105:075642.166 esp = bfbe3c40 = 3216915520 = -1078051776
21450:20131105:075642.166 ebx = 0 = 0 = 0
21450:20131105:075642.166 edx = b72b2410 = 3073057808 = -1221909488
21450:20131105:075642.166 ecx = b72b23e0 = 3073057760 = -1221909536
21450:20131105:075642.196 eax = 0 = 0 = 0
21450:20131105:075642.196 trapno = e = 14 = 14
21450:20131105:075642.196 err = 4 = 4 = 4
21450:20131105:075642.196 eip = 808cacf = 134793935 = 134793935
21450:20131105:075642.210 cs = 73 = 115 = 115
21450:20131105:075642.210 efl = 210282 = 2163330 = 2163330
21450:20131105:075642.210 uesp = bfbe3c40 = 3216915520 = -1078051776
21450:20131105:075642.210 ss = 7b = 123 = 123
21450:20131105:075642.210 === Stack frame: ===
21450:20131105:075642.210 +0x40(%ebp) = ebp + 64 = 00000004 = 4 = 4
21450:20131105:075642.210 +0x3c(%ebp) = ebp + 60 = 08e02460 = 148907104 = 148907104
21450:20131105:075642.210 +0x38(%ebp) = ebp + 56 = 08df7c10 = 148864016 = 148864016
21450:20131105:075642.210 +0x34(%ebp) = ebp + 52 = b7721f32 = 3077709618 = -1217257678
21450:20131105:075642.210 +0x30(%ebp) = ebp + 48 = bfbe4378 = 3216917368 = -1078049928
21450:20131105:075642.210 +0x2c(%ebp) = ebp + 44 = 00000004 = 4 = 4
21450:20131105:075642.210 +0x28(%ebp) = ebp + 40 = 08e02460 = 148907104 = 148907104
21450:20131105:075642.210 +0x24(%ebp) = ebp + 36 = b772d22c = 3077755436 = -1217211860
21450:20131105:075642.210 +0x20(%ebp) = ebp + 32 = bfbe43e0 = 3216917472 = -1078049824
21450:20131105:075642.210 +0x1c(%ebp) = ebp + 28 = bfbe4340 = 3216917312 = -1078049984
21450:20131105:075642.210 +0x18(%ebp) = ebp + 24 = 08e0a470 = 148939888 = 148939888
21450:20131105:075642.210 +0x14(%ebp) = ebp + 20 = bfbe436b = 3216917355 = -1078049941
21450:20131105:075642.210 +0x10(%ebp) = ebp + 16 = 00004000 = 16384 = 16384
21450:20131105:075642.210 +0x0c(%ebp) = ebp + 12 = 08dfe458 = 148890712 = 148890712
21450:20131105:075642.210 +0x08(%ebp) = ebp + 8 = 08df7c10 = 148864016 = 148864016 <--- call arguments
21450:20131105:075642.210 +0x04(%ebp) = ebp + 4 = b7721403 <--- return address
21450:20131105:075642.210 (%ebp) = ebp = bfbe4318 <--- saved ebp value
21450:20131105:075642.210 0x04(%ebp) = ebp - 4 = b772d22c = 3077755436 = -1217211860 <-- local variables
21450:20131105:075642.210 -0x08(%ebp) = ebp - 8 = bfbe4340 = 3216917312 = -1078049984
21450:20131105:075642.210 -0x0c(%ebp) = ebp - 12 = 08dfe458 = 148890712 = 148890712
21450:20131105:075642.210 -0x10(%ebp) = ebp - 16 = 00004000 = 16384 = 16384
21450:20131105:075642.211 -0x14(%ebp) = ebp - 20 = 08dfe458 = 148890712 = 148890712
21450:20131105:075642.211 -0x18(%ebp) = ebp - 24 = 00000004 = 4 = 4
21450:20131105:075642.211 -0x1c(%ebp) = ebp - 28 = b7721392 = 3077706642 = -1217260654
21450:20131105:075642.211 -0x20(%ebp) = ebp - 32 = b772d22c = 3077755436 = -1217211860
21450:20131105:075642.211 -0x24(%ebp) = ebp - 36 = 00000000 = 0 = 0
21450:20131105:075642.211 -0x28(%ebp) = ebp - 40 = 00000000 = 0 = 0
21450:20131105:075642.211 -0x2c(%ebp) = ebp - 44 = 00000000 = 0 = 0
21450:20131105:075642.211 -0x30(%ebp) = ebp - 48 = 00000000 = 0 = 0
21450:20131105:075642.211 -0x34(%ebp) = ebp - 52 = 00000000 = 0 = 0
21450:20131105:075642.211 -0x38(%ebp) = ebp - 56 = 00000000 = 0 = 0
21450:20131105:075642.211 -0x3c(%ebp) = ebp - 60 = 00000000 = 0 = 0
21450:20131105:075642.211 -0x40(%ebp) = ebp - 64 = 00000000 = 0 = 0
21450:20131105:075642.211 === Backtrace: ===
21450:20131105:075642.311 16: /usr/local/zabbix-2.2.0rc1/sbin/zabbix_server(print_fatal_info+0x248) [0x80a7568]
21450:20131105:075642.311 15: /usr/local/zabbix-2.2.0rc1/sbin/zabbix_server() [0x80a7999]
21450:20131105:075642.311 14: linux-gate.so.1(__kernel_rt_sigreturn+0) [0xffffe410]
21450:20131105:075642.311 13: /usr/local/zabbix-2.2.0rc1/sbin/zabbix_server(DCsync_history+0xf) [0x808cacf]
21450:20131105:075642.311 12: /usr/local/zabbix-2.2.0rc1/sbin/zabbix_server(free_database_cache+0x41) [0x8090dc1]
21450:20131105:075642.311 11: /usr/local/zabbix-2.2.0rc1/sbin/zabbix_server(zbx_on_exit+0xf5) [0x805c795]
21450:20131105:075642.311 10: linux-gate.so.1(__kernel_rt_sigreturn+0) [0xffffe410]
21450:20131105:075642.311 9: /lib/libc.so.6(__read+0x1e) [0xb7206a2e]
21450:20131105:075642.311 8: /usr/local/lib/mysql/libmysqlclient.so.16(vio_read+0x32) [0xb7721392]
21450:20131105:075642.311 7: /usr/local/lib/mysql/libmysqlclient.so.16(vio_read_buff+0x63) [0xb7721403]
21450:20131105:075642.311 6: /usr/local/lib/mysql/libmysqlclient.so.16(+0x37f32) [0xb7721f32]
21450:20131105:075642.311 5: /usr/local/lib/mysql/libmysqlclient.so.16(my_net_read+0x1ea) [0xb77222fa]
21450:20131105:075642.311 4: /usr/local/lib/mysql/libmysqlclient.so.16(cli_safe_read+0x45) [0xb771cc55]
21450:20131105:075642.311 3: /usr/local/lib/mysql/libmysqlclient.so.16(+0x33444) [0xb771d444]
21450:20131105:075642.311 2: /usr/local/lib/mysql/libmysqlclient.so.16(mysql_real_query+0x4c) [0xb771b88c]
21450:20131105:075642.312 1: /usr/local/lib/mysql/libmysqlclient.so.16(mysql_query+0x2b) [0xb76fbe5b]
21450:20131105:075642.312 0: /usr/local/zabbix-2.2.0rc1/sbin/zabbix_server(zbx_db_vexecute+0x91) [0x80f6d21]
21450:20131105:075642.312 === Memory map: ===
21450:20131105:075642.312 08048000-08173000 r-xp 00000000 08:01 66301 /usr/local/zabbix-2.2.0rc1/sbin/zabbix_server (deleted)
21450:20131105:075642.312 08173000-08176000 rw-p 0012a000 08:01 66301 /usr/local/zabbix-2.2.0rc1/sbin/zabbix_server (deleted)
21450:20131105:075642.312 08176000-0817c000 rw-p 00000000 00:00 0
21450:20131105:075642.312 08def000-08e10000 rw-p 00000000 00:00 0 [heap]
21450:20131105:075642.312 08e10000-08e31000 rw-p 00000000 00:00 0 [heap]
21450:20131105:075642.312 b6ef1000-b6efc000 r-xp 00000000 08:01 823921 /lib/libnss_files-2.15.so
21450:20131105:075642.312 b6efc000-b6efd000 r--p 0000a000 08:01 823921 /lib/libnss_files-2.15.so
21450:20131105:075642.312 b6efd000-b6efe000 rw-p 0000b000 08:01 823921 /lib/libnss_files-2.15.so
21450:20131105:075642.312 b6efe000-b6f08000 r-xp 00000000 08:01 823949 /lib/libnss_nis-2.15.so
21450:20131105:075642.312 b6f08000-b6f09000 r--p 00009000 08:01 823949 /lib/libnss_nis-2.15.so
21450:20131105:075642.312 b6f09000-b6f0a000 rw-p 0000a000 08:01 823949 /lib/libnss_nis-2.15.so
21450:20131105:075642.312 b6f0a000-b6f12000 r-xp 00000000 08:01 823917 /lib/libnss_compat-2.15.so
21450:20131105:075642.312 b6f12000-b6f13000 r--p 00007000 08:01 823917 /lib/libnss_compat-2.15.so
21450:20131105:075642.312 b6f13000-b6f14000 rw-p 00008000 08:01 823917 /lib/libnss_compat-2.15.so
21450:20131105:075642.312 b6f14000-b6f16000 rw-p 00000000 00:00 0
21450:20131105:075642.312 b6f16000-b6f19000 r-xp 00000000 08:01 247671 /lib/libgpg-error.so.0.8.0
21450:20131105:075642.312 b6f19000-b6f1a000 rw-p 00002000 08:01 247671 /lib/libgpg-error.so.0.8.0
21450:20131105:075642.312 b6f1a000-b6f33000 r-xp 00000000 08:01 535843 /usr/lib/libsasl2.so.2.0.23
21450:20131105:075642.312 b6f33000-b6f34000 rw-p 00018000 08:01 535843 /usr/lib/libsasl2.so.2.0.23
21450:20131105:075642.312 b6f34000-b6f4a000 r-xp 00000000 08:01 823955 /lib/libpthread-2.15.so
21450:20131105:075642.312 b6f4a000-b6f4b000 r--p 00015000 08:01 823955 /lib/libpthread-2.15.so
21450:20131105:075642.312 b6f4b000-b6f4c000 rw-p 00016000 08:01 823955 /lib/libpthread-2.15.so
21450:20131105:075642.312 b6f4c000-b6f4e000 rw-p 00000000 00:00 0
21450:20131105:075642.312 b6f4e000-b6f53000 r-xp 00000000 08:01 533859 /usr/lib/libgdbm.so.3.0.0
21450:20131105:075642.312 b6f53000-b6f54000 rw-p 00004000 08:01 533859 /usr/lib/libgdbm.so.3.0.0
21450:20131105:075642.313 b6f54000-b6f55000 rw-p 00000000 00:00 0
21450:20131105:075642.313 b6f55000-b6f5d000 r-xp 00000000 08:01 550101 /usr/local/lib/libOpenIPMIutils.so.0.0.1
21450:20131105:075642.313 b6f5d000-b6f5e000 rw-p 00007000 08:01 550101 /usr/local/lib/libOpenIPMIutils.so.0.0.1
21450:20131105:075642.313 b6f5e000-b6fde000 r-xp 00000000 08:01 247669 /lib/libgcrypt.so.11.7.0
21450:20131105:075642.313 b6fde000-b6fe1000 rw-p 00080000 08:01 247669 /lib/libgcrypt.so.11.7.0
21450:20131105:075642.313 b6fe1000-b703d000 r-xp 00000000 08:01 267650 /lib/libssl.so.1.0.0
21450:20131105:075642.313 b703d000-b7042000 rw-p 0005c000 08:01 267650 /lib/libssl.so.1.0.0
21450:20131105:075642.313 b7042000-b708a000 r-xp 00000000 08:01 534855 /usr/lib/libldap-2.4.so.2.8.3
21450:20131105:075642.313 b708a000-b708c000 rw-p 00047000 08:01 534855 /usr/lib/libldap-2.4.so.2.8.3
21450:20131105:075642.313 b708c000-b7099000 r-xp 00000000 08:01 533650 /usr/lib/liblber-2.4.so.2.8.3
21450:20131105:075642.313 b7099000-b709a000 rw-p 0000c000 08:01 533650 /usr/lib/liblber-2.4.so.2.8.3
21450:20131105:075642.313 b709a000-b709b000 rw-p 00000000 00:00 0
21450:20131105:075642.313 b709b000-b70cc000 r-xp 00000000 08:01 535670 /usr/lib/libidn.so.11.6.8
21450:20131105:075642.313 b70cc000-b70cd000 rw-p 00031000 08:01 535670 /usr/lib/libidn.so.11.6.8
21450:20131105:075642.313 b70cd000-b70e2000 r-xp 00000000 08:01 824030 /lib/libz.so.1.2.6
21450:20131105:075642.313 b70e2000-b70e3000 rw-p 00014000 08:01 824030 /lib/libz.so.1.2.6
21450:20131105:075642.313 b70e3000-b70fa000 r-xp 00000000 08:01 823909 /lib/libnsl-2.15.so
21450:20131105:075642.313 b70fa000-b70fb000 r--p 00016000 08:01 823909 /lib/libnsl-2.15.so
21450:20131105:075642.313 b70fb000-b70fc000 rw-p 00017000 08:01 823909 /lib/libnsl-2.15.so
21450:20131105:075642.313 b70fc000-b70fe000 rw-p 00000000 00:00 0
21450:20131105:075642.313 b70fe000-b7108000 r-xp 00000000 08:01 823904 /lib/libcrypt-2.15.so
21450:20131105:075642.313 b7108000-b7109000 r--p 00009000 08:01 823904 /lib/libcrypt-2.15.so
21450:20131105:075642.313 b7109000-b710a000 rw-p 0000a000 08:01 823904 /lib/libcrypt-2.15.so
21450:20131105:075642.313 b710a000-b7131000 rw-p 00000000 00:00 0
21450:20131105:075642.313 b7131000-b72af000 r-xp 00000000 08:01 823985 /lib/libc-2.15.so
21450:20131105:075642.313 b72af000-b72b0000 ---p 0017e000 08:01 823985 /lib/libc-2.15.so
21450:20131105:075642.313 b72b0000-b72b2000 r--p 0017e000 08:01 823985 /lib/libc-2.15.so
21450:20131105:075642.313 b72b2000-b72b3000 rw-p 00180000 08:01 823985 /lib/libc-2.15.so
21450:20131105:075642.313 b72b3000-b72b6000 rw-p 00000000 00:00 0
21450:20131105:075642.314 b72b6000-b72cb000 r-xp 00000000 08:01 823978 /lib/libresolv-2.15.so
21450:20131105:075642.314 b72cb000-b72cc000 r--p 00015000 08:01 823978 /lib/libresolv-2.15.so
21450:20131105:075642.314 b72cc000-b72cd000 rw-p 00016000 08:01 823978 /lib/libresolv-2.15.so
21450:20131105:075642.314 b72cd000-b72d0000 rw-p 00000000 00:00 0
21450:20131105:075642.314 b72d0000-b72d7000 r-xp 00000000 08:01 823979 /lib/librt-2.15.so
21450:20131105:075642.314 b72d7000-b72d8000 r--p 00006000 08:01 823979 /lib/librt-2.15.so
21450:20131105:075642.314 b72d8000-b72d9000 rw-p 00007000 08:01 823979 /lib/librt-2.15.so
21450:20131105:075642.314 b72d9000-b72dc000 r-xp 00000000 08:01 823986 /lib/libdl-2.15.so
21450:20131105:075642.314 b72dc000-b72dd000 r--p 00002000 08:01 823986 /lib/libdl-2.15.so
21450:20131105:075642.314 b72dd000-b72de000 rw-p 00003000 08:01 823986 /lib/libdl-2.15.so
21450:20131105:075642.314 b72de000-b7308000 r-xp 00000000 08:01 823907 /lib/libm-2.15.so
21450:20131105:075642.314 b7308000-b7309000 r--p 00029000 08:01 823907 /lib/libm-2.15.so
21450:20131105:075642.314 b7309000-b730a000 rw-p 0002a000 08:01 823907 /lib/libm-2.15.so
21450:20131105:075642.314 b730a000-b730f000 r-xp 00000000 08:01 550110 /usr/local/lib/libOpenIPMIposix.so.0.0.1
21450:20131105:075642.314 b730f000-b7310000 rw-p 00004000 08:01 550110 /usr/local/lib/libOpenIPMIposix.so.0.0.1
21450:20131105:075642.314 b7310000-b7404000 r-xp 00000000 08:01 550115 /usr/local/lib/libOpenIPMI.so.0.0.5
21450:20131105:075642.314 b7404000-b7412000 rw-p 000f4000 08:01 550115 /usr/local/lib/libOpenIPMI.so.0.0.5
21450:20131105:075642.314 b7412000-b7415000 rw-p 00000000 00:00 0
21450:20131105:075642.314 b7415000-b743c000 r-xp 00000000 08:01 535006 /usr/lib/libssh2.so.1.0.1
21450:20131105:075642.314 b743c000-b743d000 rw-p 00027000 08:01 535006 /usr/lib/libssh2.so.1.0.1
21450:20131105:075642.314 b743d000-b743e000 rw-p 00000000 00:00 0
21450:20131105:075642.314 b743e000-b75dd000 r-xp 00000000 08:01 267653 /lib/libcrypto.so.1.0.0
21450:20131105:075642.314 b75dd000-b75f2000 rw-p 0019f000 08:01 267653 /lib/libcrypto.so.1.0.0
21450:20131105:075642.314 b75f2000-b75f5000 rw-p 00000000 00:00 0
21450:20131105:075642.314 b75f5000-b766e000 r-xp 00000000 08:01 534566 /usr/lib/libnetsnmp.so.25.0.1
21450:20131105:075642.314 b766e000-b7671000 rw-p 00078000 08:01 534566 /usr/lib/libnetsnmp.so.25.0.1
21450:20131105:075642.314 b7671000-b7690000 rw-p 00000000 00:00 0
21450:20131105:075642.314 b7690000-b76e8000 r-xp 00000000 08:01 533651 /usr/lib/libcurl.so.4.2.0
21450:20131105:075642.315 b76e8000-b76ea000 rw-p 00057000 08:01 533651 /usr/lib/libcurl.so.4.2.0
21450:20131105:075642.315 b76ea000-b772c000 r-xp 00000000 08:01 76172 /usr/local/lib/mysql/libmysqlclient.so.16.0.0
21450:20131105:075642.315 b772c000-b7756000 rw-p 00041000 08:01 76172 /usr/local/lib/mysql/libmysqlclient.so.16.0.0
21450:20131105:075642.315 b7756000-b7758000 rw-p 00000000 00:00 0
21450:20131105:075642.315 b7758000-b7774000 r-xp 00000000 08:01 534922 /usr/lib/libgcc_s.so.1
21450:20131105:075642.315 b7774000-b7775000 rw-p 0001b000 08:01 534922 /usr/lib/libgcc_s.so.1
21450:20131105:075642.315 b7775000-b7776000 rw-p 00000000 00:00 0
21450:20131105:075642.315 b7776000-b7777000 rw-p 00000000 00:00 0
21450:20131105:075642.315 b7777000-b7798000 r-xp 00000000 08:01 824020 /lib/ld-2.15.so
21450:20131105:075642.315 b7798000-b7799000 r--p 00020000 08:01 824020 /lib/ld-2.15.so
21450:20131105:075642.315 b7799000-b779a000 rw-p 00021000 08:01 824020 /lib/ld-2.15.so
21450:20131105:075642.315 bfbc8000-bfbe9000 rw-p 00000000 00:00 0 [stack]
21450:20131105:075642.315 ffffe000-fffff000 r-xp 00000000 00:00 0 [vdso]
21450:20131105:075642.315 ================================
21450:20131105:075642.315 Please consider attaching a disassembly listing to your bug report.
21450:20131105:075642.315 This listing can be produced with, e.g., objdump -DSswx zabbix_server.
21450:20131105:075642.315 ================================