-
Incident report
-
Resolution: Fixed
-
Critical
-
1.8.4
-
None
-
Zabbix 1.8.4 (stable), CentOS 5.5 (2.6.18-194.26.1.el5.i686) and CentOS 5.6 (2.6.18-238.5.1.el5.i686), IBM BladeCenter HS20 Server, 3GB RAM, Xeon CPU 3.20GHz - dual. MySQL-server-community-5.0.51a-0.rhel5
The Server process crashes. After the crash and manually starting the server - does not start and keeps crashing on any subsequent start.
This was identified as the cause:
9641:20110414:174107.219 Escalation cancelled: Trigger 'Timezone Verification -
{HOSTNAME}' disabled.
9641:20110414:174107.336 Escalation cancelled: Host 'pdc' disabled.
9641:20110414:174107.451 Got signal [signal:11(SIGSEGV),reason:1,refaddr:(nil)]. Crashing ...
...
9202:20110414:174107.464 One child process died (PID:9641,exitcode/signal:255). Exiting ...
The solution was to empty the "escalations" table, disable the Action related to escalated alerts and starting the Server.
Here is the initial crash log (prior messages not available):
30746:20110414:034809.640 Got signal [signal:11(SIGSEGV),reason:1,refaddr:(nil)]. Crashing ...
30746:20110414:034809.641 ====== Fatal information: ======
30746:20110414:034809.641 Program counter: 0x619333
30746:20110414:034809.641 === Registers: ===
30746:20110414:034809.641 gs = 33 = 51 = 51
30746:20110414:034809.641 fs = 0 = 0 = 0
30746:20110414:034809.641 es = c040007b = 3225419899 = -1069547397
30746:20110414:034809.641 ds = c040007b = 3225419899 = -1069547397
30746:20110414:034809.641 edi = 0 = 0 = 0
30746:20110414:034809.641 esi = 0 = 0 = 0
30746:20110414:034809.641 ebp = bf941b88 = 3214154632 = -1080812664
30746:20110414:034809.641 esp = bf941b6c = 3214154604 = -1080812692
30746:20110414:034809.641 ebx = 6fdff4 = 7331828 = 7331828
30746:20110414:034809.641 edx = 1d = 29 = 29
30746:20110414:034809.641 ecx = 0 = 0 = 0
30746:20110414:034809.641 eax = 0 = 0 = 0
30746:20110414:034809.641 trapno = e = 14 = 14
30746:20110414:034809.641 err = 4 = 4 = 4
30746:20110414:034809.641 eip = 619333 = 6394675 = 6394675
30746:20110414:034809.641 cs = 73 = 115 = 115
30746:20110414:034809.641 efl = 210246 = 2163270 = 2163270
30746:20110414:034809.641 uesp = bf941b6c = 3214154604 = -1080812692
30746:20110414:034809.641 ss = 7b = 123 = 123
30746:20110414:034809.641 === Stack frame: ===
30746:20110414:034809.641 +0x40(%ebp) = ebp + 64 = 00000000 = 0 = 0
30746:20110414:034809.642 +0x3c(%ebp) = ebp + 60 = 000007f7 = 2039 = 2039
30746:20110414:034809.642 +0x38(%ebp) = ebp + 56 = 0817160e = 135730702 = 135730702
30746:20110414:034809.642 +0x34(%ebp) = ebp + 52 = 080cfe55 = 135069269 = 135069269
30746:20110414:034809.642 +0x30(%ebp) = ebp + 48 = bf941ce8 = 3214154984 = -1080812312
30746:20110414:034809.642 +0x2c(%ebp) = ebp + 44 = bf941e34 = 3214155316 = -1080811980
30746:20110414:034809.642 +0x28(%ebp) = ebp + 40 = 0000000b = 11 = 11
30746:20110414:034809.642 +0x24(%ebp) = ebp + 36 = 096f53a6 = 158290854 = 158290854
30746:20110414:034809.642 +0x20(%ebp) = ebp + 32 = 00000002 = 2 = 2
30746:20110414:034809.642 +0x1c(%ebp) = ebp + 28 = 52fdd800 = 1392367616 = 1392367616
30746:20110414:034809.642 +0x18(%ebp) = ebp + 24 = 096efe88 = 158269064 = 158269064
30746:20110414:034809.642 +0x14(%ebp) = ebp + 20 = 00000025 = 37 = 37
30746:20110414:034809.642 +0x10(%ebp) = ebp + 16 = 00000000 = 0 = 0
30746:20110414:034809.642 +0x0c(%ebp) = ebp + 12 = 096f5396 = 158290838 = 158290838
30746:20110414:034809.642 +0x08(%ebp) = ebp + 8 = 00000000 = 0 = 0 <--- call arguments
30746:20110414:034809.642 +0x04(%ebp) = ebp + 4 = 080c0cea <--- return address
30746:20110414:034809.642 (%ebp) = ebp = bf941bb8 <--- saved ebp value
30746:20110414:034809.642 0x04(%ebp) = ebp - 4 = 081718ea = 135731434 = 135731434 <-- local variables
30746:20110414:034809.642 -0x08(%ebp) = ebp - 8 = 00000000 = 0 = 0
30746:20110414:034809.642 -0x0c(%ebp) = ebp - 12 = 00000009 = 9 = 9
30746:20110414:034809.642 -0x10(%ebp) = ebp - 16 = 096f4ec8 = 158289608 = 158289608
30746:20110414:034809.642 -0x14(%ebp) = ebp - 20 = 006fdff4 = 7331828 = 7331828
30746:20110414:034809.642 -0x18(%ebp) = ebp - 24 = 00000000 = 0 = 0
30746:20110414:034809.642 -0x1c(%ebp) = ebp - 28 = 00619075 = 6393973 = 6393973
30746:20110414:034809.642 -0x20(%ebp) = ebp - 32 = bf941b88 = 3214154632 = -1080812664
30746:20110414:034809.643 -0x24(%ebp) = ebp - 36 = 0880cd00 = 142658816 = 142658816
30746:20110414:034809.643 -0x28(%ebp) = ebp - 40 = 0000adb8 = 44472 = 44472
30746:20110414:034809.643 -0x2c(%ebp) = ebp - 44 = 00000000 = 0 = 0
30746:20110414:034809.643 -0x30(%ebp) = ebp - 48 = 00000000 = 0 = 0
30746:20110414:034809.643 -0x34(%ebp) = ebp - 52 = 00000000 = 0 = 0
30746:20110414:034809.643 -0x38(%ebp) = ebp - 56 = 00000000 = 0 = 0
30746:20110414:034809.643 -0x3c(%ebp) = ebp - 60 = 00000000 = 0 = 0
30746:20110414:034809.643 -0x40(%ebp) = ebp - 64 = 00000000 = 0 = 0
30746:20110414:034809.643 === Backtrace: ===
30746:20110414:034809.670 14: /usr/local/zabbix/sbin/zabbix_server(print_fatal_info+0x311) [0x80bc631]
30746:20110414:034809.670 13: /usr/local/zabbix/sbin/zabbix_server(child_signal_handler+0xec) [0x80bbbcc]
30746:20110414:034809.670 12: [0x728440]
30746:20110414:034809.670 11: /lib/libc.so.6(strlen+0x33) [0x619333]
30746:20110414:034809.670 10: /lib/libc.so.6(__strdup+0x25) [0x619075]
30746:20110414:034809.670 9: /usr/local/zabbix/sbin/zabbix_server(zbx_strdup2+0x2a) [0x80c0cea]
30746:20110414:034809.670 8: /usr/local/zabbix/sbin/zabbix_server(substitute_simple_macros+0x675) [0x80cfe55]
30746:20110414:034809.670 7: /usr/local/zabbix/sbin/zabbix_server(substitute_macros+0xce) [0x80d229e]
30746:20110414:034809.670 6: /usr/local/zabbix/sbin/zabbix_server [0x80a6594]
30746:20110414:034809.670 5: /usr/local/zabbix/sbin/zabbix_server(main_escalator_loop+0x109) [0x80a6ff9]
30746:20110414:034809.671 4: /usr/local/zabbix/sbin/zabbix_server(MAIN_ZABBIX_ENTRY+0x8b6) [0x808a426]
30746:20110414:034809.671 3: /usr/local/zabbix/sbin/zabbix_server(daemon_start+0x264) [0x80bb974]
30746:20110414:034809.671 2: /usr/local/zabbix/sbin/zabbix_server(main+0x141) [0x8089ac1]
30746:20110414:034809.671 1: /lib/libc.so.6(__libc_start_main+0xdc) [0x5bee9c]
30746:20110414:034809.671 0: /usr/local/zabbix/sbin/zabbix_server(exit+0xed) [0x807d381]
30746:20110414:034809.671 === Memory map: ===
30746:20110414:034809.671 00110000-001a8000 r-xp 00000000 08:03 15163873 /usr/lib/libnetsnmp.so.10.0.3
30746:20110414:034809.671 001a8000-001aa000 rwxp 00098000 08:03 15163873 /usr/lib/libnetsnmp.so.10.0.3
30746:20110414:034809.671 001aa000-001c9000 rwxp 001aa000 00:00 0
30746:20110414:034809.671 001c9000-002a5000 r-xp 00000000 08:03 15161775 /usr/lib/libOpenIPMI.so.0.0.5.#prelink#.gLfBaC (deleted)
30746:20110414:034809.671 002a5000-002b4000 rwxp 000db000 08:03 15161775 /usr/lib/libOpenIPMI.so.0.0.5.#prelink#.gLfBaC (deleted)
30746:20110414:034809.671 002b4000-002b6000 rwxp 002b4000 00:00 0
30746:20110414:034809.671 002b6000-002bd000 r-xp 00000000 08:03 15163911 /usr/lib/libOpenIPMIutils.so.0.0.1.#prelink#.Rh5d5n (deleted)
30746:20110414:034809.671 002bd000-002be000 rwxp 00006000 08:03 15163911 /usr/lib/libOpenIPMIutils.so.0.0.1.#prelink#.Rh5d5n (deleted)
30746:20110414:034809.671 002be000-002c4000 r-xp 00000000 08:03 15162166 /usr/lib/libgdbm.so.2.0.0
30746:20110414:034809.671 002c4000-002c5000 rwxp 00005000 08:03 15162166 /usr/lib/libgdbm.so.2.0.0
30746:20110414:034809.671 002c5000-002cf000 r-xp 00000000 08:03 8675946 /lib/libnss_files-2.5.so
30746:20110414:034809.671 002cf000-002d0000 r-xp 00009000 08:03 8675946 /lib/libnss_files-2.5.so
30746:20110414:034809.671 002d0000-002d1000 rwxp 0000a000 08:03 8675946 /lib/libnss_files-2.5.so
30746:20110414:034809.671 00301000-00306000 r-xp 00000000 08:03 15162407 /usr/lib/libOpenIPMIposix.so.0.0.1.#prelink#.xcFBeT (deleted)
30746:20110414:034809.672 00306000-00307000 rwxp 00004000 08:03 15162407 /usr/lib/libOpenIPMIposix.so.0.0.1.#prelink#.xcFBeT (deleted)
30746:20110414:034809.672 00307000-00347000 r-xp 00000000 08:03 15165238 /usr/lib/libncurses.so.5.5
30746:20110414:034809.672 00347000-0034f000 rwxp 00040000 08:03 15165238 /usr/lib/libncurses.so.5.5
30746:20110414:034809.672 0034f000-00350000 rwxp 0034f000 00:00 0
30746:20110414:034809.672 004da000-00515000 r-xp 00000000 08:03 15167749 /usr/lib/libcurl.so.3.0.0
30746:20110414:034809.672 00515000-00516000 rwxp 0003b000 08:03 15167749 /usr/lib/libcurl.so.3.0.0
30746:20110414:034809.672 0058a000-005a5000 r-xp 00000000 08:03 8675055 /lib/ld-2.5.so
30746:20110414:034809.672 005a5000-005a6000 r-xp 0001a000 08:03 8675055 /lib/ld-2.5.so
30746:20110414:034809.672 005a6000-005a7000 rwxp 0001b000 08:03 8675055 /lib/ld-2.5.so
30746:20110414:034809.672 005a9000-006fc000 r-xp 00000000 08:03 8675056 /lib/libc-2.5.so
30746:20110414:034809.672 006fc000-006fe000 r-xp 00152000 08:03 8675056 /lib/libc-2.5.so
30746:20110414:034809.672 006fe000-006ff000 rwxp 00154000 08:03 8675056 /lib/libc-2.5.so
30746:20110414:034809.672 006ff000-00702000 rwxp 006ff000 00:00 0
30746:20110414:034809.672 00704000-0070b000 r-xp 00000000 08:03 8675094 /lib/libwrap.so.0.7.6
30746:20110414:034809.672 0070b000-0070c000 rwxp 00007000 08:03 8675094 /lib/libwrap.so.0.7.6
30746:20110414:034809.672 00728000-00729000 r-xp 00728000 00:00 0 [vdso]
30746:20110414:034809.672 0072e000-00730000 r-xp 00000000 08:03 8675067 /lib/libcom_err.so.2.1
30746:20110414:034809.672 00730000-00731000 rwxp 00001000 08:03 8675067 /lib/libcom_err.so.2.1
30746:20110414:034809.672 00751000-00763000 r-xp 00000000 08:03 15161916 /usr/lib/libz.so.1.2.3
30746:20110414:034809.672 00763000-00764000 rwxp 00011000 08:03 15161916 /usr/lib/libz.so.1.2.3
30746:20110414:034809.672 00771000-00786000 r-xp 00000000 08:03 8675092 /lib/libnsl-2.5.so
30746:20110414:034809.672 00786000-00787000 r-xp 00014000 08:03 8675092 /lib/libnsl-2.5.so
30746:20110414:034809.673 00787000-00788000 rwxp 00015000 08:03 8675092 /lib/libnsl-2.5.so
30746:20110414:034809.673 00788000-0078a000 rwxp 00788000 00:00 0
30746:20110414:034809.673 0078b000-007c6000 r-xp 00000000 08:03 8675074 /lib/libsepol.so.1
30746:20110414:034809.673 007c6000-007c7000 rwxp 0003b000 08:03 8675074 /lib/libsepol.so.1
30746:20110414:034809.673 007c7000-007d1000 rwxp 007c7000 00:00 0
30746:20110414:034809.673 007d3000-007de000 r-xp 00000000 08:03 8675062 /lib/libgcc_s-4.1.2-20080825.so.1
30746:20110414:034809.673 007de000-007df000 rwxp 0000a000 08:03 8675062 /lib/libgcc_s-4.1.2-20080825.so.1
30746:20110414:034809.673 007e9000-00819000 r-xp 00000000 08:03 15166969 /usr/lib/libidn.so.11.5.19
30746:20110414:034809.673 00819000-0081a000 rwxp 0002f000 08:03 15166969 /usr/lib/libidn.so.11.5.19
30746:20110414:034809.673 009a0000-009a2000 r-xp 00000000 08:03 8675246 /lib/libkeyutils-1.2.so
30746:20110414:034809.673 009a2000-009a3000 rwxp 00001000 08:03 8675246 /lib/libkeyutils-1.2.so
30746:20110414:034809.673 009a5000-009b5000 r-xp 00000000 08:03 8675260 /lib/libresolv-2.5.so
30746:20110414:034809.673 009b5000-009b6000 r-xp 0000f000 08:03 8675260 /lib/libresolv-2.5.so
30746:20110414:034809.673 009b6000-009b7000 rwxp 00010000 08:03 8675260 /lib/libresolv-2.5.so
30746:20110414:034809.673 009b7000-009b9000 rwxp 009b7000 00:00 0
30746:20110414:034809.673 009bb000-009e8000 r-xp 00000000 08:03 15165295 /usr/lib/libgssapi_krb5.so.2.2
30746:20110414:034809.673 009e8000-009e9000 rwxp 0002d000 08:03 15165295 /usr/lib/libgssapi_krb5.so.2.2
30746:20110414:034809.673 009eb000-00a2f000 r-xp 00000000 08:03 8675270 /lib/libssl.so.0.9.8e
30746:20110414:034809.673 00a2f000-00a33000 rwxp 00043000 08:03 8675270 /lib/libssl.so.0.9.8e
30746:20110414:034809.674 00a35000-00a5a000 r-xp 00000000 08:03 15165232 /usr/lib/libk5crypto.so.3.1
30746:20110414:034809.674 00a5a000-00a5b000 rwxp 00025000 08:03 15165232 /usr/lib/libk5crypto.so.3.1
30746:20110414:034809.674 00a63000-00a79000 r-xp 00000000 08:03 8675268 /lib/libselinux.so.1
30746:20110414:034809.674 00a79000-00a7b000 rwxp 00015000 08:03 8675268 /lib/libselinux.so.1
30746:20110414:034809.674 00a7d000-00a80000 r-xp 00000000 08:03 8675226 /lib/libdl-2.5.so
30746:20110414:034809.674 00a80000-00a81000 r-xp 00002000 08:03 8675226 /lib/libdl-2.5.so
30746:20110414:034809.674 00a81000-00a82000 rwxp 00003000 08:03 8675226 /lib/libdl-2.5.so
30746:20110414:034809.674 00a84000-00b17000 r-xp 00000000 08:03 15165274 /usr/lib/libkrb5.so.3.3
30746:20110414:034809.674 00b17000-00b1a000 rwxp 00092000 08:03 15165274 /usr/lib/libkrb5.so.3.3
30746:20110414:034809.674 00b1c000-00b43000 r-xp 00000000 08:03 8675271 /lib/libm-2.5.so
30746:20110414:034809.674 00b43000-00b44000 r-xp 00026000 08:03 8675271 /lib/libm-2.5.so
30746:20110414:034809.674 00b44000-00b45000 rwxp 00027000 08:03 8675271 /lib/libm-2.5.so
30746:20110414:034809.674 00b47000-00c71000 r-xp 00000000 08:03 8675269 /lib/libcrypto.so.0.9.8e
30746:20110414:034809.674 00c71000-00c84000 rwxp 00129000 08:03 8675269 /lib/libcrypto.so.0.9.8e
30746:20110414:034809.674 00c84000-00c88000 rwxp 00c84000 00:00 0
30746:20110414:034809.674 00d77000-00d7f000 r-xp 00000000 08:03 15163516 /usr/lib/libkrb5support.so.0.1
30746:20110414:034809.674 00d7f000-00d80000 rwxp 00007000 08:03 15163516 /usr/lib/libkrb5support.so.0.1
30746:20110414:034809.674 08048000-08272000 r-xp 00000000 08:03 15519052 /usr/local/zabbix/sbin/zabbix_server
30746:20110414:034809.674 08272000-082cd000 rw-p 00229000 08:03 15519052 /usr/local/zabbix/sbin/zabbix_server
30746:20110414:034809.675 082cd000-082d4000 rw-p 082cd000 00:00 0
30746:20110414:034809.675 096e8000-09738000 rw-p 096e8000 00:00 0 [heap]
30746:20110414:034809.675 b5b32000-b5b33000 rw-p b5b32000 00:00 0
30746:20110414:034809.675 b5b33000-b5c67000 rw-s 00000000 00:09 195428363 /SYSV7303cd72 (deleted)
30746:20110414:034809.675 b5c67000-b6334000 rw-s 00000000 00:09 195395594 /SYSV6703cd72 (deleted)
30746:20110414:034809.675 b6334000-b6735000 rw-s 00000000 00:09 195362825 /SYSV7403cd72 (deleted)
30746:20110414:034809.675 b6735000-b7736000 rw-s 00000000 00:09 195330056 /SYSV7803cd72 (deleted)
30746:20110414:034809.675 b7736000-b7f3e000 rw-s 00000000 00:09 195297287 /SYSV6803cd72 (deleted)
30746:20110414:034809.675 b7f3e000-b7f45000 rw-p b7f3e000 00:00 0
30746:20110414:034809.675 b7f4a000-b7f4b000 rw-p b7f4a000 00:00 0
30746:20110414:034809.675 bf91f000-bf944000 rw-p bffd9000 00:00 0 [stack]
30746:20110414:034809.675 ================================
30746:20110414:034809.675 Please consider attaching a disassembly listing to your bug report.
30746:20110414:034809.675 This listing can be produced with, e.g., objdump -D -S zabbix_server.
30746:20110414:034809.675 ================================
30313:20110414:034809.744 One child process died (PID:30746,exitcode/signal:255). Exiting ...
30464:20110414:034809.786 NODE 1: Received history_uint from node 3 for node 3 datalen 360
30313:20110414:034811.987 Syncing history data...
30313:20110414:034812.107 Division by zero. Cannot evaluate expression [0.000000/0.000000]
30313:20110414:034812.108 Division by zero. Cannot evaluate expression [0.000000/0.000000]
30313:20110414:034812.108 Expression [(0)/(0)>1.1] cannot be evaluated: Division by zero. Cannot evaluate expression [0.000000/0.000000]
30313:20110414:034812.108 Division by zero. Cannot evaluate expression [0.000000/0.000000]
30313:20110414:034812.108 Division by zero. Cannot evaluate expression [0.000000/0.000000]
30313:20110414:034812.108 Expression [(0)/(0)>2] cannot be evaluated: Division by zero. Cannot evaluate expression [0.000000/0.000000]
30313:20110414:034812.168 Syncing history data... done.
30313:20110414:034812.168 Syncing trends data...
30313:20110414:034827.437 Syncing trends data... done.
30313:20110414:034827.438 Zabbix Server stopped. Zabbix 1.8.4 (revision 16604).
The disassembly listing is available upon request (due to attachment size limit of 10MB - can not be provided here, zipped size - 17MB).
- duplicates
-
ZBX-3628 server crash in escalator while expanding macros
- Closed