Critical Zabbix proxy is crashing after 5-10 minutes of uptime. There are were 16 crashes for last 3 weeks (15 times on unreachable poller and 2 on configuration syncer). Example 1:
27012:20170109:112653.493 proxy #1 started [configuration syncer #1] ... ... ... 27012:20170109:115712.583 received configuration data from server at "10.5.5.239", datalen 62104 27012:20170109:115712.603 Got signal [signal:11(SIGSEGV),reason:1,refaddr:0x58734f1d]. Crashing ... 27012:20170109:115712.603 ====== Fatal information: ====== 27012:20170109:115712.603 Program counter: 0x8090eac 27012:20170109:115712.603 === Registers: === 27012:20170109:115712.603 gs = 33 = 51 = 51 27012:20170109:115712.603 fs = c0100000 = 3222274048 = -1072693248 27012:20170109:115712.603 es = ffff007b = 4294901883 = -65413 27012:20170109:115712.603 ds = 7b = 123 = 123 27012:20170109:115712.603 edi = 0 = 0 = 0 27012:20170109:115712.603 esi = 0 = 0 = 0 27012:20170109:115712.603 ebp = bfbb4588 = 3216721288 = -1078246008 27012:20170109:115712.603 esp = bfbb4540 = 3216721216 = -1078246080 27012:20170109:115712.603 ebx = 0 = 0 = 0 27012:20170109:115712.603 edx = b5bf1000 = 3049197568 = -1245769728 27012:20170109:115712.603 ecx = b5bf1000 = 3049197568 = -1245769728 27012:20170109:115712.603 eax = 58734f1d = 1483951901 = 1483951901 27012:20170109:115712.603 trapno = e = 14 = 14 27012:20170109:115712.603 err = 4 = 4 = 4 27012:20170109:115712.603 eip = 8090eac = 134811308 = 134811308 27012:20170109:115712.603 cs = 73 = 115 = 115 27012:20170109:115712.603 efl = 210202 = 2163202 = 2163202 27012:20170109:115712.603 uesp = bfbb4540 = 3216721216 = -1078246080 27012:20170109:115712.603 ss = 7b = 123 = 123 27012:20170109:115712.603 === Stack frame: === 27012:20170109:115712.603 +0x40(%ebp) = ebp + 64 = 1a262802 = 438708226 = 438708226 27012:20170109:115712.603 +0x3c(%ebp) = ebp + 60 = 41d61cd4 = 1104551124 = 1104551124 27012:20170109:115712.603 +0x38(%ebp) = ebp + 56 = 1a26273d = 438708029 = 438708029 27012:20170109:115712.603 +0x34(%ebp) = ebp + 52 = 41d61cd4 = 1104551124 = 1104551124 27012:20170109:115712.603 +0x30(%ebp) = ebp + 48 = 1a2620c9 = 438706377 = 438706377 27012:20170109:115712.603 +0x2c(%ebp) = ebp + 44 = 0831dd88 = 137485704 = 137485704 27012:20170109:115712.603 +0x28(%ebp) = ebp + 40 = 08366f88 = 137785224 = 137785224 27012:20170109:115712.603 +0x24(%ebp) = ebp + 36 = ffffffff = 4294967295 = -1 27012:20170109:115712.603 +0x20(%ebp) = ebp + 32 = 3ea19400 = 1050776576 = 1050776576 27012:20170109:115712.603 +0x1c(%ebp) = ebp + 28 = 00000000 = 0 = 0 27012:20170109:115712.603 +0x18(%ebp) = ebp + 24 = 405881ce = 1079542222 = 1079542222 27012:20170109:115712.603 +0x14(%ebp) = ebp + 20 = 015a2559 = 22685017 = 22685017 27012:20170109:115712.603 +0x10(%ebp) = ebp + 16 = 0824216f = 136585583 = 136585583 27012:20170109:115712.603 +0x0c(%ebp) = ebp + 12 = 08242243 = 136585795 = 136585795 27012:20170109:115712.603 +0x08(%ebp) = ebp + 8 = b5bf1000 = 3049197568 = -1245769728 <--- call arguments 27012:20170109:115712.603 +0x04(%ebp) = ebp + 4 = 0808c57f <--- return address 27012:20170109:115712.604 (%ebp) = ebp = bfbb48d8 <--- saved ebp value 27012:20170109:115712.604 -0x04(%ebp) = ebp - 4 = 082dd864 = 137222244 = 137222244 <--- local variables 27012:20170109:115712.604 -0x08(%ebp) = ebp - 8 = 00000000 = 0 = 0 27012:20170109:115712.604 -0x0c(%ebp) = ebp - 12 = 00000000 = 0 = 0 27012:20170109:115712.604 -0x10(%ebp) = ebp - 16 = bfbb4588 = 3216721288 = -1078246008 27012:20170109:115712.604 -0x14(%ebp) = ebp - 20 = 00000018 = 24 = 24 27012:20170109:115712.604 -0x18(%ebp) = ebp - 24 = bfbb4580 = 3216721280 = -1078246016 27012:20170109:115712.604 -0x1c(%ebp) = ebp - 28 = 00000000 = 0 = 0 27012:20170109:115712.604 -0x20(%ebp) = ebp - 32 = 00000000 = 0 = 0 27012:20170109:115712.604 -0x24(%ebp) = ebp - 36 = ffffffff = 4294967295 = -1 27012:20170109:115712.604 -0x28(%ebp) = ebp - 40 = ffffffff = 4294967295 = -1 27012:20170109:115712.604 -0x2c(%ebp) = ebp - 44 = 00000000 = 0 = 0 27012:20170109:115712.604 -0x30(%ebp) = ebp - 48 = 00000000 = 0 = 0 27012:20170109:115712.604 -0x34(%ebp) = ebp - 52 = 00000000 = 0 = 0 27012:20170109:115712.604 -0x38(%ebp) = ebp - 56 = 00000594 = 1428 = 1428 27012:20170109:115712.604 -0x3c(%ebp) = ebp - 60 = 00000000 = 0 = 0 27012:20170109:115712.604 -0x40(%ebp) = ebp - 64 = b5bf10b8 = 3049197752 = -1245769544 27012:20170109:115712.604 === Backtrace: === 27012:20170109:115712.604 11: [0x809af90] 27012:20170109:115712.604 10: [0x809b800] 27012:20170109:115712.604 9: [0xffffe410] 27012:20170109:115712.604 8: [0x808c57f] 27012:20170109:115712.604 7: [0x80b45e5] 27012:20170109:115712.604 6: [0x804e45b] 27012:20170109:115712.604 5: [0x809bd68] 27012:20170109:115712.604 4: [0x8049200] 27012:20170109:115712.604 3: [0x809a01f] 27012:20170109:115712.604 2: [0x80499a2] 27012:20170109:115712.604 1: [0x81c04d9] 27012:20170109:115712.604 0: [0x80482a1] 27012:20170109:115712.604 === Memory map: === 27012:20170109:115712.604 08048000-082ce000 r-xp 00000000 08:01 886456 /zabbix/sbin/zabbix_proxy 27012:20170109:115712.604 082ce000-082dd000 rw-p 00286000 08:01 886456 /zabbix/sbin/zabbix_proxy 27012:20170109:115712.604 082dd000-083f2000 rw-p 00000000 00:00 0 [heap] 27012:20170109:115712.604 b4bee000-b4bef000 rw-p 00000000 00:00 0 27012:20170109:115712.604 b4bef000-b4bf0000 ---p 00000000 00:00 0 27012:20170109:115712.604 b4bf0000-b53ef000 rw-p 00000000 00:00 0 27012:20170109:115712.604 b53ef000-b53f0000 ---p 00000000 00:00 0 27012:20170109:115712.604 b53f0000-b5bef000 rw-p 00000000 00:00 0 27012:20170109:115712.604 b5bef000-b5bf1000 rw-s 00000000 00:04 557060 /SYSV53010002 (deleted) 27012:20170109:115712.604 b5bf1000-b5d25000 rw-s 00000000 00:04 524291 /SYSV73010002 (deleted) 27012:20170109:115712.604 b5d25000-b63f2000 rw-s 00000000 00:04 491522 /SYSV67010002 (deleted) 27012:20170109:115712.604 b63f2000-b67f2000 rw-s 00000000 00:04 458753 /SYSV48010002 (deleted) 27012:20170109:115712.604 b67f2000-b77f2000 rw-s 00000000 00:04 425984 /SYSV68010002 (deleted) 27012:20170109:115712.604 b77f2000-b77f3000 rw-p 00000000 00:00 0 27012:20170109:115712.604 bfba3000-bfbb8000 rw-p 00000000 00:00 0 [stack] 27012:20170109:115712.604 ffffe000-fffff000 r-xp 00000000 00:00 0 [vdso] 27012:20170109:115712.604 ================================ 27012:20170109:115712.604 Please consider attaching a disassembly listing to your bug report. 27012:20170109:115712.604 This listing can be produced with, e.g., objdump -DSswx zabbix_proxy. 27012:20170109:115712.604 ================================ 26877:20170109:115712.605 One child process died (PID:27012,exitcode/signal:1). Exiting ... 26877:20170109:115714.616 syncing history data... 26877:20170109:115714.616 syncing history data done 26877:20170109:115714.616 Zabbix Proxy stopped. Zabbix 3.0.6rc1 (revision 64124).
Example 2:
3904:20170124:083106.722 proxy #9 started [unreachable poller #1]
... ... ...
3898:20170124:083440.625 Connection restored.
3904:20170124:083503.377 Got signal [signal:11(SIGSEGV),reason:1,refaddr:0x5886e798]. Crashing ...
3904:20170124:083503.377 ====== Fatal information: ======
3904:20170124:083503.377 Program counter: 0x8090839
3904:20170124:083503.377 === Registers: ===
3904:20170124:083503.377 gs = 33 = 51 = 51
3904:20170124:083503.377 fs = 0 = 0 = 0
3904:20170124:083503.377 es = 7b = 123 = 123
3904:20170124:083503.377 ds = 7b = 123 = 123
3904:20170124:083503.377 edi = b5bfe000 = 3049250816 = -1245716480
3904:20170124:083503.377 esi = c = 12 = 12
3904:20170124:083503.377 ebp = bfab63c8 = 3215680456 = -1079286840
3904:20170124:083503.377 esp = bfab63bc = 3215680444 = -1079286852
3904:20170124:083503.377 ebx = b5bfe778 = 3049252728 = -1245714568
3904:20170124:083503.377 edx = 0 = 0 = 0
3904:20170124:083503.377 ecx = c = 12 = 12
3904:20170124:083503.377 eax = 5886e768 = 1485236072 = 1485236072
3904:20170124:083503.377 trapno = e = 14 = 14
3904:20170124:083503.377 err = 4 = 4 = 4
3904:20170124:083503.377 eip = 8090839 = 134809657 = 134809657
3904:20170124:083503.377 cs = 73 = 115 = 115
3904:20170124:083503.377 efl = 210246 = 2163270 = 2163270
3904:20170124:083503.377 uesp = bfab63bc = 3215680444 = -1079286852
3904:20170124:083503.377 ss = 7b = 123 = 123
3904:20170124:083503.377 === Stack frame: ===
3904:20170124:083503.377 +0x40(%ebp) = ebp + 64 = b5bfe000 = 3049250816 = -1245716480
3904:20170124:083503.377 +0x3c(%ebp) = ebp + 60 = 00000035 = 53 = 53
3904:20170124:083503.377 +0x38(%ebp) = ebp + 56 = 08242d97 = 136588695 = 136588695
3904:20170124:083503.377 +0x34(%ebp) = ebp + 52 = 08091e8a = 134815370 = 134815370
3904:20170124:083503.377 +0x30(%ebp) = ebp + 48 = bfab6418 = 3215680536 = -1079286760
3904:20170124:083503.377 +0x2c(%ebp) = ebp + 44 = b5bfe0d8 = 3049251032 = -1245716264
3904:20170124:083503.377 +0x28(%ebp) = ebp + 40 = b5bfe780 = 3049252736 = -1245714560
3904:20170124:083503.377 +0x24(%ebp) = ebp + 36 = 082ea0f8 = 137273592 = 137273592
3904:20170124:083503.377 +0x20(%ebp) = ebp + 32 = 082ea128 = 137273640 = 137273640
3904:20170124:083503.377 +0x1c(%ebp) = ebp + 28 = 00000000 = 0 = 0
3904:20170124:083503.377 +0x18(%ebp) = ebp + 24 = 00000000 = 0 = 0
3904:20170124:083503.377 +0x14(%ebp) = ebp + 20 = 00000000 = 0 = 0
3904:20170124:083503.377 +0x10(%ebp) = ebp + 16 = 00000000 = 0 = 0
3904:20170124:083503.377 +0x0c(%ebp) = ebp + 12 = 00000000 = 0 = 0
3904:20170124:083503.377 +0x08(%ebp) = ebp + 8 = 00000000 = 0 = 0 <--- call arguments
3904:20170124:083503.377 +0x04(%ebp) = ebp + 4 = 080911fb <--- return address
3904:20170124:083503.377 (%ebp) = ebp = bfab63f8 <--- saved ebp value
3904:20170124:083503.378 -0x04(%ebp) = ebp - 4 = b5bfe780 = 3049252736 = -1245714560 <--- local variables
3904:20170124:083503.378 -0x08(%ebp) = ebp - 8 = b5bfe780 = 3049252736 = -1245714560
3904:20170124:083503.378 -0x0c(%ebp) = ebp - 12 = b5bfe000 = 3049250816 = -1245716480
3904:20170124:083503.378 -0x10(%ebp) = ebp - 16 = 46505845 = 1179670597 = 1179670597
3904:20170124:083503.378 -0x14(%ebp) = ebp - 20 = 00000000 = 0 = 0
3904:20170124:083503.378 -0x18(%ebp) = ebp - 24 = 00000000 = 0 = 0
3904:20170124:083503.378 -0x1c(%ebp) = ebp - 28 = 00000000 = 0 = 0
3904:20170124:083503.378 -0x20(%ebp) = ebp - 32 = 00000000 = 0 = 0
3904:20170124:083503.378 -0x24(%ebp) = ebp - 36 = 00000000 = 0 = 0
3904:20170124:083503.378 -0x28(%ebp) = ebp - 40 = 00000000 = 0 = 0
3904:20170124:083503.378 -0x2c(%ebp) = ebp - 44 = 00000000 = 0 = 0
3904:20170124:083503.378 -0x30(%ebp) = ebp - 48 = 00000000 = 0 = 0
3904:20170124:083503.378 -0x34(%ebp) = ebp - 52 = 00000000 = 0 = 0
3904:20170124:083503.378 -0x38(%ebp) = ebp - 56 = 00000000 = 0 = 0
3904:20170124:083503.378 -0x3c(%ebp) = ebp - 60 = 00000000 = 0 = 0
3904:20170124:083503.378 -0x40(%ebp) = ebp - 64 = 00000000 = 0 = 0
3904:20170124:083503.378 === Backtrace: ===
3904:20170124:083503.410 19: [0x809af90]
3904:20170124:083503.410 18: [0x809b800]
3904:20170124:083503.410 17: [0xffffe410]
3904:20170124:083503.410 16: [0x80911fb]
3904:20170124:083503.410 15: [0x8091e8a]
3904:20170124:083503.410 14: [0x8094ae5]
3904:20170124:083503.410 13: [0x8094caa]
3904:20170124:083503.410 12: [0x8092021]
3904:20170124:083503.410 11: [0x807f412]
3904:20170124:083503.410 10: [0x807f596]
3904:20170124:083503.410 9: [0x807f787]
3904:20170124:083503.410 8: [0x807f962]
3904:20170124:083503.410 7: [0x8055362]
3904:20170124:083503.410 6: [0x8055d8a]
3904:20170124:083503.410 5: [0x809bd68]
3904:20170124:083503.410 4: [0x8048eee]
3904:20170124:083503.410 3: [0x809a01f]
3904:20170124:083503.410 2: [0x80499a2]
3904:20170124:083503.410 1: [0x81c04d9]
3904:20170124:083503.410 0: [0x80482a1]
3904:20170124:083503.410 === Memory map: ===
3904:20170124:083503.411 08048000-082ce000 r-xp 00000000 08:01 886456 /zabbix/sbin/zabbix_proxy
3904:20170124:083503.411 082ce000-082dd000 rw-p 00286000 08:01 886456 /zabbix/sbin/zabbix_proxy
3904:20170124:083503.411 082dd000-083a7000 rw-p 00000000 00:00 0 [heap]
3904:20170124:083503.411 b4bfb000-b4bfc000 rw-p 00000000 00:00 0
3904:20170124:083503.411 b4bfc000-b4bfd000 ---p 00000000 00:00 0
3904:20170124:083503.411 b4bfd000-b53fc000 rw-p 00000000 00:00 0
3904:20170124:083503.411 b53fc000-b53fd000 ---p 00000000 00:00 0
3904:20170124:083503.411 b53fd000-b5bfc000 rw-p 00000000 00:00 0
3904:20170124:083503.411 b5bfc000-b5bfe000 rw-s 00000000 00:04 196613 /SYSV53010002 (deleted)
3904:20170124:083503.411 b5bfe000-b5d32000 rw-s 00000000 00:04 163844 /SYSV73010002 (deleted)
3904:20170124:083503.411 b5d32000-b63ff000 rw-s 00000000 00:04 131075 /SYSV67010002 (deleted)
3904:20170124:083503.411 b63ff000-b67ff000 rw-s 00000000 00:04 98306 /SYSV48010002 (deleted)
3904:20170124:083503.411 b67ff000-b77ff000 rw-s 00000000 00:04 65537 /SYSV68010002 (deleted)
3904:20170124:083503.411 b77ff000-b7800000 rw-p 00000000 00:00 0
3904:20170124:083503.411 bfa83000-bfc29000 rw-p 00000000 00:00 0 [stack]
3904:20170124:083503.411 ffffe000-fffff000 r-xp 00000000 00:00 0 [vdso]
3904:20170124:083503.411 ================================
3904:20170124:083503.411 Please consider attaching a disassembly listing to your bug report.
3904:20170124:083503.411 This listing can be produced with, e.g., objdump -DSswx zabbix_proxy.
3904:20170124:083503.411 ================================
3800:20170124:083503.411 One child process died (PID:3904,exitcode/signal:1). Exiting ...
3800:20170124:083505.420 syncing history data...
3800:20170124:083505.420 syncing history data done
3800:20170124:083505.420 Zabbix Proxy stopped. Zabbix 3.0.6rc1 (revision 64124).
Note, the "cannot remove existing shared memory" errors appear sometimes in the proxy log (ZBX-5662). Assuming that this is effect of frequent crashes.
- duplicates
-
ZBX-5662 keys "vfs.dev.read|write" with ops, bps paremeters cause crash of agent under FreeBSD
-
- Closed
-