Uploaded image for project: 'ZABBIX BUGS AND ISSUES'
  1. ZABBIX BUGS AND ISSUES
  2. ZBX-11747

Zabbix proxy is crashing on unreachable poller or configuration syncer in 5-10 minutes after the start.

XMLWordPrintable

    • Icon: Incident report Incident report
    • Resolution: Incomplete
    • Icon: Critical Critical
    • None
    • 3.0.6
    • Proxy (P)
    • zabbix-proxy 3.0.6

      Zabbix proxy is crashing after 5-10 minutes of uptime. There are were 16 crashes for last 3 weeks (15 times on unreachable poller and 2 on configuration syncer). Example 1:

        27012:20170109:112653.493 proxy #1 started [configuration syncer #1]
      ... ... ...
       27012:20170109:115712.583 received configuration data from server at "10.5.5.239", datalen 62104
      27012:20170109:115712.603 Got signal [signal:11(SIGSEGV),reason:1,refaddr:0x58734f1d]. Crashing ...
       27012:20170109:115712.603 ====== Fatal information: ======
       27012:20170109:115712.603 Program counter: 0x8090eac
       27012:20170109:115712.603 === Registers: ===
       27012:20170109:115712.603 gs      =               33 =                   51 =                   51
       27012:20170109:115712.603 fs      =         c0100000 =           3222274048 =          -1072693248
       27012:20170109:115712.603 es      =         ffff007b =           4294901883 =               -65413
       27012:20170109:115712.603 ds      =               7b =                  123 =                  123
       27012:20170109:115712.603 edi     =                0 =                    0 =                    0
       27012:20170109:115712.603 esi     =                0 =                    0 =                    0
       27012:20170109:115712.603 ebp     =         bfbb4588 =           3216721288 =          -1078246008
       27012:20170109:115712.603 esp     =         bfbb4540 =           3216721216 =          -1078246080
       27012:20170109:115712.603 ebx     =                0 =                    0 =                    0
       27012:20170109:115712.603 edx     =         b5bf1000 =           3049197568 =          -1245769728
       27012:20170109:115712.603 ecx     =         b5bf1000 =           3049197568 =          -1245769728
       27012:20170109:115712.603 eax     =         58734f1d =           1483951901 =           1483951901
       27012:20170109:115712.603 trapno  =                e =                   14 =                   14
       27012:20170109:115712.603 err     =                4 =                    4 =                    4
       27012:20170109:115712.603 eip     =          8090eac =            134811308 =            134811308
       27012:20170109:115712.603 cs      =               73 =                  115 =                  115
       27012:20170109:115712.603 efl     =           210202 =              2163202 =              2163202
       27012:20170109:115712.603 uesp    =         bfbb4540 =           3216721216 =          -1078246080
       27012:20170109:115712.603 ss      =               7b =                  123 =                  123
       27012:20170109:115712.603 === Stack frame: ===
       27012:20170109:115712.603 +0x40(%ebp) = ebp + 64 = 1a262802 =  438708226 =   438708226
       27012:20170109:115712.603 +0x3c(%ebp) = ebp + 60 = 41d61cd4 = 1104551124 =  1104551124
       27012:20170109:115712.603 +0x38(%ebp) = ebp + 56 = 1a26273d =  438708029 =   438708029
       27012:20170109:115712.603 +0x34(%ebp) = ebp + 52 = 41d61cd4 = 1104551124 =  1104551124
       27012:20170109:115712.603 +0x30(%ebp) = ebp + 48 = 1a2620c9 =  438706377 =   438706377
       27012:20170109:115712.603 +0x2c(%ebp) = ebp + 44 = 0831dd88 =  137485704 =   137485704
       27012:20170109:115712.603 +0x28(%ebp) = ebp + 40 = 08366f88 =  137785224 =   137785224
       27012:20170109:115712.603 +0x24(%ebp) = ebp + 36 = ffffffff = 4294967295 =          -1
       27012:20170109:115712.603 +0x20(%ebp) = ebp + 32 = 3ea19400 = 1050776576 =  1050776576
       27012:20170109:115712.603 +0x1c(%ebp) = ebp + 28 = 00000000 =          0 =           0
       27012:20170109:115712.603 +0x18(%ebp) = ebp + 24 = 405881ce = 1079542222 =  1079542222
       27012:20170109:115712.603 +0x14(%ebp) = ebp + 20 = 015a2559 =   22685017 =    22685017
       27012:20170109:115712.603 +0x10(%ebp) = ebp + 16 = 0824216f =  136585583 =   136585583
       27012:20170109:115712.603 +0x0c(%ebp) = ebp + 12 = 08242243 =  136585795 =   136585795
       27012:20170109:115712.603 +0x08(%ebp) = ebp +  8 = b5bf1000 = 3049197568 = -1245769728 <--- call arguments
       27012:20170109:115712.603 +0x04(%ebp) = ebp +  4 = 0808c57f                            <--- return address
       27012:20170109:115712.604      (%ebp) = ebp      = bfbb48d8                            <--- saved ebp value
       27012:20170109:115712.604 -0x04(%ebp) = ebp -  4 = 082dd864 =  137222244 =   137222244 <--- local variables
       27012:20170109:115712.604 -0x08(%ebp) = ebp -  8 = 00000000 =          0 =           0
       27012:20170109:115712.604 -0x0c(%ebp) = ebp - 12 = 00000000 =          0 =           0
       27012:20170109:115712.604 -0x10(%ebp) = ebp - 16 = bfbb4588 = 3216721288 = -1078246008
       27012:20170109:115712.604 -0x14(%ebp) = ebp - 20 = 00000018 =         24 =          24
       27012:20170109:115712.604 -0x18(%ebp) = ebp - 24 = bfbb4580 = 3216721280 = -1078246016
       27012:20170109:115712.604 -0x1c(%ebp) = ebp - 28 = 00000000 =          0 =           0
       27012:20170109:115712.604 -0x20(%ebp) = ebp - 32 = 00000000 =          0 =           0
       27012:20170109:115712.604 -0x24(%ebp) = ebp - 36 = ffffffff = 4294967295 =          -1
       27012:20170109:115712.604 -0x28(%ebp) = ebp - 40 = ffffffff = 4294967295 =          -1
       27012:20170109:115712.604 -0x2c(%ebp) = ebp - 44 = 00000000 =          0 =           0
       27012:20170109:115712.604 -0x30(%ebp) = ebp - 48 = 00000000 =          0 =           0
       27012:20170109:115712.604 -0x34(%ebp) = ebp - 52 = 00000000 =          0 =           0
       27012:20170109:115712.604 -0x38(%ebp) = ebp - 56 = 00000594 =       1428 =        1428
       27012:20170109:115712.604 -0x3c(%ebp) = ebp - 60 = 00000000 =          0 =           0
       27012:20170109:115712.604 -0x40(%ebp) = ebp - 64 = b5bf10b8 = 3049197752 = -1245769544
       27012:20170109:115712.604 === Backtrace: ===
       27012:20170109:115712.604 11: [0x809af90]
       27012:20170109:115712.604 10: [0x809b800]
       27012:20170109:115712.604 9: [0xffffe410]
       27012:20170109:115712.604 8: [0x808c57f]
       27012:20170109:115712.604 7: [0x80b45e5]
       27012:20170109:115712.604 6: [0x804e45b]
       27012:20170109:115712.604 5: [0x809bd68]
       27012:20170109:115712.604 4: [0x8049200]
       27012:20170109:115712.604 3: [0x809a01f]
       27012:20170109:115712.604 2: [0x80499a2]
       27012:20170109:115712.604 1: [0x81c04d9]
       27012:20170109:115712.604 0: [0x80482a1]
       27012:20170109:115712.604 === Memory map: ===
       27012:20170109:115712.604 08048000-082ce000 r-xp 00000000 08:01 886456     /zabbix/sbin/zabbix_proxy
       27012:20170109:115712.604 082ce000-082dd000 rw-p 00286000 08:01 886456     /zabbix/sbin/zabbix_proxy
       27012:20170109:115712.604 082dd000-083f2000 rw-p 00000000 00:00 0          [heap]
       27012:20170109:115712.604 b4bee000-b4bef000 rw-p 00000000 00:00 0 
       27012:20170109:115712.604 b4bef000-b4bf0000 ---p 00000000 00:00 0 
       27012:20170109:115712.604 b4bf0000-b53ef000 rw-p 00000000 00:00 0 
       27012:20170109:115712.604 b53ef000-b53f0000 ---p 00000000 00:00 0 
       27012:20170109:115712.604 b53f0000-b5bef000 rw-p 00000000 00:00 0 
       27012:20170109:115712.604 b5bef000-b5bf1000 rw-s 00000000 00:04 557060     /SYSV53010002 (deleted)
       27012:20170109:115712.604 b5bf1000-b5d25000 rw-s 00000000 00:04 524291     /SYSV73010002 (deleted)
       27012:20170109:115712.604 b5d25000-b63f2000 rw-s 00000000 00:04 491522     /SYSV67010002 (deleted)
       27012:20170109:115712.604 b63f2000-b67f2000 rw-s 00000000 00:04 458753     /SYSV48010002 (deleted)
       27012:20170109:115712.604 b67f2000-b77f2000 rw-s 00000000 00:04 425984     /SYSV68010002 (deleted)
       27012:20170109:115712.604 b77f2000-b77f3000 rw-p 00000000 00:00 0 
       27012:20170109:115712.604 bfba3000-bfbb8000 rw-p 00000000 00:00 0          [stack]
       27012:20170109:115712.604 ffffe000-fffff000 r-xp 00000000 00:00 0          [vdso]
       27012:20170109:115712.604 ================================
       27012:20170109:115712.604 Please consider attaching a disassembly listing to your bug report.
       27012:20170109:115712.604 This listing can be produced with, e.g., objdump -DSswx zabbix_proxy.
       27012:20170109:115712.604 ================================
       26877:20170109:115712.605 One child process died (PID:27012,exitcode/signal:1). Exiting ...
       26877:20170109:115714.616 syncing history data...
       26877:20170109:115714.616 syncing history data done
       26877:20170109:115714.616 Zabbix Proxy stopped. Zabbix 3.0.6rc1 (revision 64124).

      Example 2:

        3904:20170124:083106.722 proxy #9 started [unreachable poller #1]
      ... ... ...
        3898:20170124:083440.625 Connection restored.
        3904:20170124:083503.377 Got signal [signal:11(SIGSEGV),reason:1,refaddr:0x5886e798]. Crashing ...
        3904:20170124:083503.377 ====== Fatal information: ======
        3904:20170124:083503.377 Program counter: 0x8090839
        3904:20170124:083503.377 === Registers: ===
        3904:20170124:083503.377 gs      =               33 =                   51 =                   51
        3904:20170124:083503.377 fs      =                0 =                    0 =                    0
        3904:20170124:083503.377 es      =               7b =                  123 =                  123
        3904:20170124:083503.377 ds      =               7b =                  123 =                  123
        3904:20170124:083503.377 edi     =         b5bfe000 =           3049250816 =          -1245716480
        3904:20170124:083503.377 esi     =                c =                   12 =                   12
        3904:20170124:083503.377 ebp     =         bfab63c8 =           3215680456 =          -1079286840
        3904:20170124:083503.377 esp     =         bfab63bc =           3215680444 =          -1079286852
        3904:20170124:083503.377 ebx     =         b5bfe778 =           3049252728 =          -1245714568
        3904:20170124:083503.377 edx     =                0 =                    0 =                    0
        3904:20170124:083503.377 ecx     =                c =                   12 =                   12
        3904:20170124:083503.377 eax     =         5886e768 =           1485236072 =           1485236072
        3904:20170124:083503.377 trapno  =                e =                   14 =                   14
        3904:20170124:083503.377 err     =                4 =                    4 =                    4
        3904:20170124:083503.377 eip     =          8090839 =            134809657 =            134809657
        3904:20170124:083503.377 cs      =               73 =                  115 =                  115
        3904:20170124:083503.377 efl     =           210246 =              2163270 =              2163270
        3904:20170124:083503.377 uesp    =         bfab63bc =           3215680444 =          -1079286852
        3904:20170124:083503.377 ss      =               7b =                  123 =                  123
        3904:20170124:083503.377 === Stack frame: ===
        3904:20170124:083503.377 +0x40(%ebp) = ebp + 64 = b5bfe000 = 3049250816 = -1245716480
        3904:20170124:083503.377 +0x3c(%ebp) = ebp + 60 = 00000035 =         53 =          53
        3904:20170124:083503.377 +0x38(%ebp) = ebp + 56 = 08242d97 =  136588695 =   136588695
        3904:20170124:083503.377 +0x34(%ebp) = ebp + 52 = 08091e8a =  134815370 =   134815370
        3904:20170124:083503.377 +0x30(%ebp) = ebp + 48 = bfab6418 = 3215680536 = -1079286760
        3904:20170124:083503.377 +0x2c(%ebp) = ebp + 44 = b5bfe0d8 = 3049251032 = -1245716264
        3904:20170124:083503.377 +0x28(%ebp) = ebp + 40 = b5bfe780 = 3049252736 = -1245714560
        3904:20170124:083503.377 +0x24(%ebp) = ebp + 36 = 082ea0f8 =  137273592 =   137273592
        3904:20170124:083503.377 +0x20(%ebp) = ebp + 32 = 082ea128 =  137273640 =   137273640
        3904:20170124:083503.377 +0x1c(%ebp) = ebp + 28 = 00000000 =          0 =           0
        3904:20170124:083503.377 +0x18(%ebp) = ebp + 24 = 00000000 =          0 =           0
        3904:20170124:083503.377 +0x14(%ebp) = ebp + 20 = 00000000 =          0 =           0
        3904:20170124:083503.377 +0x10(%ebp) = ebp + 16 = 00000000 =          0 =           0
        3904:20170124:083503.377 +0x0c(%ebp) = ebp + 12 = 00000000 =          0 =           0
        3904:20170124:083503.377 +0x08(%ebp) = ebp +  8 = 00000000 =          0 =           0 <--- call arguments
        3904:20170124:083503.377 +0x04(%ebp) = ebp +  4 = 080911fb                            <--- return address
        3904:20170124:083503.377      (%ebp) = ebp      = bfab63f8                            <--- saved ebp value
        3904:20170124:083503.378 -0x04(%ebp) = ebp -  4 = b5bfe780 = 3049252736 = -1245714560 <--- local variables
        3904:20170124:083503.378 -0x08(%ebp) = ebp -  8 = b5bfe780 = 3049252736 = -1245714560
        3904:20170124:083503.378 -0x0c(%ebp) = ebp - 12 = b5bfe000 = 3049250816 = -1245716480
        3904:20170124:083503.378 -0x10(%ebp) = ebp - 16 = 46505845 = 1179670597 =  1179670597
        3904:20170124:083503.378 -0x14(%ebp) = ebp - 20 = 00000000 =          0 =           0
        3904:20170124:083503.378 -0x18(%ebp) = ebp - 24 = 00000000 =          0 =           0
        3904:20170124:083503.378 -0x1c(%ebp) = ebp - 28 = 00000000 =          0 =           0
        3904:20170124:083503.378 -0x20(%ebp) = ebp - 32 = 00000000 =          0 =           0
        3904:20170124:083503.378 -0x24(%ebp) = ebp - 36 = 00000000 =          0 =           0
        3904:20170124:083503.378 -0x28(%ebp) = ebp - 40 = 00000000 =          0 =           0
        3904:20170124:083503.378 -0x2c(%ebp) = ebp - 44 = 00000000 =          0 =           0
        3904:20170124:083503.378 -0x30(%ebp) = ebp - 48 = 00000000 =          0 =           0
        3904:20170124:083503.378 -0x34(%ebp) = ebp - 52 = 00000000 =          0 =           0
        3904:20170124:083503.378 -0x38(%ebp) = ebp - 56 = 00000000 =          0 =           0
        3904:20170124:083503.378 -0x3c(%ebp) = ebp - 60 = 00000000 =          0 =           0
        3904:20170124:083503.378 -0x40(%ebp) = ebp - 64 = 00000000 =          0 =           0
        3904:20170124:083503.378 === Backtrace: ===
        3904:20170124:083503.410 19: [0x809af90]
        3904:20170124:083503.410 18: [0x809b800]
        3904:20170124:083503.410 17: [0xffffe410]
        3904:20170124:083503.410 16: [0x80911fb]
        3904:20170124:083503.410 15: [0x8091e8a]
        3904:20170124:083503.410 14: [0x8094ae5]
        3904:20170124:083503.410 13: [0x8094caa]
        3904:20170124:083503.410 12: [0x8092021]
        3904:20170124:083503.410 11: [0x807f412]
        3904:20170124:083503.410 10: [0x807f596]
        3904:20170124:083503.410 9: [0x807f787]
        3904:20170124:083503.410 8: [0x807f962]
        3904:20170124:083503.410 7: [0x8055362]
        3904:20170124:083503.410 6: [0x8055d8a]
        3904:20170124:083503.410 5: [0x809bd68]
        3904:20170124:083503.410 4: [0x8048eee]
        3904:20170124:083503.410 3: [0x809a01f]
        3904:20170124:083503.410 2: [0x80499a2]
        3904:20170124:083503.410 1: [0x81c04d9]
        3904:20170124:083503.410 0: [0x80482a1]
        3904:20170124:083503.410 === Memory map: ===
        3904:20170124:083503.411 08048000-082ce000 r-xp 00000000 08:01 886456     /zabbix/sbin/zabbix_proxy
        3904:20170124:083503.411 082ce000-082dd000 rw-p 00286000 08:01 886456     /zabbix/sbin/zabbix_proxy
        3904:20170124:083503.411 082dd000-083a7000 rw-p 00000000 00:00 0          [heap]
        3904:20170124:083503.411 b4bfb000-b4bfc000 rw-p 00000000 00:00 0 
        3904:20170124:083503.411 b4bfc000-b4bfd000 ---p 00000000 00:00 0 
        3904:20170124:083503.411 b4bfd000-b53fc000 rw-p 00000000 00:00 0 
        3904:20170124:083503.411 b53fc000-b53fd000 ---p 00000000 00:00 0 
        3904:20170124:083503.411 b53fd000-b5bfc000 rw-p 00000000 00:00 0 
        3904:20170124:083503.411 b5bfc000-b5bfe000 rw-s 00000000 00:04 196613     /SYSV53010002 (deleted)
        3904:20170124:083503.411 b5bfe000-b5d32000 rw-s 00000000 00:04 163844     /SYSV73010002 (deleted)
        3904:20170124:083503.411 b5d32000-b63ff000 rw-s 00000000 00:04 131075     /SYSV67010002 (deleted)
        3904:20170124:083503.411 b63ff000-b67ff000 rw-s 00000000 00:04 98306      /SYSV48010002 (deleted)
        3904:20170124:083503.411 b67ff000-b77ff000 rw-s 00000000 00:04 65537      /SYSV68010002 (deleted)
        3904:20170124:083503.411 b77ff000-b7800000 rw-p 00000000 00:00 0 
        3904:20170124:083503.411 bfa83000-bfc29000 rw-p 00000000 00:00 0          [stack]
        3904:20170124:083503.411 ffffe000-fffff000 r-xp 00000000 00:00 0          [vdso]
        3904:20170124:083503.411 ================================
        3904:20170124:083503.411 Please consider attaching a disassembly listing to your bug report.
        3904:20170124:083503.411 This listing can be produced with, e.g., objdump -DSswx zabbix_proxy.
        3904:20170124:083503.411 ================================
        3800:20170124:083503.411 One child process died (PID:3904,exitcode/signal:1). Exiting ...
        3800:20170124:083505.420 syncing history data...
        3800:20170124:083505.420 syncing history data done
        3800:20170124:083505.420 Zabbix Proxy stopped. Zabbix 3.0.6rc1 (revision 64124).

      Note, the "cannot remove existing shared memory" errors appear sometimes in the proxy log (ZBX-5662). Assuming that this is effect of frequent crashes.

        1. ZBX-11747.patch
          13 kB
          Glebs Ivanovskis
        2. ZBX-11747.patch
          15 kB
          Glebs Ivanovskis

            Unassigned Unassigned
            oleg.ivanivskyi Oleg Ivanivskyi
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

              Created:
              Updated:
              Resolved: