Uploaded image for project: 'ZABBIX BUGS AND ISSUES'
  1. ZABBIX BUGS AND ISSUES
  2. ZBX-2634

proxy trapper gets SIGSEGV in write on solaris

    Details

    • Type: Incident report
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 1.8.2
    • Fix Version/s: 1.8.3, 1.9.0 (alpha)
    • Component/s: Proxy (P)
    • Labels:
      None
    • Environment:
      Solaris 10, Spark. Unknown if this happens with x86

      Description

      I send application performance data to zabbix via sender and the proxy. About half the time I stop my script one of the proxy trapper child processes die. I have done enough debugging to convince myself it is crashing in zbx_tcp_send_ext in the write system calls (not always the same one, there are 3). This is a Sparc Solaris 10 system. Upgrading to 1.8.3 did not fix the problem.

      10318:20100629:084210.117 Timeout while answering request
      10318:20100629:084210.117 Got signal [signal:11(SIGSEGV),reason:1,refaddr:c]. Crashing ...
      10318:20100629:084210.117 ====== Fatal information: ======
      10318:20100629:084210.117 program counter not available for this architecture
      10318:20100629:084210.117 === Registers: ===
      10318:20100629:084210.117 register dump not available for this architecture
      10318:20100629:084210.117 === Backtrace: ===
      10318:20100629:084210.117 backtrace not available for this platform
      10318:20100629:084210.118 === Memory map: ===
      10318:20100629:084210.118 memory map not available for this platform
      10318:20100629:084210.118 ================================
      10291:20100629:084210.126 One child process died (PID:10318,exitcode/signal:-1). Exiting ...
      10291:20100629:084210.126 zbx_on_exit() called

      Output from runme_on_app_crash
      Program: zabbix_proxy
      Process ID: 10318
      Received signal: 11

      Application Debugging Data
      --------------------------

      > /bin/pstack 10318
      10318: /opt/thunder/sbin/zabbix_proxy -c /opt/thunder/etc/zabbix_proxy.conf
      0003ab68 child_signal_handler (d, 622a0, ffbeb8f0, 0, 0, 0) + 100
      fec44b4c __sighndlr (d, 0, ffbeb8f0, 3aa68, 0, 1) + c
      fec39b24 call_user_handler (d, 0, 8, 0, feed2a00, ffbeb8f0) + 3b8
      fec45f64 _write (6, ffbebc68, 8, 0, 0, 0) + c
      000448e4 zbx_tcp_send_ext (ffbfef40, ffbebce4, 0, 0, ffbebc70, a39b0) + 9c
      00045668 zbx_send_response (ffbfef40, 0, ffffefe8, 3, fa, 73800) + cc
      00022750 process_trapper_child (ffbfef40, 5c8b8, 0, e, 10b4, fec73a80) + 534
      00022cf4 child_trapper_main (5c800, ffbfef40, 9f000, 9e800, ffbfeed0, a39b0) + b4
      00018d2c MAIN_ZABBIX_ENTRY (1a, fec39cac, 85c00, 85c00, 85c00, 85c00) + 570
      0003af04 daemon_start (ffbffbd0, ffbffc6c, ffbffc7c, a3cfc, feed0140, feed0180) + 2fc
      00017edc _start (0, 0, 0, 0, 0, 0) + 5c

      > /bin/pmap -x 10318
      10318: /opt/thunder/sbin/zabbix_proxy -c /opt/thunder/etc/zabbix_proxy.conf
      Address Kbytes RSS Anon Locked Mode Mapped File
      00010000 408 352 - - r-x-- zabbix_proxy
      00084000 104 24 8 - rwx-- zabbix_proxy
      0009E000 16 8 8 - rwx-- zabbix_proxy
      000A2000 744 680 648 - rwx-- [ heap ]
      FA000000 13928 96 - - rwxs- [ shmid=0x1000001 ]
      FB000000 16392 16 - - rwxs- [ shmid=0x1000000 ]
      FC400000 16392 16 - - rwxs- [ shmid=0x7f ]
      FD580000 2464 16 - - rwxs- [ shmid=0x1000002 ]
      FD800000 16416 24 - - rwxs- [ shmid=0x7e ]
      FEA40000 32 32 - - r-x-- nss_files.so.1
      FEA58000 8 8 - - rwx-- nss_files.so.1
      FEA5C000 8 8 - - rwxs- [ anon ]
      FEA70000 64 64 8 - rwx-- [ anon ]
      FEA90000 64 32 - - rwx-- [ anon ]
      FEAB0000 8 8 - - rwx-- [ anon ]
      FEAC0000 128 88 - - r-x-- libelf.so.1
      FEAE0000 8 8 - - rwx-- libelf.so.1
      FEAF0000 80 16 - - r-x-- libmd.so.1
      FEB14000 8 8 - - rwx-- libmd.so.1
      FEB20000 32 16 - - r-x-- libaio.so.1
      FEB38000 8 8 8 - rwx-- libaio.so.1
      FEB40000 40 24 - - r-x-- libintl.so.3.4.3
      FEB58000 8 8 - - rwx-- libintl.so.3.4.3
      FEB60000 40 32 - - r-x-- libgcc_s.so.1
      FEB78000 8 8 - - rwx-- libgcc_s.so.1
      FEB80000 888 832 - - r-x-- libc.so.1
      FEC6E000 32 32 24 - rwx-- libc.so.1
      FEC76000 8 8 8 - rwx-- libc.so.1
      FEC80000 920 32 - - r-x-- libiconv.so.2.4.0
      FED74000 16 16 - - rwx-- libiconv.so.2.4.0
      FED80000 248 208 - - r-x-- libresolv.so.2
      FEDCE000 16 16 - - rwx-- libresolv.so.2
      FEDE0000 8 8 - - r-x-- libkstat.so.1
      FEDF2000 8 8 - - rwx-- libkstat.so.1
      FEE00000 680 144 - - r-x-- libm.so.2
      FEEB8000 32 24 - - rwx-- libm.so.2
      FEED0000 24 16 8 - rwx-- [ anon ]
      FEEE0000 16 8 - - r-x-- libkvm.so.1
      FEEF4000 8 8 - - rwx-- libkvm.so.1
      FEF00000 584 304 - - r-x-- libnsl.so.1
      FEFA2000 40 40 8 - rwx-- libnsl.so.1
      FEFAC000 24 16 - - rwx-- libnsl.so.1
      FEFC0000 8 8 - - rwx-- [ anon ]
      FEFD0000 8 8 - - rwx-- [ anon ]
      FEFE0000 72 24 - - r-x-- libz.so.1.0.2
      FF000000 8 8 - - rwx-- libz.so.1.0.2
      FF010000 8 8 - - r-x-- libdl.so.1
      FF022000 8 8 - - rwx-- libdl.so.1
      FF030000 48 40 - - r-x-- libsocket.so.1
      FF04C000 8 8 8 - rwx-- libsocket.so.1
      FF060000 24 24 - - r-x-- librt.so.1
      FF076000 8 8 - - rwx-- librt.so.1
      FF080000 1168 744 - - r-x-- libcrypto.so.0.9.8
      FF1B2000 96 96 - - rwx-- libcrypto.so.0.9.8
      FF1CA000 8 - - - rwx-- libcrypto.so.0.9.8
      FF1E0000 256 256 - - r-x-- libssl.so.0.9.8
      FF22E000 24 24 - - rwx-- libssl.so.0.9.8
      FF240000 192 32 - - r-x-- libidn.so.11.6.1
      FF27E000 16 16 - - rwx-- libidn.so.11.6.1
      FF290000 8 8 - - rwx-- [ anon ]
      FF2A0000 296 64 - - r-x-- libcurl.so.4.2.0
      FF2F8000 16 16 - - rwx-- libcurl.so.4.2.0
      FF300000 488 456 - - r-x-- libsqlite3.so.0.8.6
      FF388000 16 16 16 - rwx-- libsqlite3.so.0.8.6
      FF3A0000 16 16 - - r-x-- libc_psr.so.1
      FF3B0000 208 208 - - r-x-- ld.so.1
      FF3F4000 8 8 8 - rwx-- ld.so.1
      FF3F6000 8 8 8 - rwx-- ld.so.1
      FFBC8000 224 224 176 - rwx-- [ stack ]
      -------- ------- ------- ------- -------
      total Kb 74208 5656 944 -

      > /bin/pfiles 10318
      10318: /opt/thunder/sbin/zabbix_proxy -c /opt/thunder/etc/zabbix_proxy.conf
      Current rlimit: 256 file descriptors
      0: S_IFCHR mode:0666 dev:311,0 ino:6815752 uid:0 gid:3 rdev:13,2
      O_RDONLY
      /devices/pseudo/mm@0:null
      1: S_IFREG mode:0664 dev:136,8 ino:552348 uid:125 gid:1 size:5330396
      O_WRONLY|O_APPEND|O_CREAT
      /opt/thunder/var/log/zabbix_proxy.log
      2: S_IFREG mode:0664 dev:136,8 ino:552348 uid:125 gid:1 size:5330396
      O_WRONLY|O_APPEND|O_CREAT
      /opt/thunder/var/log/zabbix_proxy.log
      3: S_IFREG mode:0664 dev:136,8 ino:734536 uid:125 gid:1 size:5
      O_WRONLY|O_CREAT|O_TRUNC
      advisory write lock set by process 10291
      /opt/thunder/var/run/zabbix_proxy.pid
      4: S_IFSOCK mode:0666 dev:317,0 ino:57069 uid:0 gid:0 size:0
      O_RDWR
      SOCK_STREAM
      SO_REUSEADDR,SO_SNDBUF(49152),SO_RCVBUF(49152),IP_ NEXTHOP(0.0.192.0)
      sockname: AF_INET 0.0.0.0 port: 10052
      5: S_IFREG mode:0644 dev:136,8 ino:552349 uid:125 gid:1 size:336896
      O_RDWR|O_LARGEFILE FD_CLOEXEC
      /opt/thunder/var/data/zabbix_proxy.db
      6: S_IFSOCK mode:0666 dev:317,0 ino:21831 uid:0 gid:0 size:0
      O_RDWR
      SOCK_STREAM
      SO_REUSEADDR,SO_SNDBUF(49152),SO_RCVBUF(49152),IP_ NEXTHOP(0.0.192.0)
      sockname: AF_INET 0.0.0.0 port: 0

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                frankg frankg gleason
              • Votes:
                0 Vote for this issue
                Watchers:
                0 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: