ZABBIX BUGS AND ISSUES
  1. ZABBIX BUGS AND ISSUES
  2. ZBX-7653

Got signal [signal:11(SIGSEGV),reason:128,refaddr:(nil)]. Crashing ... - in "escalator"

    Details

    • Type: Incident report Incident report
    • Status: Closed
    • Priority: Blocker Blocker
    • Resolution: Fixed
    • Affects Version/s: 2.2.1
    • Fix Version/s: 2.2.2rc1, 2.3.1
    • Component/s: Server (S)
    • Labels:
    • Environment:
      Linux zabbix01 3.2.0-4-amd64 #1 SMP Debian 3.2.51-1 x86_64 GNU/Linux (Debian Wheezy)
      zabbix-server-mysql 2.2.1-1+wheezy (amd64)

      Description

      Before upgrade to zabbix 2.21, the server was very stable and reliable.
      After we performed upgrade from 2.0.9 -> 2.2.1:
      Several times a day, zabbix-server crashes with following notification in its logfile:

       18123:20140114:200848.879 Got signal [signal:11(SIGSEGV),reason:128,refaddr:(nil)]. Crashing ...
       18123:20140114:200848.879 ====== Fatal information: ======
       18123:20140114:200848.879 Program counter: 0x7fa5e257ae21
       18123:20140114:200848.879 === Registers: ===
       18123:20140114:200848.879 r8      =          153fc20 =             22281248 =             22281248
       18123:20140114:200848.879 r9      = 6972756365532d73 =  7598264591089610099 =  7598264591089610099
       18123:20140114:200848.879 r10     = 74696475412d7974 =  8388346235721709940 =  8388346235721709940
       18123:20140114:200848.879 r11     =              768 =                 1896 =                 1896
       18123:20140114:200848.880 r12     =           4bf2b3 =              4977331 =              4977331
       18123:20140114:200848.880 r13     =              2ac =                  684 =                  684
       18123:20140114:200848.880 r14     =     7fa54f972128 =      140347981635880 =      140347981635880
       18123:20140114:200848.880 r15     =     7fa54e29c59c =      140347957691804 =      140347957691804
       18123:20140114:200848.880 rdi     = 3320646e61202965 =  3684054920427743589 =  3684054920427743589
       18123:20140114:200848.880 rsi     =              2ac =                  684 =                  684
       18123:20140114:200848.880 rbp     = 3320646e61202965 =  3684054920427743589 =  3684054920427743589
       18123:20140114:200848.880 rbx     = 3320646e61202965 =  3684054920427743589 =  3684054920427743589
       18123:20140114:200848.880 rdx     =                0 =                    0 =                    0
       18123:20140114:200848.880 rax     =                0 =                    0 =                    0
       18123:20140114:200848.880 rcx     =               25 =                   37 =                   37
       18123:20140114:200848.880 rsp     =     7fffa6d06708 =      140735992063752 =      140735992063752
       18123:20140114:200848.880 rip     =     7fa5e257ae21 =      140350443728417 =      140350443728417
       18123:20140114:200848.880 efl     =            10287 =                66183 =                66183
       18123:20140114:200848.880 csgsfs  =               33 =                   51 =                   51
       18123:20140114:200848.880 err     =                0 =                    0 =                    0
       18123:20140114:200848.880 trapno  =                d =                   13 =                   13
       18123:20140114:200848.880 oldmask =                0 =                    0 =                    0
       18123:20140114:200848.880 cr2     =                0 =                    0 =                    0
       18123:20140114:200848.880 === Backtrace: ===
       18123:20140114:200848.881 14: /usr/sbin/zabbix_server: history syncer #1 [synced 0 items in 0.000017 sec, syncing history](print_fatal_info+0x9e) [0x461cae]
       18123:20140114:200848.881 13: /usr/sbin/zabbix_server: history syncer #1 [synced 0 items in 0.000017 sec, syncing history]() [0x4620b7]
       18123:20140114:200848.881 12: /lib/x86_64-linux-gnu/libc.so.6(+0x324f0) [0x7fa5e252c4f0]
       18123:20140114:200848.881 11: /lib/x86_64-linux-gnu/libc.so.6(+0x80e21) [0x7fa5e257ae21]
       18123:20140114:200848.881 10: /lib/x86_64-linux-gnu/libc.so.6(__strdup+0x16) [0x7fa5e257ab36]
       18123:20140114:200848.881 9: /usr/sbin/zabbix_server: history syncer #1 [synced 0 items in 0.000017 sec, syncing history](zbx_strdup2+0x3f) [0x46622f]
       18123:20140114:200848.882 8: /usr/sbin/zabbix_server: history syncer #1 [synced 0 items in 0.000017 sec, syncing history]() [0x45879f]
       18123:20140114:200848.882 7: /usr/sbin/zabbix_server: history syncer #1 [synced 0 items in 0.000017 sec, syncing history]() [0x458ce1]
       18123:20140114:200848.882 6: /usr/sbin/zabbix_server: history syncer #1 [synced 0 items in 0.000017 sec, syncing history](zbx_vc_add_value+0x5d1) [0x45ba61]
       18123:20140114:200848.882 5: /usr/sbin/zabbix_server: history syncer #1 [synced 0 items in 0.000017 sec, syncing history](DCsync_history+0x1cff) [0x44d15f]
       18123:20140114:200848.882 4: /usr/sbin/zabbix_server: history syncer #1 [synced 0 items in 0.000017 sec, syncing history](main_dbsyncer_loop+0x205) [0x41b3f5]
       18123:20140114:200848.882 3: /usr/sbin/zabbix_server: history syncer #1 [synced 0 items in 0.000017 sec, syncing history](MAIN_ZABBIX_ENTRY+0x7e8) [0x41a828]
       18123:20140114:200848.882 2: /usr/sbin/zabbix_server: history syncer #1 [synced 0 items in 0.000017 sec, syncing history](daemon_start+0x18d) [0x4615cd]
       18123:20140114:200848.882 1: /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xfd) [0x7fa5e2518ead]
       18123:20140114:200848.882 0: /usr/sbin/zabbix_server: history syncer #1 [synced 0 items in 0.000017 sec, syncing history]() [0x415841]
       18123:20140114:200848.882 === Memory map: ===
       18123:20140114:200848.882 00400000-0054b000 r-xp 00000000 fe:05 3601                               /usr/sbin/zabbix_server
       18123:20140114:200848.882 0074a000-0074e000 rw-p 0014a000 fe:05 3601                               /usr/sbin/zabbix_server
       18123:20140114:200848.882 0074e000-00755000 rw-p 00000000 00:00 0 
       18123:20140114:200848.882 0151c000-0153d000 rw-p 00000000 00:00 0                                  [heap]
       18123:20140114:200848.882 0153d000-01574000 rw-p 00000000 00:00 0                                  [heap]
       18123:20140114:200848.882 01574000-015df000 rw-p 00000000 00:00 0                                  [heap]
       18123:20140114:200848.882 7fa54d9d1000-7fa54d9d2000 rw-p 00000000 00:00 0 
       18123:20140114:200848.882 7fa54d9d2000-7fa54da07000 r--s 00000000 fe:03 2285                       /var/cache/nscd/services
       18123:20140114:200848.882 7fa54da07000-7fa54da08000 ---p 00000000 00:00 0 
       18123:20140114:200848.882 7fa54da08000-7fa54e208000 rw-p 00000000 00:00 0 
       18123:20140114:200848.882 7fa54e208000-7fa5ce208000 rw-s 00000000 00:04 786440                     /SYSV760005ee (deleted)
       18123:20140114:200848.883 7fa5ce208000-7fa5cea08000 rw-s 00000000 00:04 753671                     /SYSV770005ee (deleted)
       18123:20140114:200848.883 7fa5cea08000-7fa5cea10000 rw-s 00000000 00:04 720902                     /SYSV530005ee (deleted)
       18123:20140114:200848.883 7fa5cea10000-7fa5cf3aa000 rw-s 00000000 00:04 688133                     /SYSV730005ee (deleted)
       18123:20140114:200848.883 7fa5cf3aa000-7fa5d2a11000 rw-s 00000000 00:04 655364                     /SYSV670005ee (deleted)
       18123:20140114:200848.883 7fa5d2a11000-7fa5d3211000 rw-s 00000000 00:04 622595                     /SYSV740005ee (deleted)
       18123:20140114:200848.883 7fa5d3211000-7fa5db211000 rw-s 00000000 00:04 589826                     /SYSV780005ee (deleted)
       18123:20140114:200848.883 7fa5db211000-7fa5df211000 rw-s 00000000 00:04 557057                     /SYSV680005ee (deleted)
       18123:20140114:200848.883 7fa5df211000-7fa5df236000 r-xp 00000000 fe:00 123                        /lib/x86_64-linux-gnu/libtinfo.so.5.9
       18123:20140114:200848.883 7fa5df236000-7fa5df435000 ---p 00025000 fe:00 123                        /lib/x86_64-linux-gnu/libtinfo.so.5.9
       18123:20140114:200848.883 7fa5df435000-7fa5df439000 r--p 00024000 fe:00 123                        /lib/x86_64-linux-gnu/libtinfo.so.5.9
       18123:20140114:200848.883 7fa5df439000-7fa5df43a000 rw-p 00028000 fe:00 123                        /lib/x86_64-linux-gnu/libtinfo.so.5.9
       18123:20140114:200848.883 7fa5df43a000-7fa5df43d000 r-xp 00000000 fe:00 574                        /lib/x86_64-linux-gnu/libgpg-error.so.0.8.0
       18123:20140114:200848.883 7fa5df43d000-7fa5df63c000 ---p 00003000 fe:00 574                        /lib/x86_64-linux-gnu/libgpg-error.so.0.8.0
       18123:20140114:200848.883 7fa5df63c000-7fa5df63d000 rw-p 00002000 fe:00 574                        /lib/x86_64-linux-gnu/libgpg-error.so.0.8.0
       18123:20140114:200848.883 7fa5df63d000-7fa5df640000 r-xp 00000000 fe:00 562                        /lib/x86_64-linux-gnu/libkeyutils.so.1.4
       18123:20140114:200848.883 7fa5df640000-7fa5df83f000 ---p 00003000 fe:00 562                        /lib/x86_64-linux-gnu/libkeyutils.so.1.4
       18123:20140114:200848.883 7fa5df83f000-7fa5df840000 r--p 00002000 fe:00 562                        /lib/x86_64-linux-gnu/libkeyutils.so.1.4
       18123:20140114:200848.883 7fa5df840000-7fa5df841000 rw-p 00003000 fe:00 562                        /lib/x86_64-linux-gnu/libkeyutils.so.1.4
       18123:20140114:200848.883 7fa5df841000-7fa5df849000 r-xp 00000000 fe:05 10741                      /usr/lib/x86_64-linux-gnu/libkrb5support.so.0.1
       18123:20140114:200848.883 7fa5df849000-7fa5dfa48000 ---p 00008000 fe:05 10741                      /usr/lib/x86_64-linux-gnu/libkrb5support.so.0.1
       18123:20140114:200848.883 7fa5dfa48000-7fa5dfa49000 r--p 00007000 fe:05 10741                      /usr/lib/x86_64-linux-gnu/libkrb5support.so.0.1
       18123:20140114:200848.884 7fa5dfa49000-7fa5dfa4a000 rw-p 00008000 fe:05 10741                      /usr/lib/x86_64-linux-gnu/libkrb5support.so.0.1
       18123:20140114:200848.884 7fa5dfa4a000-7fa5dfa4d000 r-xp 00000000 fe:00 207                        /lib/x86_64-linux-gnu/libcom_err.so.2.1
       18123:20140114:200848.884 7fa5dfa4d000-7fa5dfc4c000 ---p 00003000 fe:00 207                        /lib/x86_64-linux-gnu/libcom_err.so.2.1
       18123:20140114:200848.884 7fa5dfc4c000-7fa5dfc4d000 r--p 00002000 fe:00 207                        /lib/x86_64-linux-gnu/libcom_err.so.2.1
       18123:20140114:200848.884 7fa5dfc4d000-7fa5dfc4e000 rw-p 00003000 fe:00 207                        /lib/x86_64-linux-gnu/libcom_err.so.2.1
       18123:20140114:200848.884 7fa5dfc4e000-7fa5dfc74000 r-xp 00000000 fe:05 10747                      /usr/lib/x86_64-linux-gnu/libk5crypto.so.3.1
       18123:20140114:200848.884 7fa5dfc74000-7fa5dfe74000 ---p 00026000 fe:05 10747                      /usr/lib/x86_64-linux-gnu/libk5crypto.so.3.1
       18123:20140114:200848.884 7fa5dfe74000-7fa5dfe75000 r--p 00026000 fe:05 10747                      /usr/lib/x86_64-linux-gnu/libk5crypto.so.3.1
       18123:20140114:200848.884 7fa5dfe75000-7fa5dfe76000 rw-p 00027000 fe:05 10747                      /usr/lib/x86_64-linux-gnu/libk5crypto.so.3.1
       18123:20140114:200848.884 7fa5dfe76000-7fa5dfe77000 rw-p 00000000 00:00 0 
       18123:20140114:200848.884 7fa5dfe77000-7fa5dff40000 r-xp 00000000 fe:05 10753                      /usr/lib/x86_64-linux-gnu/libkrb5.so.3.3
       18123:20140114:200848.884 7fa5dff40000-7fa5e013f000 ---p 000c9000 fe:05 10753                      /usr/lib/x86_64-linux-gnu/libkrb5.so.3.3
       18123:20140114:200848.884 7fa5e013f000-7fa5e0149000 r--p 000c8000 fe:05 10753                      /usr/lib/x86_64-linux-gnu/libkrb5.so.3.3
       18123:20140114:200848.884 7fa5e0149000-7fa5e014b000 rw-p 000d2000 fe:05 10753                      /usr/lib/x86_64-linux-gnu/libkrb5.so.3.3
       18123:20140114:200848.884 7fa5e014b000-7fa5e015c000 r-xp 00000000 fe:05 4646                       /usr/lib/x86_64-linux-gnu/libp11-kit.so.0.0.0
       18123:20140114:200848.884 7fa5e015c000-7fa5e035b000 ---p 00011000 fe:05 4646                       /usr/lib/x86_64-linux-gnu/libp11-kit.so.0.0.0
       18123:20140114:200848.884 7fa5e035b000-7fa5e035c000 r--p 00010000 fe:05 4646                       /usr/lib/x86_64-linux-gnu/libp11-kit.so.0.0.0
       18123:20140114:200848.884 7fa5e035c000-7fa5e035d000 rw-p 00011000 fe:05 4646                       /usr/lib/x86_64-linux-gnu/libp11-kit.so.0.0.0
       18123:20140114:200848.884 7fa5e035d000-7fa5e036d000 r-xp 00000000 fe:05 4656                       /usr/lib/x86_64-linux-gnu/libtasn1.so.3.1.16
       18123:20140114:200848.884 7fa5e036d000-7fa5e056c000 ---p 00010000 fe:05 4656                       /usr/lib/x86_64-linux-gnu/libtasn1.so.3.1.16
       18123:20140114:200848.885 7fa5e056c000-7fa5e056d000 r--p 0000f000 fe:05 4656                       /usr/lib/x86_64-linux-gnu/libtasn1.so.3.1.16
       18123:20140114:200848.885 7fa5e056d000-7fa5e056e000 rw-p 00010000 fe:05 4656                       /usr/lib/x86_64-linux-gnu/libtasn1.so.3.1.16
       18123:20140114:200848.885 7fa5e056e000-7fa5e0588000 r-xp 00000000 fe:05 35080                      /usr/lib/x86_64-linux-gnu/libsasl2.so.2.0.25
       18123:20140114:200848.885 7fa5e0588000-7fa5e0787000 ---p 0001a000 fe:05 35080                      /usr/lib/x86_64-linux-gnu/libsasl2.so.2.0.25
       18123:20140114:200848.885 7fa5e0787000-7fa5e0788000 r--p 00019000 fe:05 35080                      /usr/lib/x86_64-linux-gnu/libsasl2.so.2.0.25
       18123:20140114:200848.885 7fa5e0788000-7fa5e0789000 rw-p 0001a000 fe:05 35080                      /usr/lib/x86_64-linux-gnu/libsasl2.so.2.0.25
       18123:20140114:200848.885 7fa5e0789000-7fa5e07aa000 r-xp 00000000 fe:00 131                        /lib/x86_64-linux-gnu/libncurses.so.5.9
       18123:20140114:200848.885 7fa5e07aa000-7fa5e09a9000 ---p 00021000 fe:00 131                        /lib/x86_64-linux-gnu/libncurses.so.5.9
       18123:20140114:200848.885 7fa5e09a9000-7fa5e09aa000 r--p 00020000 fe:00 131                        /lib/x86_64-linux-gnu/libncurses.so.5.9
       18123:20140114:200848.885 7fa5e09aa000-7fa5e09ab000 rw-p 00021000 fe:00 131                        /lib/x86_64-linux-gnu/libncurses.so.5.9
       18123:20140114:200848.885 7fa5e09ab000-7fa5e09b3000 r-xp 00000000 fe:05 35163                      /usr/lib/libOpenIPMIutils.so.0.0.1
       18123:20140114:200848.885 7fa5e09b3000-7fa5e0bb2000 ---p 00008000 fe:05 35163                      /usr/lib/libOpenIPMIutils.so.0.0.1
       18123:20140114:200848.885 7fa5e0bb2000-7fa5e0bb3000 rw-p 00007000 fe:05 35163                      /usr/lib/libOpenIPMIutils.so.0.0.1
       18123:20140114:200848.885 7fa5e0bb3000-7fa5e0bbc000 r-xp 00000000 fe:05 28181                      /usr/lib/x86_64-linux-gnu/libltdl.so.7.3.0
       18123:20140114:200848.885 7fa5e0bbc000-7fa5e0dbb000 ---p 00009000 fe:05 28181                      /usr/lib/x86_64-linux-gnu/libltdl.so.7.3.0
       18123:20140114:200848.885 7fa5e0dbb000-7fa5e0dbc000 rw-p 00008000 fe:05 28181                      /usr/lib/x86_64-linux-gnu/libltdl.so.7.3.0
       18123:20140114:200848.885 7fa5e0dbc000-7fa5e0dd5000 r-xp 00000000 fe:05 35098                      /usr/lib/x86_64-linux-gnu/librtmp.so.0
       18123:20140114:200848.885 7fa5e0dd5000-7fa5e0fd5000 ---p 00019000 fe:05 35098                      /usr/lib/x86_64-linux-gnu/librtmp.so.0
       18123:20140114:200848.885 7fa5e0fd5000-7fa5e0fd6000 rw-p 00019000 fe:05 35098                      /usr/lib/x86_64-linux-gnu/librtmp.so.0
       18123:20140114:200848.885 7fa5e0fd6000-7fa5e1050000 r-xp 00000000 fe:00 4850                       /lib/x86_64-linux-gnu/libgcrypt.so.11.7.0
       18123:20140114:200848.885 7fa5e1050000-7fa5e1250000 ---p 0007a000 fe:00 4850                       /lib/x86_64-linux-gnu/libgcrypt.so.11.7.0
       18123:20140114:200848.886 7fa5e1250000-7fa5e1251000 r--p 0007a000 fe:00 4850                       /lib/x86_64-linux-gnu/libgcrypt.so.11.7.0
       18123:20140114:200848.886 7fa5e1251000-7fa5e1254000 rw-p 0007b000 fe:00 4850                       /lib/x86_64-linux-gnu/libgcrypt.so.11.7.0
       18123:20140114:200848.886 7fa5e1254000-7fa5e1290000 r-xp 00000000 fe:05 10765                      /usr/lib/x86_64-linux-gnu/libgssapi_krb5.so.2.2
       18123:20140114:200848.886 7fa5e1290000-7fa5e1490000 ---p 0003c000 fe:05 10765                      /usr/lib/x86_64-linux-gnu/libgssapi_krb5.so.2.2
       18123:20140114:200848.886 7fa5e1490000-7fa5e1491000 r--p 0003c000 fe:05 10765                      /usr/lib/x86_64-linux-gnu/libgssapi_krb5.so.2.2
       18123:20140114:200848.886 7fa5e1491000-7fa5e1493000 rw-p 0003d000 fe:05 10765                      /usr/lib/x86_64-linux-gnu/libgssapi_krb5.so.2.2
       18123:20140114:200848.886 7fa5e1493000-7fa5e14c5000 r-xp 00000000 fe:05 4683                       /usr/lib/x86_64-linux-gnu/libidn.so.11.6.8
       18123:20140114:200848.886 7fa5e14c5000-7fa5e16c5000 ---p 00032000 fe:05 4683                       /usr/lib/x86_64-linux-gnu/libidn.so.11.6.8
       18123:20140114:200848.886 7fa5e16c5000-7fa5e16c6000 r--p 00032000 fe:05 4683                       /usr/lib/x86_64-linux-gnu/libidn.so.11.6.8
       18123:20140114:200848.886 7fa5e16c6000-7fa5e16c7000 rw-p 00033000 fe:05 4683                       /usr/lib/x86_64-linux-gnu/libidn.so.11.6.8
       18123:20140114:200848.886 7fa5e16c7000-7fa5e16e9000 r-xp 00000000 fe:00 105                        /lib/x86_64-linux-gnu/liblzma.so.5.0.0
       18123:20140114:200848.886 7fa5e16e9000-7fa5e18e8000 ---p 00022000 fe:00 105                        /lib/x86_64-linux-gnu/liblzma.so.5.0.0
       18123:20140114:200848.886 7fa5e18e8000-7fa5e18e9000 r--p 00021000 fe:00 105                        /lib/x86_64-linux-gnu/liblzma.so.5.0.0
       18123:20140114:200848.886 7fa5e18e9000-7fa5e18ea000 rw-p 00022000 fe:00 105                        /lib/x86_64-linux-gnu/liblzma.so.5.0.0
       18123:20140114:200848.886 7fa5e18ea000-7fa5e19a3000 r-xp 00000000 fe:05 4679                       /usr/lib/x86_64-linux-gnu/libgnutls.so.26.22.4
       18123:20140114:200848.886 7fa5e19a3000-7fa5e1ba2000 ---p 000b9000 fe:05 4679                       /usr/lib/x86_64-linux-gnu/libgnutls.so.26.22.4
       18123:20140114:200848.886 7fa5e1ba2000-7fa5e1ba8000 r--p 000b8000 fe:05 4679                       /usr/lib/x86_64-linux-gnu/libgnutls.so.26.22.4
       18123:20140114:200848.886 7fa5e1ba8000-7fa5e1baa000 rw-p 000be000 fe:05 4679                       /usr/lib/x86_64-linux-gnu/libgnutls.so.26.22.4
       18123:20140114:200848.886 7fa5e1baa000-7fa5e1bbf000 r-xp 00000000 fe:00 30                         /lib/x86_64-linux-gnu/libgcc_s.so.1
       18123:20140114:200848.886 7fa5e1bbf000-7fa5e1dbf000 ---p 00015000 fe:00 30                         /lib/x86_64-linux-gnu/libgcc_s.so.1
       18123:20140114:200848.886 7fa5e1dbf000-7fa5e1dc0000 rw-p 00015000 fe:00 30                         /lib/x86_64-linux-gnu/libgcc_s.so.1
       18123:20140114:200848.887 7fa5e1dc0000-7fa5e1ea8000 r-xp 00000000 fe:05 4034                       /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.17
       18123:20140114:200848.887 7fa5e1ea8000-7fa5e20a8000 ---p 000e8000 fe:05 4034                       /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.17
       18123:20140114:200848.887 7fa5e20a8000-7fa5e20b0000 r--p 000e8000 fe:05 4034                       /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.17
       18123:20140114:200848.887 7fa5e20b0000-7fa5e20b2000 rw-p 000f0000 fe:05 4034                       /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.17
       18123:20140114:200848.887 7fa5e20b2000-7fa5e20c7000 rw-p 00000000 00:00 0 
       18123:20140114:200848.887 7fa5e20c7000-7fa5e20dd000 r-xp 00000000 fe:00 332                        /lib/x86_64-linux-gnu/libz.so.1.2.7
       18123:20140114:200848.887 7fa5e20dd000-7fa5e22dc000 ---p 00016000 fe:00 332                        /lib/x86_64-linux-gnu/libz.so.1.2.7
       18123:20140114:200848.887 7fa5e22dc000-7fa5e22dd000 r--p 00015000 fe:00 332                        /lib/x86_64-linux-gnu/libz.so.1.2.7
       18123:20140114:200848.887 7fa5e22dd000-7fa5e22de000 rw-p 00016000 fe:00 332                        /lib/x86_64-linux-gnu/libz.so.1.2.7
       18123:20140114:200848.887 7fa5e22de000-7fa5e22f5000 r-xp 00000000 fe:00 57                         /lib/x86_64-linux-gnu/libpthread-2.13.so
       18123:20140114:200848.887 7fa5e22f5000-7fa5e24f4000 ---p 00017000 fe:00 57                         /lib/x86_64-linux-gnu/libpthread-2.13.so
       18123:20140114:200848.887 7fa5e24f4000-7fa5e24f5000 r--p 00016000 fe:00 57                         /lib/x86_64-linux-gnu/libpthread-2.13.so
       18123:20140114:200848.887 7fa5e24f5000-7fa5e24f6000 rw-p 00017000 fe:00 57                         /lib/x86_64-linux-gnu/libpthread-2.13.so
       18123:20140114:200848.887 7fa5e24f6000-7fa5e24fa000 rw-p 00000000 00:00 0 
       18123:20140114:200848.887 7fa5e24fa000-7fa5e267a000 r-xp 00000000 fe:00 68                         /lib/x86_64-linux-gnu/libc-2.13.so
       18123:20140114:200848.887 7fa5e267a000-7fa5e287a000 ---p 00180000 fe:00 68                         /lib/x86_64-linux-gnu/libc-2.13.so
       18123:20140114:200848.887 7fa5e287a000-7fa5e287e000 r--p 00180000 fe:00 68                         /lib/x86_64-linux-gnu/libc-2.13.so
       18123:20140114:200848.887 7fa5e287e000-7fa5e287f000 rw-p 00184000 fe:00 68                         /lib/x86_64-linux-gnu/libc-2.13.so
       18123:20140114:200848.887 7fa5e287f000-7fa5e2884000 rw-p 00000000 00:00 0 
       18123:20140114:200848.887 7fa5e2884000-7fa5e2897000 r-xp 00000000 fe:00 70                         /lib/x86_64-linux-gnu/libresolv-2.13.so
       18123:20140114:200848.888 7fa5e2897000-7fa5e2a96000 ---p 00013000 fe:00 70                         /lib/x86_64-linux-gnu/libresolv-2.13.so
       18123:20140114:200848.888 7fa5e2a96000-7fa5e2a97000 r--p 00012000 fe:00 70                         /lib/x86_64-linux-gnu/libresolv-2.13.so
       18123:20140114:200848.888 7fa5e2a97000-7fa5e2a98000 rw-p 00013000 fe:00 70                         /lib/x86_64-linux-gnu/libresolv-2.13.so
       18123:20140114:200848.888 7fa5e2a98000-7fa5e2a9a000 rw-p 00000000 00:00 0 
       18123:20140114:200848.888 7fa5e2a9a000-7fa5e2aa1000 r-xp 00000000 fe:00 62                         /lib/x86_64-linux-gnu/librt-2.13.so
       18123:20140114:200848.888 7fa5e2aa1000-7fa5e2ca0000 ---p 00007000 fe:00 62                         /lib/x86_64-linux-gnu/librt-2.13.so
       18123:20140114:200848.888 7fa5e2ca0000-7fa5e2ca1000 r--p 00006000 fe:00 62                         /lib/x86_64-linux-gnu/librt-2.13.so
       18123:20140114:200848.888 7fa5e2ca1000-7fa5e2ca2000 rw-p 00007000 fe:00 62                         /lib/x86_64-linux-gnu/librt-2.13.so
       18123:20140114:200848.888 7fa5e2ca2000-7fa5e2ca4000 r-xp 00000000 fe:00 60                         /lib/x86_64-linux-gnu/libdl-2.13.so
       18123:20140114:200848.888 7fa5e2ca4000-7fa5e2ea4000 ---p 00002000 fe:00 60                         /lib/x86_64-linux-gnu/libdl-2.13.so
       18123:20140114:200848.888 7fa5e2ea4000-7fa5e2ea5000 r--p 00002000 fe:00 60                         /lib/x86_64-linux-gnu/libdl-2.13.so
       18123:20140114:200848.888 7fa5e2ea5000-7fa5e2ea6000 rw-p 00003000 fe:00 60                         /lib/x86_64-linux-gnu/libdl-2.13.so
       18123:20140114:200848.888 7fa5e2ea6000-7fa5e2f27000 r-xp 00000000 fe:00 65                         /lib/x86_64-linux-gnu/libm-2.13.so
       18123:20140114:200848.888 7fa5e2f27000-7fa5e3126000 ---p 00081000 fe:00 65                         /lib/x86_64-linux-gnu/libm-2.13.so
       18123:20140114:200848.888 7fa5e3126000-7fa5e3127000 r--p 00080000 fe:00 65                         /lib/x86_64-linux-gnu/libm-2.13.so
       18123:20140114:200848.888 7fa5e3127000-7fa5e3128000 rw-p 00081000 fe:00 65                         /lib/x86_64-linux-gnu/libm-2.13.so
       18123:20140114:200848.888 7fa5e3128000-7fa5e3136000 r-xp 00000000 fe:05 35093                      /usr/lib/x86_64-linux-gnu/liblber-2.4.so.2.8.3
       18123:20140114:200848.888 7fa5e3136000-7fa5e3335000 ---p 0000e000 fe:05 35093                      /usr/lib/x86_64-linux-gnu/liblber-2.4.so.2.8.3
       18123:20140114:200848.888 7fa5e3335000-7fa5e3336000 r--p 0000d000 fe:05 35093                      /usr/lib/x86_64-linux-gnu/liblber-2.4.so.2.8.3
       18123:20140114:200848.888 7fa5e3336000-7fa5e3337000 rw-p 0000e000 fe:05 35093                      /usr/lib/x86_64-linux-gnu/liblber-2.4.so.2.8.3
       18123:20140114:200848.889 7fa5e3337000-7fa5e3383000 r-xp 00000000 fe:05 35094                      /usr/lib/x86_64-linux-gnu/libldap_r-2.4.so.2.8.3
       18123:20140114:200848.889 7fa5e3383000-7fa5e3583000 ---p 0004c000 fe:05 35094                      /usr/lib/x86_64-linux-gnu/libldap_r-2.4.so.2.8.3
       18123:20140114:200848.889 7fa5e3583000-7fa5e3585000 r--p 0004c000 fe:05 35094                      /usr/lib/x86_64-linux-gnu/libldap_r-2.4.so.2.8.3
       18123:20140114:200848.889 7fa5e3585000-7fa5e3586000 rw-p 0004e000 fe:05 35094                      /usr/lib/x86_64-linux-gnu/libldap_r-2.4.so.2.8.3
       18123:20140114:200848.889 7fa5e3586000-7fa5e3588000 rw-p 00000000 00:00 0 
       18123:20140114:200848.889 7fa5e3588000-7fa5e358d000 r-xp 00000000 fe:05 35162                      /usr/lib/libOpenIPMIposix.so.0.0.1
       18123:20140114:200848.889 7fa5e358d000-7fa5e378c000 ---p 00005000 fe:05 35162                      /usr/lib/libOpenIPMIposix.so.0.0.1
       18123:20140114:200848.889 7fa5e378c000-7fa5e378d000 rw-p 00004000 fe:05 35162                      /usr/lib/libOpenIPMIposix.so.0.0.1
       18123:20140114:200848.889 7fa5e378d000-7fa5e3876000 r-xp 00000000 fe:05 35160                      /usr/lib/libOpenIPMI.so.0.0.5
       18123:20140114:200848.889 7fa5e3876000-7fa5e3a76000 ---p 000e9000 fe:05 35160                      /usr/lib/libOpenIPMI.so.0.0.5
       18123:20140114:200848.889 7fa5e3a76000-7fa5e3a92000 rw-p 000e9000 fe:05 35160                      /usr/lib/libOpenIPMI.so.0.0.5
       18123:20140114:200848.889 7fa5e3a92000-7fa5e3a96000 rw-p 00000000 00:00 0 
       18123:20140114:200848.889 7fa5e3a96000-7fa5e3abd000 r-xp 00000000 fe:05 35108                      /usr/lib/x86_64-linux-gnu/libssh2.so.1.0.1
       18123:20140114:200848.889 7fa5e3abd000-7fa5e3cbd000 ---p 00027000 fe:05 35108                      /usr/lib/x86_64-linux-gnu/libssh2.so.1.0.1
       18123:20140114:200848.889 7fa5e3cbd000-7fa5e3cbe000 r--p 00027000 fe:05 35108                      /usr/lib/x86_64-linux-gnu/libssh2.so.1.0.1
       18123:20140114:200848.889 7fa5e3cbe000-7fa5e3cbf000 rw-p 00028000 fe:05 35108                      /usr/lib/x86_64-linux-gnu/libssh2.so.1.0.1
       18123:20140114:200848.889 7fa5e3cbf000-7fa5e3e76000 r-xp 00000000 fe:05 4960                       /usr/lib/x86_64-linux-gnu/libcrypto.so.1.0.0
       18123:20140114:200848.889 7fa5e3e76000-7fa5e4075000 ---p 001b7000 fe:05 4960                       /usr/lib/x86_64-linux-gnu/libcrypto.so.1.0.0
       18123:20140114:200848.889 7fa5e4075000-7fa5e4090000 r--p 001b6000 fe:05 4960                       /usr/lib/x86_64-linux-gnu/libcrypto.so.1.0.0
       18123:20140114:200848.889 7fa5e4090000-7fa5e409f000 rw-p 001d1000 fe:05 4960                       /usr/lib/x86_64-linux-gnu/libcrypto.so.1.0.0
       18123:20140114:200848.889 7fa5e409f000-7fa5e40a3000 rw-p 00000000 00:00 0 
       18123:20140114:200848.890 7fa5e40a3000-7fa5e413d000 r-xp 00000000 fe:05 35291                      /usr/lib/libnetsnmp.so.15.1.2
       18123:20140114:200848.890 7fa5e413d000-7fa5e433d000 ---p 0009a000 fe:05 35291                      /usr/lib/libnetsnmp.so.15.1.2
       18123:20140114:200848.890 7fa5e433d000-7fa5e433e000 r--p 0009a000 fe:05 35291                      /usr/lib/libnetsnmp.so.15.1.2
       18123:20140114:200848.890 7fa5e433e000-7fa5e4341000 rw-p 0009b000 fe:05 35291                      /usr/lib/libnetsnmp.so.15.1.2
       18123:20140114:200848.890 7fa5e4341000-7fa5e4376000 rw-p 00000000 00:00 0 
       18123:20140114:200848.890 7fa5e4376000-7fa5e43d6000 r-xp 00000000 fe:05 28194                      /usr/lib/x86_64-linux-gnu/libodbc.so.1.0.0
       18123:20140114:200848.890 7fa5e43d6000-7fa5e45d5000 ---p 00060000 fe:05 28194                      /usr/lib/x86_64-linux-gnu/libodbc.so.1.0.0
       18123:20140114:200848.890 7fa5e45d5000-7fa5e45dd000 rw-p 0005f000 fe:05 28194                      /usr/lib/x86_64-linux-gnu/libodbc.so.1.0.0
       18123:20140114:200848.890 7fa5e45dd000-7fa5e45de000 rw-p 00000000 00:00 0 
       18123:20140114:200848.890 7fa5e45de000-7fa5e463f000 r-xp 00000000 fe:05 28125                      /usr/lib/x86_64-linux-gnu/libcurl-gnutls.so.4.2.0
       18123:20140114:200848.890 7fa5e463f000-7fa5e483e000 ---p 00061000 fe:05 28125                      /usr/lib/x86_64-linux-gnu/libcurl-gnutls.so.4.2.0
       18123:20140114:200848.890 7fa5e483e000-7fa5e4841000 r--p 00060000 fe:05 28125                      /usr/lib/x86_64-linux-gnu/libcurl-gnutls.so.4.2.0
       18123:20140114:200848.890 7fa5e4841000-7fa5e4842000 rw-p 00063000 fe:05 28125                      /usr/lib/x86_64-linux-gnu/libcurl-gnutls.so.4.2.0
       18123:20140114:200848.890 7fa5e4842000-7fa5e484e000 r-xp 00000000 fe:05 35135                      /usr/lib/libiksemel.so.3.0.0
       18123:20140114:200848.890 7fa5e484e000-7fa5e4a4e000 ---p 0000c000 fe:05 35135                      /usr/lib/libiksemel.so.3.0.0
       18123:20140114:200848.890 7fa5e4a4e000-7fa5e4a4f000 rw-p 0000c000 fe:05 35135                      /usr/lib/libiksemel.so.3.0.0
       18123:20140114:200848.890 7fa5e4a4f000-7fa5e4d0e000 r-xp 00000000 fe:05 34531                      /usr/lib/x86_64-linux-gnu/libmysqlclient.so.18.0.0
       18123:20140114:200848.890 7fa5e4d0e000-7fa5e4f0d000 ---p 002bf000 fe:05 34531                      /usr/lib/x86_64-linux-gnu/libmysqlclient.so.18.0.0
       18123:20140114:200848.890 7fa5e4f0d000-7fa5e4f13000 r--p 002be000 fe:05 34531                      /usr/lib/x86_64-linux-gnu/libmysqlclient.so.18.0.0
       18123:20140114:200848.891 7fa5e4f13000-7fa5e4f91000 rw-p 002c4000 fe:05 34531                      /usr/lib/x86_64-linux-gnu/libmysqlclient.so.18.0.0
       18123:20140114:200848.891 7fa5e4f91000-7fa5e4f96000 rw-p 00000000 00:00 0 
       18123:20140114:200848.891 7fa5e4f96000-7fa5e4fb6000 r-xp 00000000 fe:00 71                         /lib/x86_64-linux-gnu/ld-2.13.so
       18123:20140114:200848.891 7fa5e4fcc000-7fa5e5001000 r--s 00000000 fe:03 2279                       /var/cache/nscd/group
       18123:20140114:200848.891 7fa5e5001000-7fa5e5036000 r--s 00000000 fe:03 2277                       /var/cache/nscd/passwd
       18123:20140114:200848.891 7fa5e5036000-7fa5e5049000 rw-p 00000000 00:00 0 
       18123:20140114:200848.891 7fa5e5049000-7fa5e519f000 r-xp 00000000 fe:05 5787                       /usr/lib/x86_64-linux-gnu/libxml2.so.2.8.0
       18123:20140114:200848.891 7fa5e519f000-7fa5e51a0000 ---p 00156000 fe:05 5787                       /usr/lib/x86_64-linux-gnu/libxml2.so.2.8.0
       18123:20140114:200848.891 7fa5e51a0000-7fa5e51a8000 r--p 00156000 fe:05 5787                       /usr/lib/x86_64-linux-gnu/libxml2.so.2.8.0
       18123:20140114:200848.891 7fa5e51a8000-7fa5e51aa000 rw-p 0015e000 fe:05 5787                       /usr/lib/x86_64-linux-gnu/libxml2.so.2.8.0
       18123:20140114:200848.891 7fa5e51aa000-7fa5e51ac000 rw-p 00000000 00:00 0 
       18123:20140114:200848.891 7fa5e51b2000-7fa5e51b3000 rw-p 00000000 00:00 0 
       18123:20140114:200848.891 7fa5e51b3000-7fa5e51b5000 rw-p 00000000 00:00 0 
       18123:20140114:200848.891 7fa5e51b5000-7fa5e51b6000 r--p 0001f000 fe:00 71                         /lib/x86_64-linux-gnu/ld-2.13.so
       18123:20140114:200848.891 7fa5e51b6000-7fa5e51b7000 rw-p 00020000 fe:00 71                         /lib/x86_64-linux-gnu/ld-2.13.so
       18123:20140114:200848.891 7fa5e51b7000-7fa5e51b8000 rw-p 00000000 00:00 0 
       18123:20140114:200848.891 7fffa6ce8000-7fffa6d09000 rw-p 00000000 00:00 0                          [stack]
       18123:20140114:200848.891 7fffa6dbd000-7fffa6dbe000 r-xp 00000000 00:00 0                          [vdso]
       18123:20140114:200848.891 ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0                  [vsyscall]
       18123:20140114:200848.891 ================================
       18123:20140114:200848.891 Please consider attaching a disassembly listing to your bug report.
       18123:20140114:200848.891 This listing can be produced with, e.g., objdump -DSswx zabbix_server.
       18123:20140114:200848.891 ================================
       17996:20140114:200848.900 One child process died (PID:18123,exitcode/signal:255). Exiting ...
      /usr/lib/zabbix/externalscripts/check_ipmi_numsensor.sh: line 7: echo: write error: Broken pipe
       17996:20140114:200850.901 syncing history data...
       17996:20140114:200850.907 Got signal [signal:11(SIGSEGV),reason:128,refaddr:(nil)]. Crashing ...
       17996:20140114:200850.907 ====== Fatal information: ======
       17996:20140114:200850.907 Program counter: 0x7fa5e257ae21
       17996:20140114:200850.907 === Registers: ===
       17996:20140114:200850.907 r8      =     7fa5e287fed8 =      140350446894808 =      140350446894808
       17996:20140114:200850.907 r9      = 6972756365532d73 =  7598264591089610099 =  7598264591089610099
       17996:20140114:200850.907 r10     = 74696475412d7974 =  8388346235721709940 =  8388346235721709940
       17996:20140114:200850.907 r11     =              768 =                 1896 =                 1896
       17996:20140114:200850.907 r12     =           4bf2b3 =              4977331 =              4977331
       17996:20140114:200850.907 r13     =              2ac =                  684 =                  684
       17996:20140114:200850.907 r14     =     7fa54f972128 =      140347981635880 =      140347981635880
       17996:20140114:200850.907 r15     =     7fa54e29c59c =      140347957691804 =      140347957691804
       17996:20140114:200850.907 rdi     = 3320646e61202965 =  3684054920427743589 =  3684054920427743589
       17996:20140114:200850.907 rsi     =              2ac =                  684 =                  684
       17996:20140114:200850.908 rbp     = 3320646e61202965 =  3684054920427743589 =  3684054920427743589
       17996:20140114:200850.908 rbx     = 3320646e61202965 =  3684054920427743589 =  3684054920427743589
       17996:20140114:200850.908 rdx     =                0 =                    0 =                    0
       17996:20140114:200850.908 rax     =                0 =                    0 =                    0
       17996:20140114:200850.908 rcx     =               25 =                   37 =                   37
       17996:20140114:200850.908 rsp     =     7fffa6d06208 =      140735992062472 =      140735992062472
       17996:20140114:200850.908 rip     =     7fa5e257ae21 =      140350443728417 =      140350443728417
       17996:20140114:200850.908 efl     =            10287 =                66183 =                66183
       17996:20140114:200850.908 csgsfs  =               33 =                   51 =                   51
       17996:20140114:200850.908 err     =                0 =                    0 =                    0
       17996:20140114:200850.908 trapno  =                d =                   13 =                   13
       17996:20140114:200850.908 oldmask =            10000 =                65536 =                65536
       17996:20140114:200850.908 cr2     =                0 =                    0 =                    0
       17996:20140114:200850.908 === Backtrace: ===
       17996:20140114:200850.917 17: /usr/sbin/zabbix_server(print_fatal_info+0x9e) [0x461cae]
       17996:20140114:200850.917 16: /usr/sbin/zabbix_server() [0x4620b7]
       17996:20140114:200850.917 15: /lib/x86_64-linux-gnu/libc.so.6(+0x324f0) [0x7fa5e252c4f0]
       17996:20140114:200850.917 14: /lib/x86_64-linux-gnu/libc.so.6(+0x80e21) [0x7fa5e257ae21]
       17996:20140114:200850.917 13: /lib/x86_64-linux-gnu/libc.so.6(__strdup+0x16) [0x7fa5e257ab36]
       17996:20140114:200850.917 12: /usr/sbin/zabbix_server(zbx_strdup2+0x3f) [0x46622f]
       17996:20140114:200850.917 11: /usr/sbin/zabbix_server() [0x45879f]
       17996:20140114:200850.917 10: /usr/sbin/zabbix_server() [0x458ce1]
       17996:20140114:200850.917 9: /usr/sbin/zabbix_server(zbx_vc_add_value+0x5d1) [0x45ba61]
       17996:20140114:200850.917 8: /usr/sbin/zabbix_server(DCsync_history+0x1cff) [0x44d15f]
       17996:20140114:200850.917 7: /usr/sbin/zabbix_server(free_database_cache+0x35) [0x44e215]
       17996:20140114:200850.917 6: /usr/sbin/zabbix_server(zbx_on_exit+0xee) [0x419fbe]
       17996:20140114:200850.917 5: /lib/x86_64-linux-gnu/libc.so.6(+0x324f0) [0x7fa5e252c4f0]
       17996:20140114:200850.917 4: /lib/x86_64-linux-gnu/libc.so.6(wait+0x53) [0x7fa5e25a5753]
       17996:20140114:200850.917 3: /usr/sbin/zabbix_server(MAIN_ZABBIX_ENTRY+0x4ab) [0x41a4eb]
       17996:20140114:200850.917 2: /usr/sbin/zabbix_server(daemon_start+0x18d) [0x4615cd]
       17996:20140114:200850.917 1: /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xfd) [0x7fa5e2518ead]
       17996:20140114:200850.917 0: /usr/sbin/zabbix_server() [0x415841]
       17996:20140114:200850.917 === Memory map: ===
       17996:20140114:200850.917 00400000-0054b000 r-xp 00000000 fe:05 3601                               /usr/sbin/zabbix_server
       17996:20140114:200850.917 0074a000-0074e000 rw-p 0014a000 fe:05 3601                               /usr/sbin/zabbix_server
       17996:20140114:200850.917 0074e000-00755000 rw-p 00000000 00:00 0 
       17996:20140114:200850.918 0151c000-0153d000 rw-p 00000000 00:00 0                                  [heap]
       17996:20140114:200850.918 0153d000-01574000 rw-p 00000000 00:00 0                                  [heap]
       17996:20140114:200850.918 7fa54d9d1000-7fa54d9d2000 rw-p 00000000 00:00 0 
       17996:20140114:200850.918 7fa54d9d2000-7fa54da07000 r--s 00000000 fe:03 2285                       /var/cache/nscd/services
       17996:20140114:200850.918 7fa54da07000-7fa54da08000 ---p 00000000 00:00 0 
       17996:20140114:200850.918 7fa54da08000-7fa54e208000 rw-p 00000000 00:00 0 
       17996:20140114:200850.918 7fa54e208000-7fa5ce208000 rw-s 00000000 00:04 786440                     /SYSV760005ee (deleted)
       17996:20140114:200850.918 7fa5ce208000-7fa5cea08000 rw-s 00000000 00:04 753671                     /SYSV770005ee (deleted)
       17996:20140114:200850.918 7fa5cea08000-7fa5cea10000 rw-s 00000000 00:04 720902                     /SYSV530005ee (deleted)
       17996:20140114:200850.918 7fa5cea10000-7fa5cf3aa000 rw-s 00000000 00:04 688133                     /SYSV730005ee (deleted)
       17996:20140114:200850.918 7fa5cf3aa000-7fa5d2a11000 rw-s 00000000 00:04 655364                     /SYSV670005ee (deleted)
       17996:20140114:200850.918 7fa5d2a11000-7fa5d3211000 rw-s 00000000 00:04 622595                     /SYSV740005ee (deleted)
       17996:20140114:200850.918 7fa5d3211000-7fa5db211000 rw-s 00000000 00:04 589826                     /SYSV780005ee (deleted)
       17996:20140114:200850.918 7fa5db211000-7fa5df211000 rw-s 00000000 00:04 557057                     /SYSV680005ee (deleted)
       17996:20140114:200850.918 7fa5df211000-7fa5df236000 r-xp 00000000 fe:00 123                        /lib/x86_64-linux-gnu/libtinfo.so.5.9
       17996:20140114:200850.918 7fa5df236000-7fa5df435000 ---p 00025000 fe:00 123                        /lib/x86_64-linux-gnu/libtinfo.so.5.9
       17996:20140114:200850.918 7fa5df435000-7fa5df439000 r--p 00024000 fe:00 123                        /lib/x86_64-linux-gnu/libtinfo.so.5.9
       17996:20140114:200850.918 7fa5df439000-7fa5df43a000 rw-p 00028000 fe:00 123                        /lib/x86_64-linux-gnu/libtinfo.so.5.9
       17996:20140114:200850.918 7fa5df43a000-7fa5df43d000 r-xp 00000000 fe:00 574                        /lib/x86_64-linux-gnu/libgpg-error.so.0.8.0
       17996:20140114:200850.918 7fa5df43d000-7fa5df63c000 ---p 00003000 fe:00 574                        /lib/x86_64-linux-gnu/libgpg-error.so.0.8.0
       17996:20140114:200850.918 7fa5df63c000-7fa5df63d000 rw-p 00002000 fe:00 574                        /lib/x86_64-linux-gnu/libgpg-error.so.0.8.0
       17996:20140114:200850.918 7fa5df63d000-7fa5df640000 r-xp 00000000 fe:00 562                        /lib/x86_64-linux-gnu/libkeyutils.so.1.4
       17996:20140114:200850.918 7fa5df640000-7fa5df83f000 ---p 00003000 fe:00 562                        /lib/x86_64-linux-gnu/libkeyutils.so.1.4
       17996:20140114:200850.918 7fa5df83f000-7fa5df840000 r--p 00002000 fe:00 562                        /lib/x86_64-linux-gnu/libkeyutils.so.1.4
       17996:20140114:200850.919 7fa5df840000-7fa5df841000 rw-p 00003000 fe:00 562                        /lib/x86_64-linux-gnu/libkeyutils.so.1.4
       17996:20140114:200850.919 7fa5df841000-7fa5df849000 r-xp 00000000 fe:05 10741                      /usr/lib/x86_64-linux-gnu/libkrb5support.so.0.1
       17996:20140114:200850.919 7fa5df849000-7fa5dfa48000 ---p 00008000 fe:05 10741                      /usr/lib/x86_64-linux-gnu/libkrb5support.so.0.1
       17996:20140114:200850.919 7fa5dfa48000-7fa5dfa49000 r--p 00007000 fe:05 10741                      /usr/lib/x86_64-linux-gnu/libkrb5support.so.0.1
       17996:20140114:200850.919 7fa5dfa49000-7fa5dfa4a000 rw-p 00008000 fe:05 10741                      /usr/lib/x86_64-linux-gnu/libkrb5support.so.0.1
       17996:20140114:200850.919 7fa5dfa4a000-7fa5dfa4d000 r-xp 00000000 fe:00 207                        /lib/x86_64-linux-gnu/libcom_err.so.2.1
       17996:20140114:200850.919 7fa5dfa4d000-7fa5dfc4c000 ---p 00003000 fe:00 207                        /lib/x86_64-linux-gnu/libcom_err.so.2.1
       17996:20140114:200850.919 7fa5dfc4c000-7fa5dfc4d000 r--p 00002000 fe:00 207                        /lib/x86_64-linux-gnu/libcom_err.so.2.1
       17996:20140114:200850.919 7fa5dfc4d000-7fa5dfc4e000 rw-p 00003000 fe:00 207                        /lib/x86_64-linux-gnu/libcom_err.so.2.1
       17996:20140114:200850.919 7fa5dfc4e000-7fa5dfc74000 r-xp 00000000 fe:05 10747                      /usr/lib/x86_64-linux-gnu/libk5crypto.so.3.1
       17996:20140114:200850.919 7fa5dfc74000-7fa5dfe74000 ---p 00026000 fe:05 10747                      /usr/lib/x86_64-linux-gnu/libk5crypto.so.3.1
       17996:20140114:200850.919 7fa5dfe74000-7fa5dfe75000 r--p 00026000 fe:05 10747                      /usr/lib/x86_64-linux-gnu/libk5crypto.so.3.1
       17996:20140114:200850.919 7fa5dfe75000-7fa5dfe76000 rw-p 00027000 fe:05 10747                      /usr/lib/x86_64-linux-gnu/libk5crypto.so.3.1
       17996:20140114:200850.919 7fa5dfe76000-7fa5dfe77000 rw-p 00000000 00:00 0 
       17996:20140114:200850.919 7fa5dfe77000-7fa5dff40000 r-xp 00000000 fe:05 10753                      /usr/lib/x86_64-linux-gnu/libkrb5.so.3.3
       17996:20140114:200850.919 7fa5dff40000-7fa5e013f000 ---p 000c9000 fe:05 10753                      /usr/lib/x86_64-linux-gnu/libkrb5.so.3.3
       17996:20140114:200850.919 7fa5e013f000-7fa5e0149000 r--p 000c8000 fe:05 10753                      /usr/lib/x86_64-linux-gnu/libkrb5.so.3.3
       17996:20140114:200850.919 7fa5e0149000-7fa5e014b000 rw-p 000d2000 fe:05 10753                      /usr/lib/x86_64-linux-gnu/libkrb5.so.3.3
       17996:20140114:200850.919 7fa5e014b000-7fa5e015c000 r-xp 00000000 fe:05 4646                       /usr/lib/x86_64-linux-gnu/libp11-kit.so.0.0.0
       17996:20140114:200850.919 7fa5e015c000-7fa5e035b000 ---p 00011000 fe:05 4646                       /usr/lib/x86_64-linux-gnu/libp11-kit.so.0.0.0
       17996:20140114:200850.919 7fa5e035b000-7fa5e035c000 r--p 00010000 fe:05 4646                       /usr/lib/x86_64-linux-gnu/libp11-kit.so.0.0.0
       17996:20140114:200850.919 7fa5e035c000-7fa5e035d000 rw-p 00011000 fe:05 4646                       /usr/lib/x86_64-linux-gnu/libp11-kit.so.0.0.0
       17996:20140114:200850.919 7fa5e035d000-7fa5e036d000 r-xp 00000000 fe:05 4656                       /usr/lib/x86_64-linux-gnu/libtasn1.so.3.1.16
       17996:20140114:200850.919 7fa5e036d000-7fa5e056c000 ---p 00010000 fe:05 4656                       /usr/lib/x86_64-linux-gnu/libtasn1.so.3.1.16
       17996:20140114:200850.920 7fa5e056c000-7fa5e056d000 r--p 0000f000 fe:05 4656                       /usr/lib/x86_64-linux-gnu/libtasn1.so.3.1.16
       17996:20140114:200850.920 7fa5e056d000-7fa5e056e000 rw-p 00010000 fe:05 4656                       /usr/lib/x86_64-linux-gnu/libtasn1.so.3.1.16
       17996:20140114:200850.920 7fa5e056e000-7fa5e0588000 r-xp 00000000 fe:05 35080                      /usr/lib/x86_64-linux-gnu/libsasl2.so.2.0.25
       17996:20140114:200850.920 7fa5e0588000-7fa5e0787000 ---p 0001a000 fe:05 35080                      /usr/lib/x86_64-linux-gnu/libsasl2.so.2.0.25
       17996:20140114:200850.920 7fa5e0787000-7fa5e0788000 r--p 00019000 fe:05 35080                      /usr/lib/x86_64-linux-gnu/libsasl2.so.2.0.25
       17996:20140114:200850.920 7fa5e0788000-7fa5e0789000 rw-p 0001a000 fe:05 35080                      /usr/lib/x86_64-linux-gnu/libsasl2.so.2.0.25
       17996:20140114:200850.920 7fa5e0789000-7fa5e07aa000 r-xp 00000000 fe:00 131                        /lib/x86_64-linux-gnu/libncurses.so.5.9
       17996:20140114:200850.920 7fa5e07aa000-7fa5e09a9000 ---p 00021000 fe:00 131                        /lib/x86_64-linux-gnu/libncurses.so.5.9
       17996:20140114:200850.920 7fa5e09a9000-7fa5e09aa000 r--p 00020000 fe:00 131                        /lib/x86_64-linux-gnu/libncurses.so.5.9
       17996:20140114:200850.920 7fa5e09aa000-7fa5e09ab000 rw-p 00021000 fe:00 131                        /lib/x86_64-linux-gnu/libncurses.so.5.9
       17996:20140114:200850.920 7fa5e09ab000-7fa5e09b3000 r-xp 00000000 fe:05 35163                      /usr/lib/libOpenIPMIutils.so.0.0.1
       17996:20140114:200850.920 7fa5e09b3000-7fa5e0bb2000 ---p 00008000 fe:05 35163                      /usr/lib/libOpenIPMIutils.so.0.0.1
       17996:20140114:200850.920 7fa5e0bb2000-7fa5e0bb3000 rw-p 00007000 fe:05 35163                      /usr/lib/libOpenIPMIutils.so.0.0.1
       17996:20140114:200850.920 7fa5e0bb3000-7fa5e0bbc000 r-xp 00000000 fe:05 28181                      /usr/lib/x86_64-linux-gnu/libltdl.so.7.3.0
       17996:20140114:200850.920 7fa5e0bbc000-7fa5e0dbb000 ---p 00009000 fe:05 28181                      /usr/lib/x86_64-linux-gnu/libltdl.so.7.3.0
       17996:20140114:200850.920 7fa5e0dbb000-7fa5e0dbc000 rw-p 00008000 fe:05 28181                      /usr/lib/x86_64-linux-gnu/libltdl.so.7.3.0
       17996:20140114:200850.920 7fa5e0dbc000-7fa5e0dd5000 r-xp 00000000 fe:05 35098                      /usr/lib/x86_64-linux-gnu/librtmp.so.0
       17996:20140114:200850.920 7fa5e0dd5000-7fa5e0fd5000 ---p 00019000 fe:05 35098                      /usr/lib/x86_64-linux-gnu/librtmp.so.0
       17996:20140114:200850.920 7fa5e0fd5000-7fa5e0fd6000 rw-p 00019000 fe:05 35098                      /usr/lib/x86_64-linux-gnu/librtmp.so.0
       17996:20140114:200850.920 7fa5e0fd6000-7fa5e1050000 r-xp 00000000 fe:00 4850                       /lib/x86_64-linux-gnu/libgcrypt.so.11.7.0
       17996:20140114:200850.920 7fa5e1050000-7fa5e1250000 ---p 0007a000 fe:00 4850                       /lib/x86_64-linux-gnu/libgcrypt.so.11.7.0
       17996:20140114:200850.920 7fa5e1250000-7fa5e1251000 r--p 0007a000 fe:00 4850                       /lib/x86_64-linux-gnu/libgcrypt.so.11.7.0
       17996:20140114:200850.920 7fa5e1251000-7fa5e1254000 rw-p 0007b000 fe:00 4850                       /lib/x86_64-linux-gnu/libgcrypt.so.11.7.0
       17996:20140114:200850.920 7fa5e1254000-7fa5e1290000 r-xp 00000000 fe:05 10765                      /usr/lib/x86_64-linux-gnu/libgssapi_krb5.so.2.2
       17996:20140114:200850.920 7fa5e1290000-7fa5e1490000 ---p 0003c000 fe:05 10765                      /usr/lib/x86_64-linux-gnu/libgssapi_krb5.so.2.2
       17996:20140114:200850.920 7fa5e1490000-7fa5e1491000 r--p 0003c000 fe:05 10765                      /usr/lib/x86_64-linux-gnu/libgssapi_krb5.so.2.2
       17996:20140114:200850.920 7fa5e1491000-7fa5e1493000 rw-p 0003d000 fe:05 10765                      /usr/lib/x86_64-linux-gnu/libgssapi_krb5.so.2.2
       17996:20140114:200850.920 7fa5e1493000-7fa5e14c5000 r-xp 00000000 fe:05 4683                       /usr/lib/x86_64-linux-gnu/libidn.so.11.6.8
       17996:20140114:200850.920 7fa5e14c5000-7fa5e16c5000 ---p 00032000 fe:05 4683                       /usr/lib/x86_64-linux-gnu/libidn.so.11.6.8
       17996:20140114:200850.920 7fa5e16c5000-7fa5e16c6000 r--p 00032000 fe:05 4683                       /usr/lib/x86_64-linux-gnu/libidn.so.11.6.8
       17996:20140114:200850.921 7fa5e16c6000-7fa5e16c7000 rw-p 00033000 fe:05 4683                       /usr/lib/x86_64-linux-gnu/libidn.so.11.6.8
       17996:20140114:200850.921 7fa5e16c7000-7fa5e16e9000 r-xp 00000000 fe:00 105                        /lib/x86_64-linux-gnu/liblzma.so.5.0.0
       17996:20140114:200850.921 7fa5e16e9000-7fa5e18e8000 ---p 00022000 fe:00 105                        /lib/x86_64-linux-gnu/liblzma.so.5.0.0
       17996:20140114:200850.921 7fa5e18e8000-7fa5e18e9000 r--p 00021000 fe:00 105                        /lib/x86_64-linux-gnu/liblzma.so.5.0.0
       17996:20140114:200850.921 7fa5e18e9000-7fa5e18ea000 rw-p 00022000 fe:00 105                        /lib/x86_64-linux-gnu/liblzma.so.5.0.0
       17996:20140114:200850.921 7fa5e18ea000-7fa5e19a3000 r-xp 00000000 fe:05 4679                       /usr/lib/x86_64-linux-gnu/libgnutls.so.26.22.4
       17996:20140114:200850.921 7fa5e19a3000-7fa5e1ba2000 ---p 000b9000 fe:05 4679                       /usr/lib/x86_64-linux-gnu/libgnutls.so.26.22.4
       17996:20140114:200850.921 7fa5e1ba2000-7fa5e1ba8000 r--p 000b8000 fe:05 4679                       /usr/lib/x86_64-linux-gnu/libgnutls.so.26.22.4
       17996:20140114:200850.921 7fa5e1ba8000-7fa5e1baa000 rw-p 000be000 fe:05 4679                       /usr/lib/x86_64-linux-gnu/libgnutls.so.26.22.4
       17996:20140114:200850.921 7fa5e1baa000-7fa5e1bbf000 r-xp 00000000 fe:00 30                         /lib/x86_64-linux-gnu/libgcc_s.so.1
       17996:20140114:200850.921 7fa5e1bbf000-7fa5e1dbf000 ---p 00015000 fe:00 30                         /lib/x86_64-linux-gnu/libgcc_s.so.1
       17996:20140114:200850.921 7fa5e1dbf000-7fa5e1dc0000 rw-p 00015000 fe:00 30                         /lib/x86_64-linux-gnu/libgcc_s.so.1
       17996:20140114:200850.921 7fa5e1dc0000-7fa5e1ea8000 r-xp 00000000 fe:05 4034                       /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.17
       17996:20140114:200850.921 7fa5e1ea8000-7fa5e20a8000 ---p 000e8000 fe:05 4034                       /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.17
       17996:20140114:200850.921 7fa5e20a8000-7fa5e20b0000 r--p 000e8000 fe:05 4034                       /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.17
       17996:20140114:200850.921 7fa5e20b0000-7fa5e20b2000 rw-p 000f0000 fe:05 4034                       /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.17
       17996:20140114:200850.921 7fa5e20b2000-7fa5e20c7000 rw-p 00000000 00:00 0 
       17996:20140114:200850.921 7fa5e20c7000-7fa5e20dd000 r-xp 00000000 fe:00 332                        /lib/x86_64-linux-gnu/libz.so.1.2.7
       17996:20140114:200850.921 7fa5e20dd000-7fa5e22dc000 ---p 00016000 fe:00 332                        /lib/x86_64-linux-gnu/libz.so.1.2.7
       17996:20140114:200850.921 7fa5e22dc000-7fa5e22dd000 r--p 00015000 fe:00 332                        /lib/x86_64-linux-gnu/libz.so.1.2.7
       17996:20140114:200850.921 7fa5e22dd000-7fa5e22de000 rw-p 00016000 fe:00 332                        /lib/x86_64-linux-gnu/libz.so.1.2.7
       17996:20140114:200850.921 7fa5e22de000-7fa5e22f5000 r-xp 00000000 fe:00 57                         /lib/x86_64-linux-gnu/libpthread-2.13.so
       17996:20140114:200850.921 7fa5e22f5000-7fa5e24f4000 ---p 00017000 fe:00 57                         /lib/x86_64-linux-gnu/libpthread-2.13.so
       17996:20140114:200850.921 7fa5e24f4000-7fa5e24f5000 r--p 00016000 fe:00 57                         /lib/x86_64-linux-gnu/libpthread-2.13.so
       17996:20140114:200850.921 7fa5e24f5000-7fa5e24f6000 rw-p 00017000 fe:00 57                         /lib/x86_64-linux-gnu/libpthread-2.13.so
       17996:20140114:200850.921 7fa5e24f6000-7fa5e24fa000 rw-p 00000000 00:00 0 
       17996:20140114:200850.921 7fa5e24fa000-7fa5e267a000 r-xp 00000000 fe:00 68                         /lib/x86_64-linux-gnu/libc-2.13.so
       17996:20140114:200850.921 7fa5e267a000-7fa5e287a000 ---p 00180000 fe:00 68                         /lib/x86_64-linux-gnu/libc-2.13.so
       17996:20140114:200850.921 7fa5e287a000-7fa5e287e000 r--p 00180000 fe:00 68                         /lib/x86_64-linux-gnu/libc-2.13.so
       17996:20140114:200850.922 7fa5e287e000-7fa5e287f000 rw-p 00184000 fe:00 68                         /lib/x86_64-linux-gnu/libc-2.13.so
       17996:20140114:200850.922 7fa5e287f000-7fa5e2884000 rw-p 00000000 00:00 0 
       17996:20140114:200850.922 7fa5e2884000-7fa5e2897000 r-xp 00000000 fe:00 70                         /lib/x86_64-linux-gnu/libresolv-2.13.so
       17996:20140114:200850.922 7fa5e2897000-7fa5e2a96000 ---p 00013000 fe:00 70                         /lib/x86_64-linux-gnu/libresolv-2.13.so
       17996:20140114:200850.922 7fa5e2a96000-7fa5e2a97000 r--p 00012000 fe:00 70                         /lib/x86_64-linux-gnu/libresolv-2.13.so
       17996:20140114:200850.922 7fa5e2a97000-7fa5e2a98000 rw-p 00013000 fe:00 70                         /lib/x86_64-linux-gnu/libresolv-2.13.so
       17996:20140114:200850.922 7fa5e2a98000-7fa5e2a9a000 rw-p 00000000 00:00 0 
       17996:20140114:200850.922 7fa5e2a9a000-7fa5e2aa1000 r-xp 00000000 fe:00 62                         /lib/x86_64-linux-gnu/librt-2.13.so
       17996:20140114:200850.922 7fa5e2aa1000-7fa5e2ca0000 ---p 00007000 fe:00 62                         /lib/x86_64-linux-gnu/librt-2.13.so
       17996:20140114:200850.922 7fa5e2ca0000-7fa5e2ca1000 r--p 00006000 fe:00 62                         /lib/x86_64-linux-gnu/librt-2.13.so
       17996:20140114:200850.922 7fa5e2ca1000-7fa5e2ca2000 rw-p 00007000 fe:00 62                         /lib/x86_64-linux-gnu/librt-2.13.so
       17996:20140114:200850.922 7fa5e2ca2000-7fa5e2ca4000 r-xp 00000000 fe:00 60                         /lib/x86_64-linux-gnu/libdl-2.13.so
       17996:20140114:200850.922 7fa5e2ca4000-7fa5e2ea4000 ---p 00002000 fe:00 60                         /lib/x86_64-linux-gnu/libdl-2.13.so
       17996:20140114:200850.922 7fa5e2ea4000-7fa5e2ea5000 r--p 00002000 fe:00 60                         /lib/x86_64-linux-gnu/libdl-2.13.so
       17996:20140114:200850.922 7fa5e2ea5000-7fa5e2ea6000 rw-p 00003000 fe:00 60                         /lib/x86_64-linux-gnu/libdl-2.13.so
       17996:20140114:200850.922 7fa5e2ea6000-7fa5e2f27000 r-xp 00000000 fe:00 65                         /lib/x86_64-linux-gnu/libm-2.13.so
       17996:20140114:200850.922 7fa5e2f27000-7fa5e3126000 ---p 00081000 fe:00 65                         /lib/x86_64-linux-gnu/libm-2.13.so
       17996:20140114:200850.922 7fa5e3126000-7fa5e3127000 r--p 00080000 fe:00 65                         /lib/x86_64-linux-gnu/libm-2.13.so
       17996:20140114:200850.922 7fa5e3127000-7fa5e3128000 rw-p 00081000 fe:00 65                         /lib/x86_64-linux-gnu/libm-2.13.so
       17996:20140114:200850.922 7fa5e3128000-7fa5e3136000 r-xp 00000000 fe:05 35093                      /usr/lib/x86_64-linux-gnu/liblber-2.4.so.2.8.3
       17996:20140114:200850.922 7fa5e3136000-7fa5e3335000 ---p 0000e000 fe:05 35093                      /usr/lib/x86_64-linux-gnu/liblber-2.4.so.2.8.3
       17996:20140114:200850.922 7fa5e3335000-7fa5e3336000 r--p 0000d000 fe:05 35093                      /usr/lib/x86_64-linux-gnu/liblber-2.4.so.2.8.3
       17996:20140114:200850.922 7fa5e3336000-7fa5e3337000 rw-p 0000e000 fe:05 35093                      /usr/lib/x86_64-linux-gnu/liblber-2.4.so.2.8.3
       17996:20140114:200850.922 7fa5e3337000-7fa5e3383000 r-xp 00000000 fe:05 35094                      /usr/lib/x86_64-linux-gnu/libldap_r-2.4.so.2.8.3
       17996:20140114:200850.923 7fa5e3383000-7fa5e3583000 ---p 0004c000 fe:05 35094                      /usr/lib/x86_64-linux-gnu/libldap_r-2.4.so.2.8.3
       17996:20140114:200850.923 7fa5e3583000-7fa5e3585000 r--p 0004c000 fe:05 35094                      /usr/lib/x86_64-linux-gnu/libldap_r-2.4.so.2.8.3
       17996:20140114:200850.923 7fa5e3585000-7fa5e3586000 rw-p 0004e000 fe:05 35094                      /usr/lib/x86_64-linux-gnu/libldap_r-2.4.so.2.8.3
       17996:20140114:200850.923 7fa5e3586000-7fa5e3588000 rw-p 00000000 00:00 0 
       17996:20140114:200850.923 7fa5e3588000-7fa5e358d000 r-xp 00000000 fe:05 35162                      /usr/lib/libOpenIPMIposix.so.0.0.1
       17996:20140114:200850.923 7fa5e358d000-7fa5e378c000 ---p 00005000 fe:05 35162                      /usr/lib/libOpenIPMIposix.so.0.0.1
       17996:20140114:200850.923 7fa5e378c000-7fa5e378d000 rw-p 00004000 fe:05 35162                      /usr/lib/libOpenIPMIposix.so.0.0.1
       17996:20140114:200850.923 7fa5e378d000-7fa5e3876000 r-xp 00000000 fe:05 35160                      /usr/lib/libOpenIPMI.so.0.0.5
       17996:20140114:200850.923 7fa5e3876000-7fa5e3a76000 ---p 000e9000 fe:05 35160                      /usr/lib/libOpenIPMI.so.0.0.5
       17996:20140114:200850.923 7fa5e3a76000-7fa5e3a92000 rw-p 000e9000 fe:05 35160                      /usr/lib/libOpenIPMI.so.0.0.5
       17996:20140114:200850.923 7fa5e3a92000-7fa5e3a96000 rw-p 00000000 00:00 0 
       17996:20140114:200850.923 7fa5e3a96000-7fa5e3abd000 r-xp 00000000 fe:05 35108                      /usr/lib/x86_64-linux-gnu/libssh2.so.1.0.1
       17996:20140114:200850.923 7fa5e3abd000-7fa5e3cbd000 ---p 00027000 fe:05 35108                      /usr/lib/x86_64-linux-gnu/libssh2.so.1.0.1
       17996:20140114:200850.923 7fa5e3cbd000-7fa5e3cbe000 r--p 00027000 fe:05 35108                      /usr/lib/x86_64-linux-gnu/libssh2.so.1.0.1
       17996:20140114:200850.923 7fa5e3cbe000-7fa5e3cbf000 rw-p 00028000 fe:05 35108                      /usr/lib/x86_64-linux-gnu/libssh2.so.1.0.1
       17996:20140114:200850.923 7fa5e3cbf000-7fa5e3e76000 r-xp 00000000 fe:05 4960                       /usr/lib/x86_64-linux-gnu/libcrypto.so.1.0.0
       17996:20140114:200850.923 7fa5e3e76000-7fa5e4075000 ---p 001b7000 fe:05 4960                       /usr/lib/x86_64-linux-gnu/libcrypto.so.1.0.0
       17996:20140114:200850.923 7fa5e4075000-7fa5e4090000 r--p 001b6000 fe:05 4960                       /usr/lib/x86_64-linux-gnu/libcrypto.so.1.0.0
       17996:20140114:200850.923 7fa5e4090000-7fa5e409f000 rw-p 001d1000 fe:05 4960                       /usr/lib/x86_64-linux-gnu/libcrypto.so.1.0.0
       17996:20140114:200850.923 7fa5e409f000-7fa5e40a3000 rw-p 00000000 00:00 0 
       17996:20140114:200850.923 7fa5e40a3000-7fa5e413d000 r-xp 00000000 fe:05 35291                      /usr/lib/libnetsnmp.so.15.1.2
       17996:20140114:200850.923 7fa5e413d000-7fa5e433d000 ---p 0009a000 fe:05 35291                      /usr/lib/libnetsnmp.so.15.1.2
       17996:20140114:200850.923 7fa5e433d000-7fa5e433e000 r--p 0009a000 fe:05 35291                      /usr/lib/libnetsnmp.so.15.1.2
       17996:20140114:200850.923 7fa5e433e000-7fa5e4341000 rw-p 0009b000 fe:05 35291                      /usr/lib/libnetsnmp.so.15.1.2
       17996:20140114:200850.923 7fa5e4341000-7fa5e4376000 rw-p 00000000 00:00 0 
       17996:20140114:200850.923 7fa5e4376000-7fa5e43d6000 r-xp 00000000 fe:05 28194                      /usr/lib/x86_64-linux-gnu/libodbc.so.1.0.0
       17996:20140114:200850.923 7fa5e43d6000-7fa5e45d5000 ---p 00060000 fe:05 28194                      /usr/lib/x86_64-linux-gnu/libodbc.so.1.0.0
       17996:20140114:200850.924 7fa5e45d5000-7fa5e45dd000 rw-p 0005f000 fe:05 28194                      /usr/lib/x86_64-linux-gnu/libodbc.so.1.0.0
       17996:20140114:200850.924 7fa5e45dd000-7fa5e45de000 rw-p 00000000 00:00 0 
       17996:20140114:200850.924 7fa5e45de000-7fa5e463f000 r-xp 00000000 fe:05 28125                      /usr/lib/x86_64-linux-gnu/libcurl-gnutls.so.4.2.0
       17996:20140114:200850.924 7fa5e463f000-7fa5e483e000 ---p 00061000 fe:05 28125                      /usr/lib/x86_64-linux-gnu/libcurl-gnutls.so.4.2.0
       17996:20140114:200850.924 7fa5e483e000-7fa5e4841000 r--p 00060000 fe:05 28125                      /usr/lib/x86_64-linux-gnu/libcurl-gnutls.so.4.2.0
       17996:20140114:200850.924 7fa5e4841000-7fa5e4842000 rw-p 00063000 fe:05 28125                      /usr/lib/x86_64-linux-gnu/libcurl-gnutls.so.4.2.0
       17996:20140114:200850.924 7fa5e4842000-7fa5e484e000 r-xp 00000000 fe:05 35135                      /usr/lib/libiksemel.so.3.0.0
       17996:20140114:200850.924 7fa5e484e000-7fa5e4a4e000 ---p 0000c000 fe:05 35135                      /usr/lib/libiksemel.so.3.0.0
       17996:20140114:200850.924 7fa5e4a4e000-7fa5e4a4f000 rw-p 0000c000 fe:05 35135                      /usr/lib/libiksemel.so.3.0.0
       17996:20140114:200850.924 7fa5e4a4f000-7fa5e4d0e000 r-xp 00000000 fe:05 34531                      /usr/lib/x86_64-linux-gnu/libmysqlclient.so.18.0.0
       17996:20140114:200850.924 7fa5e4d0e000-7fa5e4f0d000 ---p 002bf000 fe:05 34531                      /usr/lib/x86_64-linux-gnu/libmysqlclient.so.18.0.0
       17996:20140114:200850.924 7fa5e4f0d000-7fa5e4f13000 r--p 002be000 fe:05 34531                      /usr/lib/x86_64-linux-gnu/libmysqlclient.so.18.0.0
       17996:20140114:200850.924 7fa5e4f13000-7fa5e4f91000 rw-p 002c4000 fe:05 34531                      /usr/lib/x86_64-linux-gnu/libmysqlclient.so.18.0.0
       17996:20140114:200850.924 7fa5e4f91000-7fa5e4f96000 rw-p 00000000 00:00 0 
       17996:20140114:200850.924 7fa5e4f96000-7fa5e4fb6000 r-xp 00000000 fe:00 71                         /lib/x86_64-linux-gnu/ld-2.13.so
       17996:20140114:200850.924 7fa5e4fcc000-7fa5e5001000 r--s 00000000 fe:03 2279                       /var/cache/nscd/group
       17996:20140114:200850.924 7fa5e5001000-7fa5e5036000 r--s 00000000 fe:03 2277                       /var/cache/nscd/passwd
       17996:20140114:200850.924 7fa5e5036000-7fa5e5049000 rw-p 00000000 00:00 0 
       17996:20140114:200850.924 7fa5e5049000-7fa5e519f000 r-xp 00000000 fe:05 5787                       /usr/lib/x86_64-linux-gnu/libxml2.so.2.8.0
       17996:20140114:200850.924 7fa5e519f000-7fa5e51a0000 ---p 00156000 fe:05 5787                       /usr/lib/x86_64-linux-gnu/libxml2.so.2.8.0
       17996:20140114:200850.924 7fa5e51a0000-7fa5e51a8000 r--p 00156000 fe:05 5787                       /usr/lib/x86_64-linux-gnu/libxml2.so.2.8.0
       17996:20140114:200850.924 7fa5e51a8000-7fa5e51aa000 rw-p 0015e000 fe:05 5787                       /usr/lib/x86_64-linux-gnu/libxml2.so.2.8.0
       17996:20140114:200850.924 7fa5e51aa000-7fa5e51ac000 rw-p 00000000 00:00 0 
       17996:20140114:200850.924 7fa5e51b2000-7fa5e51b3000 rw-p 00000000 00:00 0 
       17996:20140114:200850.924 7fa5e51b3000-7fa5e51b5000 rw-p 00000000 00:00 0 
       17996:20140114:200850.924 7fa5e51b5000-7fa5e51b6000 r--p 0001f000 fe:00 71                         /lib/x86_64-linux-gnu/ld-2.13.so
       17996:20140114:200850.924 7fa5e51b6000-7fa5e51b7000 rw-p 00020000 fe:00 71                         /lib/x86_64-linux-gnu/ld-2.13.so
       17996:20140114:200850.925 7fa5e51b7000-7fa5e51b8000 rw-p 00000000 00:00 0 
       17996:20140114:200850.925 7fffa6ce8000-7fffa6d09000 rw-p 00000000 00:00 0                          [stack]
       17996:20140114:200850.925 7fffa6dbd000-7fffa6dbe000 r-xp 00000000 00:00 0                          [vdso]
       17996:20140114:200850.925 ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0                  [vsyscall]
       17996:20140114:200850.925 ================================
       17996:20140114:200850.925 Please consider attaching a disassembly listing to your bug report.
       17996:20140114:200850.925 This listing can be produced with, e.g., objdump -DSswx zabbix_server.
       17996:20140114:200850.925 ================================
      

      Can you help me with this issue?

      Kind regards

      Andreas Niedermann
      Armacom AG

      1. objdump_zbx_221.tar.bz2
        2.99 MB
        Armacom AG
      2. x00_objdump_zbx_222rc1.tar.bz2
        6.00 MB
        Armacom AG
      3. x01_objdump_zbx_222rc1.tar.bz2
        5.84 MB
        Armacom AG
      4. zabbix_server.log
        37 kB
        Armacom AG

        Issue Links

          Activity

          Hide
          Johannes added a comment - - edited
          Show
          Johannes added a comment - - edited maybe related to https://support.zabbix.com/browse/ZBX-7521
          Hide
          Oleksiy Zagorskyi added a comment -

          senfuu, the ZBX-7521 shows a bit different backtrace.

          Andreas, you have now to re-compile "Pre-2.2.2rc1 (stable)" form here http://www.zabbix.com/developers.php and confirm is the crash still happens.

          Every time please show the crash report.

          Show
          Oleksiy Zagorskyi added a comment - senfuu, the ZBX-7521 shows a bit different backtrace. Andreas, you have now to re-compile "Pre-2.2.2rc1 (stable)" form here http://www.zabbix.com/developers.php and confirm is the crash still happens. Every time please show the crash report.
          Hide
          Armacom AG added a comment -

          Oleksiy, thank you for the reply.

          I will compile "Pre-2.2.2rc1" as soon as possible and let you know the result.

          Andreas

          Show
          Armacom AG added a comment - Oleksiy, thank you for the reply. I will compile "Pre-2.2.2rc1" as soon as possible and let you know the result. Andreas
          Hide
          Armacom AG added a comment -

          Hi Oleksiy, I've tried to launch a compiled pre-2.2.2rc1 version of zabbix-server (compiled using the following configure string):
          ./configure --enable-server --with-mysql --with-jabber --with-libxml2 --with-net-snmp --with-ssh2 --with-libcurl --with-unixodbc --with-openipmi --with-iconv

          The difference between the previous version 2.2.1 (installed by deb package management system) and the newer 2.2.2rc1 (compiled from source) is:
          2.2.1 -> crashing after some hours
          2.2.2rc1 -> crashing subsequently and directly after starting the binary

          The crash report will be attached to this doc. the objdump is split into 2 parts to allow uploading a size over 10m.

          Do you suggest more testing or anything we can do?

          Show
          Armacom AG added a comment - Hi Oleksiy, I've tried to launch a compiled pre-2.2.2rc1 version of zabbix-server (compiled using the following configure string): ./configure --enable-server --with-mysql --with-jabber --with-libxml2 --with-net-snmp --with-ssh2 --with-libcurl --with-unixodbc --with-openipmi --with-iconv The difference between the previous version 2.2.1 (installed by deb package management system) and the newer 2.2.2rc1 (compiled from source) is: 2.2.1 -> crashing after some hours 2.2.2rc1 -> crashing subsequently and directly after starting the binary The crash report will be attached to this doc. the objdump is split into 2 parts to allow uploading a size over 10m. Do you suggest more testing or anything we can do?
          Hide
          Armacom AG added a comment -

          newest log from 2.2.2rc1 binary

          Show
          Armacom AG added a comment - newest log from 2.2.2rc1 binary
          Hide
          Armacom AG added a comment -

          part one of objdump using v. 2.2.2.rc1

          Show
          Armacom AG added a comment - part one of objdump using v. 2.2.2.rc1
          Hide
          Armacom AG added a comment -

          part two of objdump using v. 2.2.2.rc1

          Show
          Armacom AG added a comment - part two of objdump using v. 2.2.2.rc1
          Hide
          Oleksiy Zagorskyi added a comment - - edited

          From the last Andreas' attached zabbix_server.log:

           14308:20140116:131747.927 Starting Zabbix Server. Zabbix 2.2.2rc1 (revision 41561).
          ...
           14308:20140116:131747.928 NodeID:                      1
          ...
           14421:20140116:131748.291 server #106 started [escalator #1]
          ...
           14421:20140116:131748.364 Got signal [signal:11(SIGSEGV),reason:1,refaddr:(nil)]. Crashing ...
          ...
           14421:20140116:131748.365 === Backtrace: ===
           14421:20140116:131748.365 13: /usr/local/sbin/zabbix_server: escalator [processed 0 escalations in 0.000000 sec, processing escalations](print_fatal_info+0x9e) [0x4620ce]
           14421:20140116:131748.365 12: /usr/local/sbin/zabbix_server: escalator [processed 0 escalations in 0.000000 sec, processing escalations]() [0x4624d7]
           14421:20140116:131748.365 11: /lib/x86_64-linux-gnu/libc.so.6(+0x324f0) [0x7fbdf48ba4f0]
           14421:20140116:131748.365 10: /lib/x86_64-linux-gnu/libc.so.6(+0x80e21) [0x7fbdf4908e21]
           14421:20140116:131748.365 9: /lib/x86_64-linux-gnu/libc.so.6(__strdup+0x16) [0x7fbdf4908b36]
           14421:20140116:131748.365 8: /usr/local/sbin/zabbix_server: escalator [processed 0 escalations in 0.000000 sec, processing escalations](zbx_strdup2+0x3f) [0x46664f]
           14421:20140116:131748.365 7: /usr/local/sbin/zabbix_server: escalator [processed 0 escalations in 0.000000 sec, processing escalations]() [0x414ec2]
           14421:20140116:131748.365 6: /usr/local/sbin/zabbix_server: escalator [processed 0 escalations in 0.000000 sec, processing escalations](substitute_simple_macros+0x3af4) [0x47ec04]
           14421:20140116:131748.365 5: /usr/local/sbin/zabbix_server: escalator [processed 0 escalations in 0.000000 sec, processing escalations]() [0x43d156]
           14421:20140116:131748.365 4: /usr/local/sbin/zabbix_server: escalator [processed 0 escalations in 0.000000 sec, processing escalations](main_escalator_loop+0x785) [0x43ddb5]
           14421:20140116:131748.365 3: /usr/local/sbin/zabbix_server: escalator [processed 0 escalations in 0.000000 sec, processing escalations](MAIN_ZABBIX_ENTRY+0x982) [0x41a822]
           14421:20140116:131748.365 2: /usr/local/sbin/zabbix_server: escalator [processed 0 escalations in 0.000000 sec, processing escalations](daemon_start+0x18d) [0x4619ed]
           14421:20140116:131748.365 1: /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xfd) [0x7fbdf48a6ead]
           14421:20140116:131748.365 0: /usr/local/sbin/zabbix_server: escalator [processed 0 escalations in 0.000000 sec, processing escalations]() [0x41569d]
          ...
           14308:20140116:131754.409 Zabbix Server stopped. Zabbix 2.2.2rc1 (revision 41561).
          

          Indeed, it's right after daemon start.
          But currently it's in an escalator process !
          One possibly interesting thing which is the same - it happens after calling zbx_strdup2() function.

          Show
          Oleksiy Zagorskyi added a comment - - edited From the last Andreas' attached zabbix_server.log: 14308:20140116:131747.927 Starting Zabbix Server. Zabbix 2.2.2rc1 (revision 41561). ... 14308:20140116:131747.928 NodeID: 1 ... 14421:20140116:131748.291 server #106 started [escalator #1] ... 14421:20140116:131748.364 Got signal [signal:11(SIGSEGV),reason:1,refaddr:(nil)]. Crashing ... ... 14421:20140116:131748.365 === Backtrace: === 14421:20140116:131748.365 13: /usr/local/sbin/zabbix_server: escalator [processed 0 escalations in 0.000000 sec, processing escalations](print_fatal_info+0x9e) [0x4620ce] 14421:20140116:131748.365 12: /usr/local/sbin/zabbix_server: escalator [processed 0 escalations in 0.000000 sec, processing escalations]() [0x4624d7] 14421:20140116:131748.365 11: /lib/x86_64-linux-gnu/libc.so.6(+0x324f0) [0x7fbdf48ba4f0] 14421:20140116:131748.365 10: /lib/x86_64-linux-gnu/libc.so.6(+0x80e21) [0x7fbdf4908e21] 14421:20140116:131748.365 9: /lib/x86_64-linux-gnu/libc.so.6(__strdup+0x16) [0x7fbdf4908b36] 14421:20140116:131748.365 8: /usr/local/sbin/zabbix_server: escalator [processed 0 escalations in 0.000000 sec, processing escalations](zbx_strdup2+0x3f) [0x46664f] 14421:20140116:131748.365 7: /usr/local/sbin/zabbix_server: escalator [processed 0 escalations in 0.000000 sec, processing escalations]() [0x414ec2] 14421:20140116:131748.365 6: /usr/local/sbin/zabbix_server: escalator [processed 0 escalations in 0.000000 sec, processing escalations](substitute_simple_macros+0x3af4) [0x47ec04] 14421:20140116:131748.365 5: /usr/local/sbin/zabbix_server: escalator [processed 0 escalations in 0.000000 sec, processing escalations]() [0x43d156] 14421:20140116:131748.365 4: /usr/local/sbin/zabbix_server: escalator [processed 0 escalations in 0.000000 sec, processing escalations](main_escalator_loop+0x785) [0x43ddb5] 14421:20140116:131748.365 3: /usr/local/sbin/zabbix_server: escalator [processed 0 escalations in 0.000000 sec, processing escalations](MAIN_ZABBIX_ENTRY+0x982) [0x41a822] 14421:20140116:131748.365 2: /usr/local/sbin/zabbix_server: escalator [processed 0 escalations in 0.000000 sec, processing escalations](daemon_start+0x18d) [0x4619ed] 14421:20140116:131748.365 1: /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xfd) [0x7fbdf48a6ead] 14421:20140116:131748.365 0: /usr/local/sbin/zabbix_server: escalator [processed 0 escalations in 0.000000 sec, processing escalations]() [0x41569d] ... 14308:20140116:131754.409 Zabbix Server stopped. Zabbix 2.2.2rc1 (revision 41561). Indeed, it's right after daemon start. But currently it's in an escalator process ! One possibly interesting thing which is the same - it happens after calling zbx_strdup2() function.
          Hide
          Oleksiy Zagorskyi added a comment -

          Andreas, if it's possible, you could truncate the escalations table and try to start 2.2.2rc1 once again.

          But remember that truncating the table will stop all running escalations, so some users may not get expecting alerts.
          But in most cases is not a problem.

          Show
          Oleksiy Zagorskyi added a comment - Andreas, if it's possible, you could truncate the escalations table and try to start 2.2.2rc1 once again. But remember that truncating the table will stop all running escalations, so some users may not get expecting alerts. But in most cases is not a problem.
          Hide
          Andris Zeila added a comment -

          The crash happens when a log (or logrt) item triggers an event which is processed by action with

          {ITEM.LOG.SOURCE} macro used in subject/message. The {ITEM.LOG.SOURCE}

          field is eventlog specific and will be NULL for log, logrt items, resulting in crash.

          Show
          Andris Zeila added a comment - The crash happens when a log (or logrt) item triggers an event which is processed by action with {ITEM.LOG.SOURCE} macro used in subject/message. The {ITEM.LOG.SOURCE} field is eventlog specific and will be NULL for log, logrt items, resulting in crash.
          Hide
          Andris Zeila added a comment - - edited

          The current idea is to substitute eventlog specific fields (eventid, source, timestamp) with * UNKNOWN * value for log,logrt items.

          Alexander Vladishev Value for {ITEM.LOG.SEVERITY}, {ITEM.LOG.NSEVERITY}, {ITEM.LOG.EVENTID} and {ITEM.LOG.SOURCE} should be *UNKNOWN* for non eventlog[] items.

          Value for {ITEM.LOG.DATE}, {ITEM.LOG.TIME} and {ITEM.LOG.AGE} should be processed for all items with "Type of information" = "Log".

          Show
          Andris Zeila added a comment - - edited The current idea is to substitute eventlog specific fields (eventid, source, timestamp) with * UNKNOWN * value for log,logrt items. Alexander Vladishev Value for {ITEM.LOG.SEVERITY}, {ITEM.LOG.NSEVERITY}, {ITEM.LOG.EVENTID} and {ITEM.LOG.SOURCE} should be * UNKNOWN * for non eventlog[] items. Value for {ITEM.LOG.DATE}, {ITEM.LOG.TIME} and {ITEM.LOG.AGE} should be processed for all items with "Type of information" = "Log".
          Hide
          Andris Zeila added a comment -

          Fixed in development branch svn://svn.zabbix.com/branches/dev/ZBX-7653

          Show
          Andris Zeila added a comment - Fixed in development branch svn://svn.zabbix.com/branches/dev/ZBX-7653
          Hide
          Oleksiy Zagorskyi added a comment - - edited

          Here two supposedly different crashes are listed.
          Both have been fixed, or only escalator related one?

          Andris Zeila I believe the history syncer (value cache related) crash has been already fixed in ZBX-7521 (it contained 2 fixes actually).

          Oleksiy Zagorskyi Thanks. I've just updated issue summary.
          Interesting thing why Andreas (issue reporter) experienced the escalator related crash (last one) after the attempt to use pre-2.2.2
          Could it be that it was a regression after 2.2.1 release ?

          Andris Zeila No, it was introduced in 2.2.0

          Show
          Oleksiy Zagorskyi added a comment - - edited Here two supposedly different crashes are listed. Both have been fixed, or only escalator related one? Andris Zeila I believe the history syncer (value cache related) crash has been already fixed in ZBX-7521 (it contained 2 fixes actually). Oleksiy Zagorskyi Thanks. I've just updated issue summary. Interesting thing why Andreas (issue reporter) experienced the escalator related crash (last one) after the attempt to use pre-2.2.2 Could it be that it was a regression after 2.2.1 release ? Andris Zeila No, it was introduced in 2.2.0
          Hide
          Alexander Vladishev added a comment - - edited

          (1) It is incorrect check for eventlog[] items. eventlog[] values can be without eventid.

                  /* the eventid, source and severity attributes are set only for eventlog[] items */                             
                  if (0 == value.value.log->logeventid)                                                                           
                          goto clean;
          

          Related issue: ZBX-6851

          Andris Zeila Then only option left is to check item key (i.e. read it directly from the database)

          Andris Zeila Added check for item key by using configuration cache. RESOLVED in r41776

          Alexander Vladishev CLOSED Please review my changes in r41787.

          Show
          Alexander Vladishev added a comment - - edited (1) It is incorrect check for eventlog[] items. eventlog[] values can be without eventid. /* the eventid, source and severity attributes are set only for eventlog[] items */ if (0 == value.value.log->logeventid) goto clean; Related issue: ZBX-6851 Andris Zeila Then only option left is to check item key (i.e. read it directly from the database) Andris Zeila Added check for item key by using configuration cache. RESOLVED in r41776 Alexander Vladishev CLOSED Please review my changes in r41787.
          Hide
          Andris Zeila added a comment -

          Released in:
          pre-2.2.2rc1 r41806
          pre-2.3.0 r41807

          Show
          Andris Zeila added a comment - Released in: pre-2.2.2rc1 r41806 pre-2.3.0 r41807
          Hide
          richlv added a comment - - edited

          (2) comment for r41776 said :

          when substituting ITEM.LOG. macros get the required item data from configuration cache instead of database
          

          added that to https://www.zabbix.com/documentation/2.2/manual/introduction/whatsnew222#daemon_improvements - please, review

          Alexander Vladishev Thanks! CLOSED.

          Show
          richlv added a comment - - edited (2) comment for r41776 said : when substituting ITEM.LOG. macros get the required item data from configuration cache instead of database added that to https://www.zabbix.com/documentation/2.2/manual/introduction/whatsnew222#daemon_improvements - please, review Alexander Vladishev Thanks! CLOSED.

            People

            • Assignee:
              Unassigned
              Reporter:
              Armacom AG
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: