[ZBX-10614] Got: Got signal [signal:11(SIGSEGV),reason:1,refaddr:(nil)]. Crashing - in "configuration syncer" Created: 2016 Apr 04  Updated: 2017 May 30  Resolved: 2016 May 06

Status: Closed
Project: ZABBIX BUGS AND ISSUES
Component/s: Server (S)
Affects Version/s: 3.0.1
Fix Version/s: 3.0.3rc1, 3.2.0alpha1

Type: Incident report Priority: Blocker
Reporter: Maxim Semenov Assignee: Unassigned
Resolution: Fixed Votes: 0
Labels: configsyncer, crash
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

vpoller.so


Attachments: Zip Archive zbxdubp.objdump.zip    
Issue Links:
Duplicate
is duplicated by ZBX-10748 crash in "configuration syncer". prob... Closed

 Description   

After about a week of working zabbix died with this error.
I'm using vpoller external library.

  1670:20160404:163920.512 Got signal [signal:11(SIGSEGV),reason:1,refaddr:(nil)]. Crashing ...
  1670:20160404:163920.512 ====== Fatal information: ======
  1670:20160404:163920.512 Program counter: 0x7f3d5dfb8d8a
  1670:20160404:163920.512 === Registers: ===
  1670:20160404:163920.512 r8      =                0 =                    0 =                    0
  1670:20160404:163920.512 r9      = ffffffffffffffff = 18446744073709551615 =                   -1
  1670:20160404:163920.512 r10     =             2881 =                10369 =                10369
  1670:20160404:163920.512 r11     =     7f3d5e0959d0 =      139901547403728 =      139901547403728
  1670:20160404:163920.512 r12     =                0 =                    0 =                    0
  1670:20160404:163920.512 r13     =          11e1aa0 =             18750112 =             18750112
  1670:20160404:163920.512 r14     =     7f3d574e0338 =      139901434463032 =      139901434463032
  1670:20160404:163920.512 r15     =                1 =                    1 =                    1
  1670:20160404:163920.512 rdi     =                0 =                    0 =                    0
  1670:20160404:163920.512 rsi     =          116c670 =             18269808 =             18269808
  1670:20160404:163920.512 rbp     =          116c670 =             18269808 =             18269808
  1670:20160404:163920.512 rbx     =     7f3d574e0350 =      139901434463056 =      139901434463056
  1670:20160404:163920.512 rdx     =                0 =                    0 =                    0
  1670:20160404:163920.512 rax     =              670 =                 1648 =                 1648
  1670:20160404:163920.512 rcx     =                8 =                    8 =                    8
  1670:20160404:163920.512 rsp     =     7ffe93b18528 =      140731376305448 =      140731376305448
  1670:20160404:163920.512 rip     =     7f3d5dfb8d8a =      139901546499466 =      139901546499466
  1670:20160404:163920.512 efl     =            10283 =                66179 =                66179
  1670:20160404:163920.512 csgsfs  =               33 =                   51 =                   51
  1670:20160404:163920.512 err     =                4 =                    4 =                    4
  1670:20160404:163920.512 trapno  =                e =                   14 =                   14
  1670:20160404:163920.512 oldmask =                0 =                    0 =                    0
  1670:20160404:163920.512 cr2     =                0 =                    0 =                    0
  1670:20160404:163920.512 === Backtrace: ===
  1670:20160404:163920.513 12: /usr/sbin/zabbix_server: configuration syncer [synced configuration in 0.091227 sec, syncing configuration](print_fatal_info+0xae) [0x47e9ae]
  1670:20160404:163920.513 11: /usr/sbin/zabbix_server: configuration syncer [synced configuration in 0.091227 sec, syncing configuration]() [0x47ec72]
  1670:20160404:163920.513 10: /lib/x86_64-linux-gnu/libc.so.6(+0x350e0) [0x7f3d5df5c0e0]
  1670:20160404:163920.513 9: /lib/x86_64-linux-gnu/libc.so.6(+0x91d8a) [0x7f3d5dfb8d8a]
  1670:20160404:163920.513 8: /usr/sbin/zabbix_server: configuration syncer [synced configuration in 0.091227 sec, syncing configuration]() [0x461556]
  1670:20160404:163920.513 7: /usr/sbin/zabbix_server: configuration syncer [synced configuration in 0.091227 sec, syncing configuration](DCsync_configuration+0x101a) [0x468a9a]
  1670:20160404:163920.513 6: /usr/sbin/zabbix_server: configuration syncer [synced configuration in 0.091227 sec, syncing configuration](dbconfig_thread+0xf9) [0x41ddf9]
  1670:20160404:163920.513 5: /usr/sbin/zabbix_server: configuration syncer [synced configuration in 0.091227 sec, syncing configuration](zbx_thread_start+0x45) [0x47f6a5]
  1670:20160404:163920.513 4: /usr/sbin/zabbix_server: configuration syncer [synced configuration in 0.091227 sec, syncing configuration](MAIN_ZABBIX_ENTRY+0x44d) [0x41d03d]
  1670:20160404:163920.513 3: /usr/sbin/zabbix_server: configuration syncer [synced configuration in 0.091227 sec, syncing configuration](daemon_start+0x1ba) [0x47e25a]
  1670:20160404:163920.513 2: /usr/sbin/zabbix_server: configuration syncer [synced configuration in 0.091227 sec, syncing configuration](main+0x2f8) [0x417e68]
  1670:20160404:163920.513 1: /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf5) [0x7f3d5df48b45]
  1670:20160404:163920.513 0: /usr/sbin/zabbix_server: configuration syncer [synced configuration in 0.091227 sec, syncing configuration]() [0x418133]
  1670:20160404:163920.513 
=== Memory map: ===
 1670:20160404:163920.513 === Memory map: ===
  1670:20160404:163920.513 00400000-00570000 r-xp 00000000 fe:00 276333                             /usr/sbin/zabbix_server
  1670:20160404:163920.513 0076f000-00770000 r--p 0016f000 fe:00 276333                             /usr/sbin/zabbix_server
  1670:20160404:163920.513 00770000-00777000 rw-p 00170000 fe:00 276333                             /usr/sbin/zabbix_server
  1670:20160404:163920.513 00777000-0077d000 rw-p 00000000 00:00 0
  1670:20160404:163920.513 01160000-011a8000 rw-p 00000000 00:00 0                                  [heap]
  1670:20160404:163920.513 011a8000-011da000 rw-p 00000000 00:00 0                                  [heap]
  1670:20160404:163920.513 011da000-01602000 rw-p 00000000 00:00 0                                  [heap]
  1670:20160404:163920.513 7f3d564be000-7f3d56cbe000 rw-s 00000000 00:04 786440                     /SYSV760000dd (deleted)
  1670:20160404:163920.513 7f3d56cbe000-7f3d574be000 rw-s 00000000 00:04 753671                     /SYSV770000dd (deleted)
  1670:20160404:163920.513 7f3d574be000-7f3d57b8b000 rw-s 00000000 00:04 655364                     /SYSV670000dd (deleted)
  1670:20160404:163920.513 7f3d57b8b000-7f3d57f8b000 rw-s 00000000 00:04 622595                     /SYSV740000dd (deleted)
  1670:20160404:163920.513 7f3d57f8b000-7f3d5838b000 rw-s 00000000 00:04 589826                     /SYSV480000dd (deleted)
  1670:20160404:163920.513 7f3d5838b000-7f3d5938b000 rw-s 00000000 00:04 557057                     /SYSV680000dd (deleted)
  1670:20160404:163920.513 7f3d5938b000-7f3d59392000 r-xp 00000000 fe:00 5396                       /lib/x86_64-linux-gnu/librt-2.19.so
  1670:20160404:163920.513 7f3d59392000-7f3d59591000 ---p 00007000 fe:00 5396                       /lib/x86_64-linux-gnu/librt-2.19.so
  1670:20160404:163920.513 7f3d59591000-7f3d59592000 r--p 00006000 fe:00 5396                       /lib/x86_64-linux-gnu/librt-2.19.so
  1670:20160404:163920.513 7f3d59592000-7f3d59593000 rw-p 00007000 fe:00 5396                       /lib/x86_64-linux-gnu/librt-2.19.so
  1670:20160404:163920.513 7f3d59593000-7f3d595e1000 r-xp 00000000 fe:00 407996                     /usr/local/lib/libzmq.so.4.0.0
  1670:20160404:163920.513 7f3d595e1000-7f3d597e1000 ---p 0004e000 fe:00 407996                     /usr/local/lib/libzmq.so.4.0.0
  1670:20160404:163920.513 7f3d597e1000-7f3d597e5000 rw-p 0004e000 fe:00 407996                     /usr/local/lib/libzmq.so.4.0.0
  1670:20160404:163920.513 7f3d597e5000-7f3d597e7000 r-xp 00000000 fe:00 407490                     /usr/local/lib/zabbix/vpoller.so
  1670:20160404:163920.513 7f3d597e7000-7f3d599e7000 ---p 00002000 fe:00 407490                     /usr/local/lib/zabbix/vpoller.so
  1670:20160404:163920.513 7f3d599e7000-7f3d599e8000 rw-p 00002000 fe:00 407490                     /usr/local/lib/zabbix/vpoller.so
  1670:20160404:163920.513 7f3d599e8000-7f3d599f3000 r-xp 00000000 fe:00 5390                       /lib/x86_64-linux-gnu/libnss_files-2.19.so
  1670:20160404:163920.513 7f3d599f3000-7f3d59bf2000 ---p 0000b000 fe:00 5390                       /lib/x86_64-linux-gnu/libnss_files-2.19.so
  1670:20160404:163920.513 7f3d59bf2000-7f3d59bf3000 r--p 0000a000 fe:00 5390                       /lib/x86_64-linux-gnu/libnss_files-2.19.so
  1670:20160404:163920.513 7f3d59bf3000-7f3d59bf4000 rw-p 0000b000 fe:00 5390                       /lib/x86_64-linux-gnu/libnss_files-2.19.so
  1670:20160404:163920.513 7f3d59bf4000-7f3d59bfe000 r-xp 00000000 fe:00 5392                       /lib/x86_64-linux-gnu/libnss_nis-2.19.so
  1670:20160404:163920.513 7f3d59bfe000-7f3d59dfd000 ---p 0000a000 fe:00 5392                       /lib/x86_64-linux-gnu/libnss_nis-2.19.so
  1670:20160404:163920.513 7f3d59dfd000-7f3d59dfe000 r--p 00009000 fe:00 5392                       /lib/x86_64-linux-gnu/libnss_nis-2.19.so
  1670:20160404:163920.513 7f3d59dfe000-7f3d59dff000 rw-p 0000a000 fe:00 5392                       /lib/x86_64-linux-gnu/libnss_nis-2.19.so
  1670:20160404:163920.513 7f3d59dff000-7f3d59e14000 r-xp 00000000 fe:00 5387                       /lib/x86_64-linux-gnu/libnsl-2.19.so
  1670:20160404:163920.513 7f3d59e14000-7f3d5a013000 ---p 00015000 fe:00 5387                       /lib/x86_64-linux-gnu/libnsl-2.19.so
  1670:20160404:163920.513 7f3d5a013000-7f3d5a014000 r--p 00014000 fe:00 5387                       /lib/x86_64-linux-gnu/libnsl-2.19.so
  1670:20160404:163920.513 7f3d5a014000-7f3d5a015000 rw-p 00015000 fe:00 5387                       /lib/x86_64-linux-gnu/libnsl-2.19.so
  1670:20160404:163920.513 7f3d5a015000-7f3d5a017000 rw-p 00000000 00:00 0
  1670:20160404:163920.513 7f3d5a017000-7f3d5a01e000 r-xp 00000000 fe:00 5388                       /lib/x86_64-linux-gnu/libnss_compat-2.19.so
  1670:20160404:163920.513 7f3d5a01e000-7f3d5a21d000 ---p 00007000 fe:00 5388                       /lib/x86_64-linux-gnu/libnss_compat-2.19.so
  1670:20160404:163920.513 7f3d5a21d000-7f3d5a21e000 r--p 00006000 fe:00 5388                       /lib/x86_64-linux-gnu/libnss_compat-2.19.so
  1670:20160404:163920.513 7f3d5a21e000-7f3d5a21f000 rw-p 00007000 fe:00 5388                       /lib/x86_64-linux-gnu/libnss_compat-2.19.so
  1670:20160404:163920.513 7f3d5a21f000-7f3d5a226000 r-xp 00000000 fe:00 263211                     /usr/lib/x86_64-linux-gnu/libffi.so.6.0.2
  1670:20160404:163920.513 7f3d5a226000-7f3d5a425000 ---p 00007000 fe:00 263211                     /usr/lib/x86_64-linux-gnu/libffi.so.6.0.2
  1670:20160404:163920.513 7f3d5a425000-7f3d5a426000 r--p 00006000 fe:00 263211                     /usr/lib/x86_64-linux-gnu/libffi.so.6.0.2
  1670:20160404:163920.513 7f3d5a426000-7f3d5a427000 rw-p 00007000 fe:00 263211                     /usr/lib/x86_64-linux-gnu/libffi.so.6.0.2
  1670:20160404:163920.513 7f3d5a427000-7f3d5a42a000 r-xp 00000000 fe:00 4620                       /lib/x86_64-linux-gnu/libkeyutils.so.1.5
  1670:20160404:163920.513 7f3d5a42a000-7f3d5a629000 ---p 00003000 fe:00 4620                       /lib/x86_64-linux-gnu/libkeyutils.so.1.5
  1670:20160404:163920.513 7f3d5a629000-7f3d5a62a000 r--p 00002000 fe:00 4620                       /lib/x86_64-linux-gnu/libkeyutils.so.1.5
  1670:20160404:163920.513 7f3d5a62a000-7f3d5a62b000 rw-p 00003000 fe:00 4620                       /lib/x86_64-linux-gnu/libkeyutils.so.1.5
 1670:20160404:163920.513 7f3d5a62b000-7f3d5a636000 r-xp 00000000 fe:00 259228                     /usr/lib/x86_64-linux-gnu/libkrb5support.so.0.1
  1670:20160404:163920.513 7f3d5a636000-7f3d5a835000 ---p 0000b000 fe:00 259228                     /usr/lib/x86_64-linux-gnu/libkrb5support.so.0.1
  1670:20160404:163920.513 7f3d5a835000-7f3d5a836000 r--p 0000a000 fe:00 259228                     /usr/lib/x86_64-linux-gnu/libkrb5support.so.0.1
  1670:20160404:163920.513 7f3d5a836000-7f3d5a837000 rw-p 0000b000 fe:00 259228                     /usr/lib/x86_64-linux-gnu/libkrb5support.so.0.1
  1670:20160404:163920.513 7f3d5a837000-7f3d5a848000 r-xp 00000000 fe:00 80                         /lib/x86_64-linux-gnu/libgpg-error.so.0.13.0
  1670:20160404:163920.513 7f3d5a848000-7f3d5aa47000 ---p 00011000 fe:00 80                         /lib/x86_64-linux-gnu/libgpg-error.so.0.13.0
  1670:20160404:163920.513 7f3d5aa47000-7f3d5aa48000 r--p 00010000 fe:00 80                         /lib/x86_64-linux-gnu/libgpg-error.so.0.13.0
  1670:20160404:163920.513 7f3d5aa48000-7f3d5aa49000 rw-p 00011000 fe:00 80                         /lib/x86_64-linux-gnu/libgpg-error.so.0.13.0
  1670:20160404:163920.513 7f3d5aa49000-7f3d5aaca000 r-xp 00000000 fe:00 263234                     /usr/lib/x86_64-linux-gnu/libgmp.so.10.2.0
  1670:20160404:163920.513 7f3d5aaca000-7f3d5acca000 ---p 00081000 fe:00 263234                     /usr/lib/x86_64-linux-gnu/libgmp.so.10.2.0
  1670:20160404:163920.513 7f3d5acca000-7f3d5accb000 r--p 00081000 fe:00 263234                     /usr/lib/x86_64-linux-gnu/libgmp.so.10.2.0
  1670:20160404:163920.513 7f3d5accb000-7f3d5accc000 rw-p 00082000 fe:00 263234                     /usr/lib/x86_64-linux-gnu/libgmp.so.10.2.0
  1670:20160404:163920.513 7f3d5accc000-7f3d5acfa000 r-xp 00000000 fe:00 263238                     /usr/lib/x86_64-linux-gnu/libhogweed.so.2.5
  1670:20160404:163920.513 7f3d5acfa000-7f3d5aef9000 ---p 0002e000 fe:00 263238                     /usr/lib/x86_64-linux-gnu/libhogweed.so.2.5
  1670:20160404:163920.513 7f3d5aef9000-7f3d5aefa000 r--p 0002d000 fe:00 263238                     /usr/lib/x86_64-linux-gnu/libhogweed.so.2.5
  1670:20160404:163920.514 7f3d5aefa000-7f3d5aefb000 rw-p 0002e000 fe:00 263238                     /usr/lib/x86_64-linux-gnu/libhogweed.so.2.5
  1670:20160404:163920.514 7f3d5aefb000-7f3d5af2b000 r-xp 00000000 fe:00 263236                     /usr/lib/x86_64-linux-gnu/libnettle.so.4.7
  1670:20160404:163920.514 7f3d5af2b000-7f3d5b12b000 ---p 00030000 fe:00 263236                     /usr/lib/x86_64-linux-gnu/libnettle.so.4.7
  1670:20160404:163920.514 7f3d5b12b000-7f3d5b12c000 r--p 00030000 fe:00 263236                     /usr/lib/x86_64-linux-gnu/libnettle.so.4.7
  1670:20160404:163920.514 7f3d5b12c000-7f3d5b12d000 rw-p 00031000 fe:00 263236                     /usr/lib/x86_64-linux-gnu/libnettle.so.4.7
  1670:20160404:163920.514 7f3d5b12d000-7f3d5b13e000 r-xp 00000000 fe:00 263737                     /usr/lib/x86_64-linux-gnu/libtasn1.so.6.3.2
  1670:20160404:163920.514 7f3d5b13e000-7f3d5b33e000 ---p 00011000 fe:00 263737                     /usr/lib/x86_64-linux-gnu/libtasn1.so.6.3.2
  1670:20160404:163920.514 7f3d5b33e000-7f3d5b340000 r--p 00011000 fe:00 263737                     /usr/lib/x86_64-linux-gnu/libtasn1.so.6.3.2
  1670:20160404:163920.514 7f3d5b340000-7f3d5b341000 rw-p 00013000 fe:00 263737                     /usr/lib/x86_64-linux-gnu/libtasn1.so.6.3.2
  1670:20160404:163920.514 7f3d5b341000-7f3d5b37d000 r-xp 00000000 fe:00 263247                     /usr/lib/x86_64-linux-gnu/libp11-kit.so.0.0.0
  1670:20160404:163920.514 7f3d5b37d000-7f3d5b57c000 ---p 0003c000 fe:00 263247                     /usr/lib/x86_64-linux-gnu/libp11-kit.so.0.0.0
  1670:20160404:163920.514 7f3d5b57c000-7f3d5b585000 r--p 0003b000 fe:00 263247                     /usr/lib/x86_64-linux-gnu/libp11-kit.so.0.0.0
  1670:20160404:163920.514 7f3d5b585000-7f3d5b587000 rw-p 00044000 fe:00 263247                     /usr/lib/x86_64-linux-gnu/libp11-kit.so.0.0.0
  1670:20160404:163920.514 7f3d5b587000-7f3d5b58a000 r-xp 00000000 fe:00 88                         /lib/x86_64-linux-gnu/libcom_err.so.2.1
  1670:20160404:163920.514 7f3d5b58a000-7f3d5b789000 ---p 00003000 fe:00 88                         /lib/x86_64-linux-gnu/libcom_err.so.2.1
  1670:20160404:163920.514 7f3d5b789000-7f3d5b78a000 r--p 00002000 fe:00 88                         /lib/x86_64-linux-gnu/libcom_err.so.2.1
  1670:20160404:163920.514 7f3d5b78a000-7f3d5b78b000 rw-p 00003000 fe:00 88                         /lib/x86_64-linux-gnu/libcom_err.so.2.1
  1670:20160404:163920.514 7f3d5b78b000-7f3d5b7b9000 r-xp 00000000 fe:00 259234                     /usr/lib/x86_64-linux-gnu/libk5crypto.so.3.1
  1670:20160404:163920.514 7f3d5b7b9000-7f3d5b9b8000 ---p 0002e000 fe:00 259234                     /usr/lib/x86_64-linux-gnu/libk5crypto.so.3.1
  1670:20160404:163920.514 7f3d5b9b8000-7f3d5b9ba000 r--p 0002d000 fe:00 259234                     /usr/lib/x86_64-linux-gnu/libk5crypto.so.3.1
  1670:20160404:163920.514 7f3d5b9ba000-7f3d5b9bb000 rw-p 0002f000 fe:00 259234                     /usr/lib/x86_64-linux-gnu/libk5crypto.so.3.1
  1670:20160404:163920.514 7f3d5b9bb000-7f3d5b9bc000 rw-p 00000000 00:00 0
  1670:20160404:163920.514 7f3d5b9bc000-7f3d5ba80000 r-xp 00000000 fe:00 259223                     /usr/lib/x86_64-linux-gnu/libkrb5.so.3.3
  1670:20160404:163920.514 7f3d5ba80000-7f3d5bc80000 ---p 000c4000 fe:00 259223                     /usr/lib/x86_64-linux-gnu/libkrb5.so.3.3
  1670:20160404:163920.514 7f3d5bc80000-7f3d5bc8d000 r--p 000c4000 fe:00 259223                     /usr/lib/x86_64-linux-gnu/libkrb5.so.3.3
  1670:20160404:163920.514 7f3d5bc8d000-7f3d5bc90000 rw-p 000d1000 fe:00 259223                     /usr/lib/x86_64-linux-gnu/libkrb5.so.3.3
  1670:20160404:163920.514 7f3d5bc90000-7f3d5bcd7000 r-xp 00000000 fe:00 259213                     /usr/lib/x86_64-linux-gnu/libgssapi_krb5.so.2.2
  1670:20160404:163920.514 7f3d5bcd7000-7f3d5bed7000 ---p 00047000 fe:00 259213                     /usr/lib/x86_64-linux-gnu/libgssapi_krb5.so.2.2
  1670:20160404:163920.514 7f3d5bed7000-7f3d5bed9000 r--p 00047000 fe:00 259213                     /usr/lib/x86_64-linux-gnu/libgssapi_krb5.so.2.2
  1670:20160404:163920.514 7f3d5bed9000-7f3d5bedb000 rw-p 00049000 fe:00 259213                     /usr/lib/x86_64-linux-gnu/libgssapi_krb5.so.2.2
  1670:20160404:163920.514 7f3d5bedb000-7f3d5bef6000 r-xp 00000000 fe:00 269922                     /usr/lib/x86_64-linux-gnu/librtmp.so.1
  1670:20160404:163920.514 7f3d5bef6000-7f3d5c0f6000 ---p 0001b000 fe:00 269922                     /usr/lib/x86_64-linux-gnu/librtmp.so.1
  1670:20160404:163920.514 7f3d5c0f6000-7f3d5c0f8000 r--p 0001b000 fe:00 269922                     /usr/lib/x86_64-linux-gnu/librtmp.so.1
  1670:20160404:163920.514 7f3d5c0f8000-7f3d5c0f9000 rw-p 0001d000 fe:00 269922                     /usr/lib/x86_64-linux-gnu/librtmp.so.1
  1670:20160404:163920.514 7f3d5c0f9000-7f3d5c12b000 r-xp 00000000 fe:00 263538                     /usr/lib/x86_64-linux-gnu/libidn.so.11.6.12
 1670:20160404:163920.514 7f3d5c347000-7f3d5c547000 ---p 0001a000 fe:00 267754                     /usr/lib/x86_64-linux-gnu/libsasl2.so.2.0.25
  1670:20160404:163920.514 7f3d5c547000-7f3d5c548000 r--p 0001a000 fe:00 267754                     /usr/lib/x86_64-linux-gnu/libsasl2.so.2.0.25
  1670:20160404:163920.514 7f3d5c548000-7f3d5c549000 rw-p 0001b000 fe:00 267754                     /usr/lib/x86_64-linux-gnu/libsasl2.so.2.0.25
  1670:20160404:163920.514 7f3d5c549000-7f3d5c56f000 r-xp 00000000 fe:00 114                        /lib/x86_64-linux-gnu/libtinfo.so.5.9
  1670:20160404:163920.514 7f3d5c56f000-7f3d5c76e000 ---p 00026000 fe:00 114                        /lib/x86_64-linux-gnu/libtinfo.so.5.9
  1670:20160404:163920.514 7f3d5c76e000-7f3d5c772000 r--p 00025000 fe:00 114                        /lib/x86_64-linux-gnu/libtinfo.so.5.9
  1670:20160404:163920.514 7f3d5c772000-7f3d5c773000 rw-p 00029000 fe:00 114                        /lib/x86_64-linux-gnu/libtinfo.so.5.9
  1670:20160404:163920.514 7f3d5c773000-7f3d5c797000 r-xp 00000000 fe:00 84                         /lib/x86_64-linux-gnu/libncurses.so.5.9
  1670:20160404:163920.514 7f3d5c797000-7f3d5c996000 ---p 00024000 fe:00 84                         /lib/x86_64-linux-gnu/libncurses.so.5.9
  1670:20160404:163920.514 7f3d5c996000-7f3d5c997000 r--p 00023000 fe:00 84                         /lib/x86_64-linux-gnu/libncurses.so.5.9
  1670:20160404:163920.514 7f3d5c997000-7f3d5c998000 rw-p 00024000 fe:00 84                         /lib/x86_64-linux-gnu/libncurses.so.5.9
  1670:20160404:163920.514 7f3d5c998000-7f3d5c99f000 r-xp 00000000 fe:00 276312                     /usr/lib/libOpenIPMIutils.so.0.0.1
  1670:20160404:163920.514 7f3d5c99f000-7f3d5cb9f000 ---p 00007000 fe:00 276312                     /usr/lib/libOpenIPMIutils.so.0.0.1
  1670:20160404:163920.514 7f3d5cb9f000-7f3d5cba0000 rw-p 00007000 fe:00 276312                     /usr/lib/libOpenIPMIutils.so.0.0.1
  1670:20160404:163920.514 7f3d5cba0000-7f3d5cc78000 r-xp 00000000 fe:00 17                         /lib/x86_64-linux-gnu/libgcrypt.so.20.0.3
  1670:20160404:163920.514 7f3d5cc78000-7f3d5ce78000 ---p 000d8000 fe:00 17                         /lib/x86_64-linux-gnu/libgcrypt.so.20.0.3
  1670:20160404:163920.514 7f3d5ce78000-7f3d5ce79000 r--p 000d8000 fe:00 17                         /lib/x86_64-linux-gnu/libgcrypt.so.20.0.3
  1670:20160404:163920.514 7f3d5ce79000-7f3d5ce82000 rw-p 000d9000 fe:00 17                         /lib/x86_64-linux-gnu/libgcrypt.so.20.0.3
  1670:20160404:163920.514 7f3d5ce82000-7f3d5ce8b000 r-xp 00000000 fe:00 276256                     /usr/lib/x86_64-linux-gnu/libltdl.so.7.3.0
  1670:20160404:163920.514 7f3d5ce8b000-7f3d5d08a000 ---p 00009000 fe:00 276256                     /usr/lib/x86_64-linux-gnu/libltdl.so.7.3.0
  1670:20160404:163920.514 7f3d5d08a000-7f3d5d08b000 r--p 00008000 fe:00 276256                     /usr/lib/x86_64-linux-gnu/libltdl.so.7.3.0
  1670:20160404:163920.514 7f3d5d08b000-7f3d5d08c000 rw-p 00009000 fe:00 276256                     /usr/lib/x86_64-linux-gnu/libltdl.so.7.3.0
  1670:20160404:163920.514 7f3d5d08c000-7f3d5d0ae000 r-xp 00000000 fe:00 51                         /lib/x86_64-linux-gnu/liblzma.so.5.0.0
  1670:20160404:163920.514 7f3d5d0ae000-7f3d5d2ad000 ---p 00022000 fe:00 51                         /lib/x86_64-linux-gnu/liblzma.so.5.0.0
  1670:20160404:163920.514 7f3d5d2ad000-7f3d5d2ae000 r--p 00021000 fe:00 51                         /lib/x86_64-linux-gnu/liblzma.so.5.0.0
  1670:20160404:163920.514 7f3d5d2ae000-7f3d5d2af000 rw-p 00022000 fe:00 51                         /lib/x86_64-linux-gnu/liblzma.so.5.0.0
  1670:20160404:163920.514 7f3d5d2af000-7f3d5d3c2000 r-xp 00000000 fe:00 263273                     /usr/lib/x86_64-linux-gnu/libgnutls-deb0.so.28.41.0
  1670:20160404:163920.514 7f3d5d3c2000-7f3d5d5c2000 ---p 00113000 fe:00 263273                     /usr/lib/x86_64-linux-gnu/libgnutls-deb0.so.28.41.0
  1670:20160404:163920.514 7f3d5d5c2000-7f3d5d5cb000 r--p 00113000 fe:00 263273                     /usr/lib/x86_64-linux-gnu/libgnutls-deb0.so.28.41.0
  1670:20160404:163920.514 7f3d5d5cb000-7f3d5d5cd000 rw-p 0011c000 fe:00 263273                     /usr/lib/x86_64-linux-gnu/libgnutls-deb0.so.28.41.0
  1670:20160404:163920.514 7f3d5d5cd000-7f3d5d5ce000 rw-p 00000000 00:00 0
  1670:20160404:163920.514 7f3d5d5ce000-7f3d5d5e4000 r-xp 00000000 fe:00 15                         /lib/x86_64-linux-gnu/libgcc_s.so.1
  1670:20160404:163920.514 7f3d5d5e4000-7f3d5d7e3000 ---p 00016000 fe:00 15                         /lib/x86_64-linux-gnu/libgcc_s.so.1
  1670:20160404:163920.514 7f3d5d7e3000-7f3d5d7e4000 rw-p 00015000 fe:00 15                         /lib/x86_64-linux-gnu/libgcc_s.so.1
  1670:20160404:163920.514 7f3d5d7e4000-7f3d5d8d0000 r-xp 00000000 fe:00 263670                     /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.20
  1670:20160404:163920.514 7f3d5d8d0000-7f3d5dad0000 ---p 000ec000 fe:00 263670                     /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.20
  1670:20160404:163920.514 7f3d5dad0000-7f3d5dad8000 r--p 000ec000 fe:00 263670                     /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.20
  1670:20160404:163920.514 7f3d5dad8000-7f3d5dada000 rw-p 000f4000 fe:00 263670                     /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.20
  1670:20160404:163920.514 7f3d5dada000-7f3d5daef000 rw-p 00000000 00:00 0
  1670:20160404:163920.514 7f3d5daef000-7f3d5db09000 r-xp 00000000 fe:00 111                        /lib/x86_64-linux-gnu/libz.so.1.2.8
  1670:20160404:163920.514 7f3d5db09000-7f3d5dd08000 ---p 0001a000 fe:00 111                        /lib/x86_64-linux-gnu/libz.so.1.2.8
  1670:20160404:163920.514 7f3d5dd08000-7f3d5dd09000 r--p 00019000 fe:00 111                        /lib/x86_64-linux-gnu/libz.so.1.2.8
  1670:20160404:163920.514 7f3d5dd09000-7f3d5dd0a000 rw-p 0001a000 fe:00 111                        /lib/x86_64-linux-gnu/libz.so.1.2.8
  1670:20160404:163920.514 7f3d5dd0a000-7f3d5dd22000 r-xp 00000000 fe:00 5377                       /lib/x86_64-linux-gnu/libpthread-2.19.so
  1670:20160404:163920.514 7f3d5dd22000-7f3d5df21000 ---p 00018000 fe:00 5377                       /lib/x86_64-linux-gnu/libpthread-2.19.so
  1670:20160404:163920.514 7f3d5df21000-7f3d5df22000 r--p 00017000 fe:00 5377                       /lib/x86_64-linux-gnu/libpthread-2.19.so
  1670:20160404:163920.514 7f3d5df22000-7f3d5df23000 rw-p 00018000 fe:00 5377                       /lib/x86_64-linux-gnu/libpthread-2.19.so
  1670:20160404:163920.514 7f3d5df23000-7f3d5df27000 rw-p 00000000 00:00 0
  1670:20160404:163920.514 7f3d5df27000-7f3d5e0c9000 r-xp 00000000 fe:00 5381                       /lib/x86_64-linux-gnu/libc-2.19.so
  1670:20160404:163920.515 7f3d5e0c9000-7f3d5e2c8000 ---p 001a2000 fe:00 5381                       /lib/x86_64-linux-gnu/libc-2.19.so
 1670:20160404:163920.514 7f3d5df27000-7f3d5e0c9000 r-xp 00000000 fe:00 5381                       /lib/x86_64-linux-gnu/libc-2.19.so
  1670:20160404:163920.515 7f3d5e0c9000-7f3d5e2c8000 ---p 001a2000 fe:00 5381                       /lib/x86_64-linux-gnu/libc-2.19.so
  1670:20160404:163920.515 7f3d5e2c8000-7f3d5e2cc000 r--p 001a1000 fe:00 5381                       /lib/x86_64-linux-gnu/libc-2.19.so
  1670:20160404:163920.515 7f3d5e2cc000-7f3d5e2ce000 rw-p 001a5000 fe:00 5381                       /lib/x86_64-linux-gnu/libc-2.19.so
  1670:20160404:163920.515 7f3d5e2ce000-7f3d5e2d2000 rw-p 00000000 00:00 0
  1670:20160404:163920.515 7f3d5e2d2000-7f3d5e2e6000 r-xp 00000000 fe:00 5395                       /lib/x86_64-linux-gnu/libresolv-2.19.so
  1670:20160404:163920.515 7f3d5e2e6000-7f3d5e4e5000 ---p 00014000 fe:00 5395                       /lib/x86_64-linux-gnu/libresolv-2.19.so
  1670:20160404:163920.515 7f3d5e4e5000-7f3d5e4e6000 r--p 00013000 fe:00 5395                       /lib/x86_64-linux-gnu/libresolv-2.19.so
  1670:20160404:163920.515 7f3d5e4e6000-7f3d5e4e7000 rw-p 00014000 fe:00 5395                       /lib/x86_64-linux-gnu/libresolv-2.19.so
  1670:20160404:163920.515 7f3d5e4e7000-7f3d5e4e9000 rw-p 00000000 00:00 0
  1670:20160404:163920.515 7f3d5e4e9000-7f3d5e4ec000 r-xp 00000000 fe:00 5384                       /lib/x86_64-linux-gnu/libdl-2.19.so
  1670:20160404:163920.515 7f3d5e4ec000-7f3d5e6eb000 ---p 00003000 fe:00 5384                       /lib/x86_64-linux-gnu/libdl-2.19.so
  1670:20160404:163920.515 7f3d5e6eb000-7f3d5e6ec000 r--p 00002000 fe:00 5384                       /lib/x86_64-linux-gnu/libdl-2.19.so
  1670:20160404:163920.515 7f3d5e6ec000-7f3d5e6ed000 rw-p 00003000 fe:00 5384                       /lib/x86_64-linux-gnu/libdl-2.19.so
  1670:20160404:163920.515 7f3d5e6ed000-7f3d5e7ed000 r-xp 00000000 fe:00 5385                       /lib/x86_64-linux-gnu/libm-2.19.so
  1670:20160404:163920.515 7f3d5e7ed000-7f3d5e9ec000 ---p 00100000 fe:00 5385                       /lib/x86_64-linux-gnu/libm-2.19.so
  1670:20160404:163920.515 7f3d5e9ec000-7f3d5e9ed000 r--p 000ff000 fe:00 5385                       /lib/x86_64-linux-gnu/libm-2.19.so
  1670:20160404:163920.515 7f3d5e9ed000-7f3d5e9ee000 rw-p 00100000 fe:00 5385                       /lib/x86_64-linux-gnu/libm-2.19.so
  1670:20160404:163920.515 7f3d5e9ee000-7f3d5ea5d000 r-xp 00000000 fe:00 276229                     /usr/lib/x86_64-linux-gnu/libcurl.so.4.3.0
  1670:20160404:163920.515 7f3d5ea5d000-7f3d5ec5d000 ---p 0006f000 fe:00 276229                     /usr/lib/x86_64-linux-gnu/libcurl.so.4.3.0
  1670:20160404:163920.515 7f3d5ec5d000-7f3d5ec61000 r--p 0006f000 fe:00 276229                     /usr/lib/x86_64-linux-gnu/libcurl.so.4.3.0
  1670:20160404:163920.515 7f3d5ec61000-7f3d5ec62000 rw-p 00073000 fe:00 276229                     /usr/lib/x86_64-linux-gnu/libcurl.so.4.3.0
  1670:20160404:163920.515 7f3d5ec62000-7f3d5ec70000 r-xp 00000000 fe:00 267764                     /usr/lib/x86_64-linux-gnu/liblber-2.4.so.2.10.3
  1670:20160404:163920.515 7f3d5ec70000-7f3d5ee6f000 ---p 0000e000 fe:00 267764                     /usr/lib/x86_64-linux-gnu/liblber-2.4.so.2.10.3
  1670:20160404:163920.515 7f3d5ee6f000-7f3d5ee70000 r--p 0000d000 fe:00 267764                     /usr/lib/x86_64-linux-gnu/liblber-2.4.so.2.10.3
  1670:20160404:163920.515 7f3d5ee70000-7f3d5ee71000 rw-p 0000e000 fe:00 267764                     /usr/lib/x86_64-linux-gnu/liblber-2.4.so.2.10.3
  1670:20160404:163920.515 7f3d5ee71000-7f3d5eebe000 r-xp 00000000 fe:00 267763                     /usr/lib/x86_64-linux-gnu/libldap_r-2.4.so.2.10.3
  1670:20160404:163920.515 7f3d5eebe000-7f3d5f0be000 ---p 0004d000 fe:00 267763                     /usr/lib/x86_64-linux-gnu/libldap_r-2.4.so.2.10.3
  1670:20160404:163920.515 7f3d5f0be000-7f3d5f0c0000 r--p 0004d000 fe:00 267763                     /usr/lib/x86_64-linux-gnu/libldap_r-2.4.so.2.10.3
  1670:20160404:163920.515 7f3d5f0c0000-7f3d5f0c1000 rw-p 0004f000 fe:00 267763                     /usr/lib/x86_64-linux-gnu/libldap_r-2.4.so.2.10.3
  1670:20160404:163920.515 7f3d5f0c1000-7f3d5f0c3000 rw-p 00000000 00:00 0
  1670:20160404:163920.515 7f3d5f0c3000-7f3d5f28f000 r-xp 00000000 fe:00 258575                     /usr/lib/x86_64-linux-gnu/libcrypto.so.1.0.0
  1670:20160404:163920.515 7f3d5f28f000-7f3d5f48f000 ---p 001cc000 fe:00 258575                     /usr/lib/x86_64-linux-gnu/libcrypto.so.1.0.0
  1670:20160404:163920.515 7f3d5f48f000-7f3d5f4ac000 r--p 001cc000 fe:00 258575                     /usr/lib/x86_64-linux-gnu/libcrypto.so.1.0.0
  1670:20160404:163920.515 7f3d5f4ac000-7f3d5f4bc000 rw-p 001e9000 fe:00 258575                     /usr/lib/x86_64-linux-gnu/libcrypto.so.1.0.0
  1670:20160404:163920.515 7f3d5f4bc000-7f3d5f4bf000 rw-p 00000000 00:00 0
  1670:20160404:163920.515 7f3d5f4bf000-7f3d5f515000 r-xp 00000000 fe:00 259203                     /usr/lib/x86_64-linux-gnu/libssl.so.1.0.0
  1670:20160404:163920.515 7f3d5f515000-7f3d5f715000 ---p 00056000 fe:00 259203                     /usr/lib/x86_64-linux-gnu/libssl.so.1.0.0
  1670:20160404:163920.515 7f3d5f715000-7f3d5f718000 r--p 00056000 fe:00 259203                     /usr/lib/x86_64-linux-gnu/libssl.so.1.0.0
  1670:20160404:163920.515 7f3d5f718000-7f3d5f71e000 rw-p 00059000 fe:00 259203                     /usr/lib/x86_64-linux-gnu/libssl.so.1.0.0
  1670:20160404:163920.515 7f3d5f71e000-7f3d5f71f000 rw-p 00000000 00:00 0
  1670:20160404:163920.515 7f3d5f71f000-7f3d5f724000 r-xp 00000000 fe:00 276310                     /usr/lib/libOpenIPMIposix.so.0.0.1
  1670:20160404:163920.515 7f3d5f724000-7f3d5f923000 ---p 00005000 fe:00 276310                     /usr/lib/libOpenIPMIposix.so.0.0.1
  1670:20160404:163920.515 7f3d5f923000-7f3d5f924000 rw-p 00004000 fe:00 276310                     /usr/lib/libOpenIPMIposix.so.0.0.1
  1670:20160404:163920.515 7f3d5f924000-7f3d5fa0a000 r-xp 00000000 fe:00 276315                     /usr/lib/libOpenIPMI.so.0.0.5
  1670:20160404:163920.515 7f3d5fa0a000-7f3d5fc0a000 ---p 000e6000 fe:00 276315                     /usr/lib/libOpenIPMI.so.0.0.5
  1670:20160404:163920.515 7f3d5fc0a000-7f3d5fc26000 rw-p 000e6000 fe:00 276315                     /usr/lib/libOpenIPMI.so.0.0.5
  1670:20160404:163920.515 7f3d5fc26000-7f3d5fc2b000 rw-p 00000000 00:00 0
  1670:20160404:163920.515 7f3d5fc2b000-7f3d5fc52000 r-xp 00000000 fe:00 259236                     /usr/lib/x86_64-linux-gnu/libssh2.so.1.0.1
  1670:20160404:163920.515 7f3d5fc52000-7f3d5fe52000 ---p 00027000 fe:00 259236                     /usr/lib/x86_64-linux-gnu/libssh2.so.1.0.1
 1670:20160404:163920.515 7f3d5fc52000-7f3d5fe52000 ---p 00027000 fe:00 259236                     /usr/lib/x86_64-linux-gnu/libssh2.so.1.0.1
  1670:20160404:163920.515 7f3d5fe52000-7f3d5fe53000 r--p 00027000 fe:00 259236                     /usr/lib/x86_64-linux-gnu/libssh2.so.1.0.1
  1670:20160404:163920.515 7f3d5fe53000-7f3d5fe54000 rw-p 00028000 fe:00 259236                     /usr/lib/x86_64-linux-gnu/libssh2.so.1.0.1
  1670:20160404:163920.515 7f3d5fe54000-7f3d5fefe000 r-xp 00000000 fe:00 276299                     /usr/lib/x86_64-linux-gnu/libnetsnmp.so.30.0.2
  1670:20160404:163920.515 7f3d5fefe000-7f3d600fd000 ---p 000aa000 fe:00 276299                     /usr/lib/x86_64-linux-gnu/libnetsnmp.so.30.0.2
  1670:20160404:163920.515 7f3d600fd000-7f3d60100000 r--p 000a9000 fe:00 276299                     /usr/lib/x86_64-linux-gnu/libnetsnmp.so.30.0.2
  1670:20160404:163920.515 7f3d60100000-7f3d60102000 rw-p 000ac000 fe:00 276299                     /usr/lib/x86_64-linux-gnu/libnetsnmp.so.30.0.2
  1670:20160404:163920.515 7f3d60102000-7f3d60138000 rw-p 00000000 00:00 0
  1670:20160404:163920.515 7f3d60138000-7f3d60197000 r-xp 00000000 fe:00 276273                     /usr/lib/x86_64-linux-gnu/libodbc.so.2.0.0
  1670:20160404:163920.515 7f3d60197000-7f3d60397000 ---p 0005f000 fe:00 276273                     /usr/lib/x86_64-linux-gnu/libodbc.so.2.0.0
  1670:20160404:163920.515 7f3d60397000-7f3d60398000 r--p 0005f000 fe:00 276273                     /usr/lib/x86_64-linux-gnu/libodbc.so.2.0.0
  1670:20160404:163920.515 7f3d60398000-7f3d6039f000 rw-p 00060000 fe:00 276273                     /usr/lib/x86_64-linux-gnu/libodbc.so.2.0.0
  1670:20160404:163920.515 7f3d6039f000-7f3d603a0000 rw-p 00000000 00:00 0
  1670:20160404:163920.515 7f3d603a0000-7f3d604fc000 r-xp 00000000 fe:00 267849                     /usr/lib/x86_64-linux-gnu/libxml2.so.2.9.1
  1670:20160404:163920.515 7f3d604fc000-7f3d606fb000 ---p 0015c000 fe:00 267849                     /usr/lib/x86_64-linux-gnu/libxml2.so.2.9.1
  1670:20160404:163920.515 7f3d606fb000-7f3d60704000 r--p 0015b000 fe:00 267849                     /usr/lib/x86_64-linux-gnu/libxml2.so.2.9.1
  1670:20160404:163920.515 7f3d60704000-7f3d60706000 rw-p 00164000 fe:00 267849                     /usr/lib/x86_64-linux-gnu/libxml2.so.2.9.1
  1670:20160404:163920.515 7f3d60706000-7f3d60707000 rw-p 00000000 00:00 0
  1670:20160404:163920.515 7f3d60707000-7f3d60714000 r-xp 00000000 fe:00 276248                     /usr/lib/x86_64-linux-gnu/libiksemel.so.3.1.1
  1670:20160404:163920.515 7f3d60714000-7f3d60914000 ---p 0000d000 fe:00 276248                     /usr/lib/x86_64-linux-gnu/libiksemel.so.3.1.1
  1670:20160404:163920.515 7f3d60914000-7f3d60915000 r--p 0000d000 fe:00 276248                     /usr/lib/x86_64-linux-gnu/libiksemel.so.3.1.1
  1670:20160404:163920.515 7f3d60915000-7f3d60916000 rw-p 0000e000 fe:00 276248                     /usr/lib/x86_64-linux-gnu/libiksemel.so.3.1.1
  1670:20160404:163920.515 7f3d60916000-7f3d60c01000 r-xp 00000000 fe:00 276068                     /usr/lib/x86_64-linux-gnu/libmysqlclient.so.18.1.0
  1670:20160404:163920.515 7f3d60c01000-7f3d60e01000 ---p 002eb000 fe:00 276068                     /usr/lib/x86_64-linux-gnu/libmysqlclient.so.18.1.0
  1670:20160404:163920.515 7f3d60e01000-7f3d60e08000 r--p 002eb000 fe:00 276068                     /usr/lib/x86_64-linux-gnu/libmysqlclient.so.18.1.0
  1670:20160404:163920.515 7f3d60e08000-7f3d60ed6000 rw-p 002f2000 fe:00 276068                     /usr/lib/x86_64-linux-gnu/libmysqlclient.so.18.1.0
  1670:20160404:163920.515 7f3d60ed6000-7f3d60edb000 rw-p 00000000 00:00 0
  1670:20160404:163920.515 7f3d60edb000-7f3d60efb000 r-xp 00000000 fe:00 5378                       /lib/x86_64-linux-gnu/ld-2.19.so
  1670:20160404:163920.515 7f3d60fa6000-7f3d610da000 rw-s 00000000 00:04 688133                     /SYSV730000dd (deleted)
  1670:20160404:163920.515 7f3d610da000-7f3d610f0000 rw-p 00000000 00:00 0
  1670:20160404:163920.516 7f3d610f5000-7f3d610f6000 rw-p 00000000 00:00 0
  1670:20160404:163920.516 7f3d610f6000-7f3d610f8000 rw-s 00000000 00:04 720902                     /SYSV530000dd (deleted)
  1670:20160404:163920.516 7f3d610f8000-7f3d610f9000 rw-p 00000000 00:00 0
  1670:20160404:163920.516 7f3d610f9000-7f3d610fb000 rw-p 00000000 00:00 0
  1670:20160404:163920.516 7f3d610fb000-7f3d610fc000 r--p 00020000 fe:00 5378                       /lib/x86_64-linux-gnu/ld-2.19.so
  1670:20160404:163920.516 7f3d610fc000-7f3d610fd000 rw-p 00021000 fe:00 5378                       /lib/x86_64-linux-gnu/ld-2.19.so
  1670:20160404:163920.516 7f3d610fd000-7f3d610fe000 rw-p 00000000 00:00 0
  1670:20160404:163920.516 7ffe93afa000-7ffe93b1b000 rw-p 00000000 00:00 0                          [stack]
  1670:20160404:163920.516 7ffe93b69000-7ffe93b6b000 r-xp 00000000 00:00 0                          [vdso]
  1670:20160404:163920.516 7ffe93b6b000-7ffe93b6d000 r--p 00000000 00:00 0                          [vvar]
  1670:20160404:163920.516 ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0                  [vsyscall]
  1670:20160404:163920.516 ================================
  1670:20160404:163920.516 Please consider attaching a disassembly listing to your bug report.
  1670:20160404:163920.516 This listing can be produced with, e.g., objdump -DSswx zabbix_server.
  1670:20160404:163920.516 ================================
  1669:20160404:163920.518 One child process died (PID:1670,exitcode/signal:1). Exiting ...
  1669:20160404:163922.518 syncing history data...
  1669:20160404:163922.522 syncing history data done
  1669:20160404:163922.522 syncing trends data...
  1669:20160404:163922.589 syncing trends data done
  1669:20160404:163922.589 Zabbix Server stopped. Zabbix 3.0.1 (revision 58734).


 Comments   
Comment by Glebs Ivanovskis (Inactive) [ 2016 Apr 04 ]

Could you please attach a gzipped result of the following command?

objdump -DSswx zabbix_server > zabbix_server.objdump
Comment by Aleksandrs Saveljevs [ 2016 Apr 04 ]

Could you please attach the disassembly listing (objdump -DSswx zabbix_server) as suggested by the log? Does your vPoller patch affect dbconfig.c file?

Comment by Maxim Semenov [ 2016 Apr 04 ]

It writes:

# objdump -DSswx zabbix_server
objdump: 'zabbix_server': No such file

I don't know about "Does your vPoller patch affect dbconfig.c file?"
http://vpoller.readthedocs.org/en/latest/vpoller-zabbix.html
I have not dig so deep in this component. Looks like it should not.

Comment by Aleksandrs Saveljevs [ 2016 Apr 04 ]

You should point objdump to where zabbix_server binary is located.

Comment by Maxim Semenov [ 2016 Apr 04 ]

object dump, I was asked for

Comment by Maxim Semenov [ 2016 Apr 04 ]

I'm not sure is it ok, but.
I have attached some file. I'm noob in linux. Sorry for my misunderstandings.

Comment by Glebs Ivanovskis (Inactive) [ 2016 Apr 04 ]

No problem Thank you very much for the objdump!

How did you install vPoller?

Comment by Maxim Semenov [ 2016 Apr 04 ]

with this instruction.
http://vpoller.readthedocs.org/en/latest/vpoller-zabbix.html#native-vpoller-support-for-zabbix

In short - downloaded zabbix sources, vpoller sources
then:

$ tar zxvf zabbix-2.2.5.tar.gz
$ cd zabbix-2.2.5
$ ./configure
$ cp -a py-vpoller/extra/zabbix/vpoller-module zabbix-2.2.5/src/modules
$ cd zabbix-2.2.5/src/modules/vpoller-module
$ make

$ sudo cp zabbix-2.2.5/src/modules/vpoller-module/vpoller.so /usr/local/lib/zabbix

then add LoadModule=vpoller.so to zabbix config

Also I have Zabbix to telegram script (it's a simple python script to send messages to telegram messandger with graph attached. I don't think, It coud affect something.
But I could look for triggers at crush time.

Comment by Glebs Ivanovskis (Inactive) [ 2016 Apr 04 ]

Maxim, please correct me if I'm wrong. You compiled vPoller module from sources inside Zabbix 2.2 source tree? And just to be 100% sure, is this module meant to be used with Zabbix server/proxy?

If the answer is "yes" to both questions, this issue looks to me like a one more ZBXNEXT-444 regression. Caused by ghost-like items with no value, no error message, no status, etc. which breaks processing logic completely.

Comment by Maxim Semenov [ 2016 Apr 05 ]

Oh! Sorry for not being perfectly clear.
Of course I compiled this module with 3.0.1 sources.
And yes. This module is meant to use with zbx server-proxy.

This is the replace for zbx vmware poller, with queue and more stable (looks - like). Plus it allows to import hosts as hosts, not lld things.

Comment by Glebs Ivanovskis (Inactive) [ 2016 Apr 05 ]

Thank you for clarification!

So far it is clear that we crash inside DCstrpool_release() and crash is caused by its second parameter *curr == NULL:

static int	DCstrpool_replace(int found, const char **curr, const char *new)
{
	if (1 == found)
	{
		if (0 == strcmp(*curr, new))
			return FAIL;
		...
	}
	...
}

Now it's time to check ~60 places where DCstrpool_replace() is called from.

Comment by Glebs Ivanovskis (Inactive) [ 2016 Apr 05 ]

Maxim, what sort of configuration do you have? TLS, IPMI, SNMP, database monitoring, SSH, telnet checks, Java gateway?

Comment by Aleksandrs Saveljevs [ 2016 Apr 05 ]

Do you remember changing configuration in the frontend just before the crash (e.g., changing an item's name or linking a template to one of the hosts)?

Comment by Maxim Semenov [ 2016 Apr 05 ]

Unfortunatly nothing strange before crash.
This is clean install, and there is only vmware monitoring via vPoller + zabbix server (standart templates: Template App MySQL, Template App Zabbix Server, Template OS Linux) + 1 test web scenario for host
I looked through audit log and the only thing between 16 and 17 is

 
2016-04-04 16:45:43	semenov	172.20.2.78	Web scenario	Disabled	0		Web scenario [Test jira web server] [1] Host [WEB -- jira.*.ru] disabled

but this action wa after crash

I found out that my server does not have access to 80 port outside, so there was a lot of errors in log.

 
1688:20160404:163744.060 cannot process step "test main page" of web scenario "Test jira web server" on host "WEB -- jira.*.ru": Timeout was reached: Connection timed out after 15001 milliseconds
  1688:20160404:163859.077 cannot process step "test main page" of web scenario "Test jira web server" on host "WEB -- jira.*.ru": Timeout was reached: Connection timed out after 15001 milliseconds
Comment by Glebs Ivanovskis (Inactive) [ 2016 Apr 05 ]

These two pieces cause crash:

		if (NULL == context)
			gmacro->context = NULL;
		else
			DCstrpool_replace(found, &gmacro->context, context);
		if (NULL == context)
			hmacro->context = NULL;
		else
			DCstrpool_replace(found, &hmacro->context, context);

Suppose that on the first pass we take true branch and assign NULL, but on the second pass we take false branch and pass pointer to NULL to DCstrpool_replace().
Introduced in ZBXNEXT-2683.

How to reproduce:

  • create macro without context;
  • wait for server configuration update;
  • add context to the same macro;
  • wait for server configuration update.
Comment by Glebs Ivanovskis (Inactive) [ 2016 Apr 05 ]

Fix for version 3.0 is available in development branch svn://svn.zabbix.com/branches/dev/ZBX-10614 revision 59298. Additionally fixed a memory leak in r59325. And polished conditions in r59338.

Comment by Sandis Neilands (Inactive) [ 2016 Apr 07 ]

Successfully tested as per provided scenarion. No more crashes with global, host, template context macros. Minor style improvements in r59349.

glebs.ivanovskis Style is an important thing! Seems like we have a strong tradition of giving a name found to the last parameter of DCfind_id(). And it's better to keep everything in one place. Do you like my suggestions in r59868?

Comment by Sandis Neilands (Inactive) [ 2016 May 03 ]

sandis.neilands Your changes are fine, CLOSED.

Comment by Glebs Ivanovskis (Inactive) [ 2016 May 03 ]

Fixed in pre-3.0.3rc1 r59873, pre-3.1.0 (trunk) r59874.

Comment by Oleksii Zagorskyi [ 2016 May 05 ]

why it doesn't have "crash" label added ?

Generated at Fri Mar 29 13:23:23 EET 2024 using Jira 9.12.4#9120004-sha1:625303b708afdb767e17cb2838290c41888e9ff0.