[ZBX-12725] Tasks without data records cause server crash Created: 2017 Sep 14  Updated: 2024 Apr 10  Resolved: 2017 Oct 10

Status: Closed
Project: ZABBIX BUGS AND ISSUES
Component/s: Server (S)
Affects Version/s: 3.4.1
Fix Version/s: 3.4.3rc1, 4.0.0alpha1, 4.0 (plan)

Type: Problem report Priority: Critical
Reporter: Alain Ganuchaud Assignee: Andris Zeila
Resolution: Fixed Votes: 1
Labels: crash, trapper
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

centos 7.3.1611
Mysql version 5.7.19
Zabbix 3.4.1 (packages)

Attachments: Zip Archive dump.zip     Text File query.txt     File zabbix_server.log.1     File zabbix_server.log.2     File zabbix_server.log.3    
Issue Links:
Duplicate
duplicates ZBX-12618 item became not supported: Item prep... Closed
is duplicated by ZBX-12808 Zabbix crash on trapper process Closed
Team: Team A
Team: Team A
Sprint: Sprint 17, Sprint 18
Story Points: 1

 Description   

Preprocessing is either having huge queue and get stuck, either crashing when trying to increase StartPreprocessors.

I attach Debug Informations of crash, could be linked with ZBX-12718

 Comments   
Comment by Vladislavs Sokurenko [ 2017 Sep 14 ]

Backtracke for easier searching

5420:20170914:124031.715 item "SWDPANTC3BI-087:ifInOctets[GigabitEthernet1/0/49]" became not supported: Item preprocessing step #1 failed: (null)
 15726:20170914:124231.778 Got signal [signal:11(SIGSEGV),reason:1,refaddr:(nil)]. Crashing ...
 15726:20170914:124231.778 ====== Fatal information: ======
 15726:20170914:124231.778 Program counter: 0x7fdd7a343d47
 15726:20170914:124231.778 === Registers: ===
 15726:20170914:124231.778 r8      =     7fdd7a6c5060 =      140589218418784 =      140589218418784
 15726:20170914:124231.778 r9      =               28 =                   40 =                   40
 15726:20170914:124231.778 r10     =                0 =                    0 =                    0
 15726:20170914:124231.778 r11     =                0 =                    0 =                    0
 15726:20170914:124231.778 r12     =              258 =                  600 =                  600
 15726:20170914:124231.778 r13     =                0 =                    0 =                    0
 15726:20170914:124231.778 r14     =                0 =                    0 =                    0
 15726:20170914:124231.778 r15     =                0 =                    0 =                    0
 15726:20170914:124231.778 rdi     =                0 =                    0 =                    0
 15726:20170914:124231.778 rsi     =                0 =                    0 =                    0
 15726:20170914:124231.778 rbp     =                0 =                    0 =                    0
 15726:20170914:124231.778 rbx     =                0 =                    0 =                    0
 15726:20170914:124231.778 rdx     =                a =                   10 =                   10
 15726:20170914:124231.778 rax     =     7fdd7a6c2260 =      140589218407008 =      140589218407008
 15726:20170914:124231.778 rcx     =                0 =                    0 =                    0
 15726:20170914:124231.778 rsp     =     7ffd821822c0 =      140726786073280 =      140726786073280
 15726:20170914:124231.778 rip     =     7fdd7a343d47 =      140589214743879 =      140589214743879
 15726:20170914:124231.778 efl     =            10283 =                66179 =                66179
 15726:20170914:124231.778 csgsfs  =               33 =                   51 =                   51
 15726:20170914:124231.778 err     =                4 =                    4 =                    4
 15726:20170914:124231.778 trapno  =                e =                   14 =                   14
 15726:20170914:124231.778 oldmask =                0 =                    0 =                    0
 15726:20170914:124231.778 cr2     =                0 =                    0 =                    0
 15726:20170914:124231.778 === Backtrace: ===
 15726:20170914:124231.779 16: /usr/sbin/zabbix_server: trapper #19 [processing data](zbx_log_fatal_info+0x13c) [0x7fdd7ded23c4]
 15726:20170914:124231.779 15: /usr/sbin/zabbix_server: trapper #19 [processing data](+0xe97d9) [0x7fdd7ded27d9]
 15726:20170914:124231.779 14: /lib64/libc.so.6(+0x35250) [0x7fdd7a33f250]
 15726:20170914:124231.779 13: /lib64/libc.so.6(+0x39d47) [0x7fdd7a343d47]
 15726:20170914:124231.779 12: /lib64/libc.so.6(atoi+0x10) [0x7fdd7a340750]
 15726:20170914:124231.779 11: /usr/sbin/zabbix_server: trapper #19 [processing data](zbx_tm_get_remote_tasks+0x247) [0x7fdd7de6f12b]
 15726:20170914:124231.779 10: /usr/sbin/zabbix_server: trapper #19 [processing data](zbx_send_proxy_data_respose+0x4b) [0x7fdd7de55413]
 15726:20170914:124231.779 9: /usr/sbin/zabbix_server: trapper #19 [processing data](zbx_recv_proxy_data+0x1b2) [0x7fdd7de556ee]
 15726:20170914:124231.779 8: /usr/sbin/zabbix_server: trapper #19 [processing data](+0x6a3cf) [0x7fdd7de533cf]
 15726:20170914:124231.779 7: /usr/sbin/zabbix_server: trapper #19 [processing data](+0x6ab37) [0x7fdd7de53b37]
 15726:20170914:124231.779 6: /usr/sbin/zabbix_server: trapper #19 [processing data](trapper_thread+0x21f) [0x7fdd7de53d58]
 15726:20170914:124231.780 5: /usr/sbin/zabbix_server: trapper #19 [processing data](zbx_thread_start+0x37) [0x7fdd7dedf7bf]
 15726:20170914:124231.780 4: /usr/sbin/zabbix_server: trapper #19 [processing data](MAIN_ZABBIX_ENTRY+0x8c8) [0x7fdd7de29041]
 15726:20170914:124231.780 3: /usr/sbin/zabbix_server: trapper #19 [processing data](daemon_start+0x325) [0x7fdd7ded1b0a]
 15726:20170914:124231.780 2: /usr/sbin/zabbix_server: trapper #19 [processing data](main+0x317) [0x7fdd7de28777]
 15726:20170914:124231.780 1: /lib64/libc.so.6(__libc_start_main+0xf5) [0x7fdd7a32bb35]
 15726:20170914:124231.780 0: /usr/sbin/zabbix_server: trapper #19 [processing data](+0x33a89) [0x7fdd7de1ca89]
 15726:20170914:124231.780 === Memory map: ===
 15726:20170914:124231.781 7fdcb4fd1000-7fdcf4fd1000 rw-s 00000000 00:04 1409031                    /SYSV00000000 (deleted)
 15726:20170914:124231.781 7fdcf4fd1000-7fdcfe96b000 rw-s 00000000 00:04 1343493                    /SYSV00000000 (deleted)
 15726:20170914:124231.781 7fdcfe96b000-7fdd34fd2000 rw-s 00000000 00:04 1310724                    /SYSV00000000 (deleted)
 15726:20170914:124231.781 7fdd34fd2000-7fdd44fd2000 rw-s 00000000 00:04 1277955                    /SYSV00000000 (deleted)
 15726:20170914:124231.781 7fdd44fd2000-7fdd54fd2000 rw-s 00000000 00:04 1245186                    /SYSV00000000 (deleted)
 15726:20170914:124231.781 7fdd54fd2000-7fdd74fd2000 rw-s 00000000 00:04 1212417                    /SYSV00000000 (deleted)
 15726:20170914:124231.781 7fdd74fd2000-7fdd74fde000 r-xp 00000000 08:03 67240557                   /usr/lib64/libnss_files-2.17.so
 15726:20170914:124231.781 7fdd74fde000-7fdd751dd000 ---p 0000c000 08:03 67240557                   /usr/lib64/libnss_files-2.17.so
 15726:20170914:124231.781 7fdd751dd000-7fdd751de000 r--p 0000b000 08:03 67240557                   /usr/lib64/libnss_files-2.17.so
 15726:20170914:124231.781 7fdd751de000-7fdd751df000 rw-p 0000c000 08:03 67240557                   /usr/lib64/libnss_files-2.17.so
 15726:20170914:124231.781 7fdd751df000-7fdd751e5000 rw-p 00000000 00:00 0 
 15726:20170914:124231.781 7fdd751e5000-7fdd751e7000 r-xp 00000000 08:03 67163955                   /usr/lib64/libfreebl3.so
 15726:20170914:124231.781 7fdd751e7000-7fdd753e6000 ---p 00002000 08:03 67163955                   /usr/lib64/libfreebl3.so
 15726:20170914:124231.781 7fdd753e6000-7fdd753e7000 r--p 00001000 08:03 67163955                   /usr/lib64/libfreebl3.so
 15726:20170914:124231.781 7fdd753e7000-7fdd753e8000 rw-p 00002000 08:03 67163955                   /usr/lib64/libfreebl3.so
 15726:20170914:124231.781 7fdd753e8000-7fdd7540c000 r-xp 00000000 08:03 67164519                   /usr/lib64/libselinux.so.1
 15726:20170914:124231.781 7fdd7540c000-7fdd7560b000 ---p 00024000 08:03 67164519                   /usr/lib64/libselinux.so.1
 15726:20170914:124231.781 7fdd7560b000-7fdd7560c000 r--p 00023000 08:03 67164519                   /usr/lib64/libselinux.so.1
 15726:20170914:124231.781 7fdd7560c000-7fdd7560d000 rw-p 00024000 08:03 67164519                   /usr/lib64/libselinux.so.1
 15726:20170914:124231.781 7fdd7560d000-7fdd7560f000 rw-p 00000000 00:00 0 
 15726:20170914:124231.782 7fdd7560f000-7fdd75616000 r-xp 00000000 08:03 67164357                   /usr/lib64/libffi.so.6.0.1
 15726:20170914:124231.782 7fdd75616000-7fdd75815000 ---p 00007000 08:03 67164357                   /usr/lib64/libffi.so.6.0.1
 15726:20170914:124231.782 7fdd75815000-7fdd75816000 r--p 00006000 08:03 67164357                   /usr/lib64/libffi.so.6.0.1
 15726:20170914:124231.782 7fdd75816000-7fdd75817000 rw-p 00007000 08:03 67164357                   /usr/lib64/libffi.so.6.0.1
 15726:20170914:124231.782 7fdd75817000-7fdd7581e000 r-xp 00000000 08:03 67313155                   /usr/lib64/librt-2.17.so
 15726:20170914:124231.782 7fdd7581e000-7fdd75a1d000 ---p 00007000 08:03 67313155                   /usr/lib64/librt-2.17.so
 15726:20170914:124231.782 7fdd75a1d000-7fdd75a1e000 r--p 00006000 08:03 67313155                   /usr/lib64/librt-2.17.so
 15726:20170914:124231.782 7fdd75a1e000-7fdd75a1f000 rw-p 00007000 08:03 67313155                   /usr/lib64/librt-2.17.so
 15726:20170914:124231.782 7fdd75a1f000-7fdd75a27000 r-xp 00000000 08:03 67240058                   /usr/lib64/libcrypt-2.17.so
 15726:20170914:124231.782 7fdd75a27000-7fdd75c26000 ---p 00008000 08:03 67240058                   /usr/lib64/libcrypt-2.17.so
 15726:20170914:124231.782 7fdd75c26000-7fdd75c27000 r--p 00007000 08:03 67240058                   /usr/lib64/libcrypt-2.17.so
 15726:20170914:124231.782 7fdd75c27000-7fdd75c28000 rw-p 00008000 08:03 67240058                   /usr/lib64/libcrypt-2.17.so
 15726:20170914:124231.782 7fdd75c28000-7fdd75c56000 rw-p 00000000 00:00 0 
 15726:20170914:124231.782 7fdd75c56000-7fdd75c59000 r-xp 00000000 08:03 67165978                   /usr/lib64/libkeyutils.so.1.5
 15726:20170914:124231.782 7fdd75c59000-7fdd75e58000 ---p 00003000 08:03 67165978                   /usr/lib64/libkeyutils.so.1.5
 15726:20170914:124231.782 7fdd75e58000-7fdd75e59000 r--p 00002000 08:03 67165978                   /usr/lib64/libkeyutils.so.1.5
 15726:20170914:124231.782 7fdd75e59000-7fdd75e5a000 rw-p 00003000 08:03 67165978                   /usr/lib64/libkeyutils.so.1.5
 15726:20170914:124231.782 7fdd75e5a000-7fdd75e67000 r-xp 00000000 08:03 67169475                   /usr/lib64/libkrb5support.so.0.1
 15726:20170914:124231.782 7fdd75e67000-7fdd76067000 ---p 0000d000 08:03 67169475                   /usr/lib64/libkrb5support.so.0.1
 15726:20170914:124231.782 7fdd76067000-7fdd76068000 r--p 0000d000 08:03 67169475                   /usr/lib64/libkrb5support.so.0.1
 15726:20170914:124231.782 7fdd76068000-7fdd76069000 rw-p 0000e000 08:03 67169475                   /usr/lib64/libkrb5support.so.0.1
 15726:20170914:124231.782 7fdd76069000-7fdd760df000 r-xp 00000000 08:03 67552203                   /usr/lib64/libgmp.so.10.2.0
 15726:20170914:124231.782 7fdd760df000-7fdd762de000 ---p 00076000 08:03 67552203                   /usr/lib64/libgmp.so.10.2.0
 15726:20170914:124231.782 7fdd762de000-7fdd762df000 r--p 00075000 08:03 67552203                   /usr/lib64/libgmp.so.10.2.0
 15726:20170914:124231.782 7fdd762df000-7fdd762e0000 rw-p 00076000 08:03 67552203                   /usr/lib64/libgmp.so.10.2.0
 15726:20170914:124231.782 7fdd762e0000-7fdd76306000 r-xp 00000000 08:03 67199309                   /usr/lib64/libhogweed.so.2.5
 15726:20170914:124231.782 7fdd76306000-7fdd76505000 ---p 00026000 08:03 67199309                   /usr/lib64/libhogweed.so.2.5
 15726:20170914:124231.782 7fdd76505000-7fdd76506000 r--p 00025000 08:03 67199309                   /usr/lib64/libhogweed.so.2.5
 15726:20170914:124231.782 7fdd76506000-7fdd76507000 rw-p 00026000 08:03 67199309                   /usr/lib64/libhogweed.so.2.5
 15726:20170914:124231.782 7fdd76507000-7fdd76536000 r-xp 00000000 08:03 67199310                   /usr/lib64/libnettle.so.4.7
 15726:20170914:124231.782 7fdd76536000-7fdd76736000 ---p 0002f000 08:03 67199310                   /usr/lib64/libnettle.so.4.7
 15726:20170914:124231.782 7fdd76736000-7fdd76737000 r--p 0002f000 08:03 67199310                   /usr/lib64/libnettle.so.4.7
 15726:20170914:124231.782 7fdd76737000-7fdd76738000 rw-p 00030000 08:03 67199310                   /usr/lib64/libnettle.so.4.7
 15726:20170914:124231.782 7fdd76738000-7fdd7674a000 r-xp 00000000 08:03 67199290                   /usr/lib64/libtasn1.so.6.2.3
 15726:20170914:124231.782 7fdd7674a000-7fdd7694a000 ---p 00012000 08:03 67199290                   /usr/lib64/libtasn1.so.6.2.3
 15726:20170914:124231.782 7fdd7694a000-7fdd7694b000 r--p 00012000 08:03 67199290                   /usr/lib64/libtasn1.so.6.2.3
 15726:20170914:124231.783 7fdd7694b000-7fdd7694c000 rw-p 00013000 08:03 67199290                   /usr/lib64/libtasn1.so.6.2.3
 15726:20170914:124231.783 7fdd7694c000-7fdd769ba000 r-xp 00000000 08:03 67239896                   /usr/lib64/libtspi.so.1.2.0
 15726:20170914:124231.783 7fdd769ba000-7fdd76bb9000 ---p 0006e000 08:03 67239896                   /usr/lib64/libtspi.so.1.2.0
 15726:20170914:124231.783 7fdd76bb9000-7fdd76bba000 r--p 0006d000 08:03 67239896                   /usr/lib64/libtspi.so.1.2.0
 15726:20170914:124231.783 7fdd76bba000-7fdd76bbc000 rw-p 0006e000 08:03 67239896                   /usr/lib64/libtspi.so.1.2.0
 15726:20170914:124231.783 7fdd76bbc000-7fdd76bbd000 rw-p 00000000 00:00 0 
 15726:20170914:124231.783 7fdd76bbd000-7fdd76bf9000 r-xp 00000000 08:03 67165728                   /usr/lib64/libp11-kit.so.0.0.0
 15726:20170914:124231.783 7fdd76bf9000-7fdd76df8000 ---p 0003c000 08:03 67165728                   /usr/lib64/libp11-kit.so.0.0.0
 15726:20170914:124231.783 7fdd76df8000-7fdd76e01000 r--p 0003b000 08:03 67165728                   /usr/lib64/libp11-kit.so.0.0.0
 15726:20170914:124231.783 7fdd76e01000-7fdd76e03000 rw-p 00044000 08:03 67165728                   /usr/lib64/libp11-kit.so.0.0.0
 15726:20170914:124231.783 7fdd76e03000-7fdd76e35000 r-xp 00000000 08:03 67165679                   /usr/lib64/libidn.so.11.6.11
 15726:20170914:124231.783 7fdd76e35000-7fdd77034000 ---p 00032000 08:03 67165679                   /usr/lib64/libidn.so.11.6.11
 15726:20170914:124231.783 7fdd77034000-7fdd77035000 r--p 00031000 08:03 67165679                   /usr/lib64/libidn.so.11.6.11
 15726:20170914:124231.783 7fdd77035000-7fdd77036000 rw-p 00032000 08:03 67165679                   /usr/lib64/libidn.so.11.6.11
 15726:20170914:124231.783 7fdd77036000-7fdd77070000 r-xp 00000000 08:03 67164528                   /usr/lib64/libnspr4.so
 15726:20170914:124231.783 7fdd77070000-7fdd7726f000 ---p 0003a000 08:03 67164528                   /usr/lib64/libnspr4.so
 15726:20170914:124231.783 7fdd7726f000-7fdd77270000 r--p 00039000 08:03 67164528                   /usr/lib64/libnspr4.so
 15726:20170914:124231.783 7fdd77270000-7fdd77272000 rw-p 0003a000 08:03 67164528                   /usr/lib64/libnspr4.so
 15726:20170914:124231.783 7fdd77272000-7fdd77274000 rw-p 00000000 00:00 0 
 15726:20170914:124231.783 7fdd77274000-7fdd77278000 r-xp 00000000 08:03 67164529                   /usr/lib64/libplc4.so
 15726:20170914:124231.783 7fdd77278000-7fdd77477000 ---p 00004000 08:03 67164529                   /usr/lib64/libplc4.so
 15726:20170914:124231.783 7fdd77477000-7fdd77478000 r--p 00003000 08:03 67164529                   /usr/lib64/libplc4.so
 15726:20170914:124231.783 7fdd77478000-7fdd77479000 rw-p 00004000 08:03 67164529                   /usr/lib64/libplc4.so
 15726:20170914:124231.783 7fdd77479000-7fdd7747c000 r-xp 00000000 08:03 67164530                   /usr/lib64/libplds4.so
 15726:20170914:124231.783 7fdd7747c000-7fdd7767b000 ---p 00003000 08:03 67164530                   /usr/lib64/libplds4.so
 15726:20170914:124231.783 7fdd7767b000-7fdd7767c000 r--p 00002000 08:03 67164530                   /usr/lib64/libplds4.so
 15726:20170914:124231.783 7fdd7767c000-7fdd7767d000 rw-p 00003000 08:03 67164530                   /usr/lib64/libplds4.so
 15726:20170914:124231.783 7fdd7767d000-7fdd776a3000 r-xp 00000000 08:03 67164531                   /usr/lib64/libnssutil3.so
 15726:20170914:124231.783 7fdd776a3000-7fdd778a2000 ---p 00026000 08:03 67164531                   /usr/lib64/libnssutil3.so
 15726:20170914:124231.783 7fdd778a2000-7fdd778a9000 r--p 00025000 08:03 67164531                   /usr/lib64/libnssutil3.so
 15726:20170914:124231.783 7fdd778a9000-7fdd778aa000 rw-p 0002c000 08:03 67164531                   /usr/lib64/libnssutil3.so
 15726:20170914:124231.783 7fdd778aa000-7fdd779cb000 r-xp 00000000 08:03 67164401                   /usr/lib64/libnss3.so
 15726:20170914:124231.783 7fdd779cb000-7fdd77bcb000 ---p 00121000 08:03 67164401                   /usr/lib64/libnss3.so
 15726:20170914:124231.783 7fdd77bcb000-7fdd77bd0000 r--p 00121000 08:03 67164401                   /usr/lib64/libnss3.so
 15726:20170914:124231.783 7fdd77bd0000-7fdd77bd2000 rw-p 00126000 08:03 67164401                   /usr/lib64/libnss3.so
 15726:20170914:124231.783 7fdd77bd2000-7fdd77bd4000 rw-p 00000000 00:00 0 
 15726:20170914:124231.783 7fdd77bd4000-7fdd77bf8000 r-xp 00000000 08:03 67164605                   /usr/lib64/libsmime3.so
 15726:20170914:124231.783 7fdd77bf8000-7fdd77df7000 ---p 00024000 08:03 67164605                   /usr/lib64/libsmime3.so
 15726:20170914:124231.783 7fdd77df7000-7fdd77dfa000 r--p 00023000 08:03 67164605                   /usr/lib64/libsmime3.so
 15726:20170914:124231.783 7fdd77dfa000-7fdd77dfb000 rw-p 00026000 08:03 67164605                   /usr/lib64/libsmime3.so
 15726:20170914:124231.783 7fdd77dfb000-7fdd77e42000 r-xp 00000000 08:03 67164606                   /usr/lib64/libssl3.so
 15726:20170914:124231.783 7fdd77e42000-7fdd78041000 ---p 00047000 08:03 67164606                   /usr/lib64/libssl3.so
 15726:20170914:124231.783 7fdd78041000-7fdd78045000 r--p 00046000 08:03 67164606                   /usr/lib64/libssl3.so
 15726:20170914:124231.784 7fdd78045000-7fdd78046000 rw-p 0004a000 08:03 67164606                   /usr/lib64/libssl3.so
 15726:20170914:124231.784 7fdd78046000-7fdd78047000 rw-p 00000000 00:00 0 
 15726:20170914:124231.784 7fdd78047000-7fdd78063000 r-xp 00000000 08:03 67169501                   /usr/lib64/libsasl2.so.3.0.0
 15726:20170914:124231.784 7fdd78063000-7fdd78262000 ---p 0001c000 08:03 67169501                   /usr/lib64/libsasl2.so.3.0.0
 15726:20170914:124231.784 7fdd78262000-7fdd78263000 r--p 0001b000 08:03 67169501                   /usr/lib64/libsasl2.so.3.0.0
 15726:20170914:124231.784 7fdd78263000-7fdd78264000 rw-p 0001c000 08:03 67169501                   /usr/lib64/libsasl2.so.3.0.0
 15726:20170914:124231.784 7fdd78264000-7fdd78293000 r-xp 00000000 08:03 67164966                   /usr/lib64/libk5crypto.so.3.1
 15726:20170914:124231.784 7fdd78293000-7fdd78492000 ---p 0002f000 08:03 67164966                   /usr/lib64/libk5crypto.so.3.1
 15726:20170914:124231.784 7fdd78492000-7fdd78494000 r--p 0002e000 08:03 67164966                   /usr/lib64/libk5crypto.so.3.1
 15726:20170914:124231.784 7fdd78494000-7fdd78495000 rw-p 00030000 08:03 67164966                   /usr/lib64/libk5crypto.so.3.1
 15726:20170914:124231.784 7fdd78495000-7fdd78496000 rw-p 00000000 00:00 0 
 15726:20170914:124231.784 7fdd78496000-7fdd78499000 r-xp 00000000 08:03 67164524                   /usr/lib64/libcom_err.so.2.1
 15726:20170914:124231.784 7fdd78499000-7fdd78698000 ---p 00003000 08:03 67164524                   /usr/lib64/libcom_err.so.2.1
 15726:20170914:124231.784 7fdd78698000-7fdd78699000 r--p 00002000 08:03 67164524                   /usr/lib64/libcom_err.so.2.1
 15726:20170914:124231.784 7fdd78699000-7fdd7869a000 rw-p 00003000 08:03 67164524                   /usr/lib64/libcom_err.so.2.1
 15726:20170914:124231.784 7fdd7869a000-7fdd78770000 r-xp 00000000 08:03 67169473                   /usr/lib64/libkrb5.so.3.3
 15726:20170914:124231.784 7fdd78770000-7fdd78970000 ---p 000d6000 08:03 67169473                   /usr/lib64/libkrb5.so.3.3
 15726:20170914:124231.784 7fdd78970000-7fdd7897e000 r--p 000d6000 08:03 67169473                   /usr/lib64/libkrb5.so.3.3
 15726:20170914:124231.784 7fdd7897e000-7fdd78981000 rw-p 000e4000 08:03 67169473                   /usr/lib64/libkrb5.so.3.3
 15726:20170914:124231.784 7fdd78981000-7fdd789cc000 r-xp 00000000 08:03 67164888                   /usr/lib64/libgssapi_krb5.so.2.2
 15726:20170914:124231.784 7fdd789cc000-7fdd78bcc000 ---p 0004b000 08:03 67164888                   /usr/lib64/libgssapi_krb5.so.2.2
 15726:20170914:124231.784 7fdd78bcc000-7fdd78bcd000 r--p 0004b000 08:03 67164888                   /usr/lib64/libgssapi_krb5.so.2.2
 15726:20170914:124231.784 7fdd78bcd000-7fdd78bcf000 rw-p 0004c000 08:03 67164888                   /usr/lib64/libgssapi_krb5.so.2.2
 15726:20170914:124231.784 7fdd78bcf000-7fdd78bd6000 r-xp 00000000 08:03 67165980                   /usr/lib64/libgdbm.so.4.0.0
 15726:20170914:124231.784 7fdd78bd6000-7fdd78dd6000 ---p 00007000 08:03 67165980                   /usr/lib64/libgdbm.so.4.0.0
 15726:20170914:124231.784 7fdd78dd6000-7fdd78dd7000 r--p 00007000 08:03 67165980                   /usr/lib64/libgdbm.so.4.0.0
 15726:20170914:124231.784 7fdd78dd7000-7fdd78dd8000 rw-p 00008000 08:03 67165980                   /usr/lib64/libgdbm.so.4.0.0
 15726:20170914:124231.784 7fdd78dd8000-7fdd78de0000 r-xp 00000000 08:03 67475723                   /usr/lib64/libOpenIPMIutils.so.0.0.1
 15726:20170914:124231.784 7fdd78de0000-7fdd78fdf000 ---p 00008000 08:03 67475723                   /usr/lib64/libOpenIPMIutils.so.0.0.1
 15726:20170914:124231.784 7fdd78fdf000-7fdd78fe0000 r--p 00007000 08:03 67475723                   /usr/lib64/libOpenIPMIutils.so.0.0.1
 15726:20170914:124231.784 7fdd78fe0000-7fdd78fe1000 rw-p 00008000 08:03 67475723                   /usr/lib64/libOpenIPMIutils.so.0.0.1
 15726:20170914:124231.784 7fdd78fe1000-7fdd78fea000 r-xp 00000000 08:03 67875057                   /usr/lib64/libltdl.so.7.3.0
 15726:20170914:124231.784 7fdd78fea000-7fdd791e9000 ---p 00009000 08:03 67875057                   /usr/lib64/libltdl.so.7.3.0
 15726:20170914:124231.784 7fdd791e9000-7fdd791ea000 r--p 00008000 08:03 67875057                   /usr/lib64/libltdl.so.7.3.0
 15726:20170914:124231.784 7fdd791ea000-7fdd791eb000 rw-p 00009000 08:03 67875057                   /usr/lib64/libltdl.so.7.3.0
 15726:20170914:124231.784 7fdd791eb000-7fdd79210000 r-xp 00000000 08:03 67164533                   /usr/lib64/liblzma.so.5.2.2
 15726:20170914:124231.784 7fdd79210000-7fdd7940f000 ---p 00025000 08:03 67164533                   /usr/lib64/liblzma.so.5.2.2
 15726:20170914:124231.784 7fdd7940f000-7fdd79410000 r--p 00024000 08:03 67164533                   /usr/lib64/liblzma.so.5.2.2
 15726:20170914:124231.784 7fdd79410000-7fdd79411000 rw-p 00025000 08:03 67164533                   /usr/lib64/liblzma.so.5.2.2
 15726:20170914:124231.784 7fdd79411000-7fdd79426000 r-xp 00000000 08:03 67164522                   /usr/lib64/libz.so.1.2.7
 15726:20170914:124231.784 7fdd79426000-7fdd79625000 ---p 00015000 08:03 67164522                   /usr/lib64/libz.so.1.2.7
 15726:20170914:124231.784 7fdd79625000-7fdd79626000 r--p 00014000 08:03 67164522                   /usr/lib64/libz.so.1.2.7
 15726:20170914:124231.784 7fdd79626000-7fdd79627000 rw-p 00015000 08:03 67164522                   /usr/lib64/libz.so.1.2.7
 15726:20170914:124231.785 7fdd79627000-7fdd7962b000 r-xp 00000000 08:03 67165337                   /usr/lib64/libgpg-error.so.0.10.0
 15726:20170914:124231.785 7fdd7962b000-7fdd7982a000 ---p 00004000 08:03 67165337                   /usr/lib64/libgpg-error.so.0.10.0
 15726:20170914:124231.785 7fdd7982a000-7fdd7982b000 r--p 00003000 08:03 67165337                   /usr/lib64/libgpg-error.so.0.10.0
 15726:20170914:124231.785 7fdd7982b000-7fdd7982c000 rw-p 00004000 08:03 67165337                   /usr/lib64/libgpg-error.so.0.10.0
 15726:20170914:124231.785 7fdd7982c000-7fdd798a8000 r-xp 00000000 08:03 67169498                   /usr/lib64/libgcrypt.so.11.8.2
 15726:20170914:124231.785 7fdd798a8000-7fdd79aa8000 ---p 0007c000 08:03 67169498                   /usr/lib64/libgcrypt.so.11.8.2
 15726:20170914:124231.785 7fdd79aa8000-7fdd79aa9000 r--p 0007c000 08:03 67169498                   /usr/lib64/libgcrypt.so.11.8.2
 15726:20170914:124231.785 7fdd79aa9000-7fdd79aac000 rw-p 0007d000 08:03 67169498                   /usr/lib64/libgcrypt.so.11.8.2
 15726:20170914:124231.785 7fdd79aac000-7fdd79aad000 rw-p 00000000 00:00 0 
 15726:20170914:124231.785 7fdd79aad000-7fdd79bd9000 r-xp 00000000 08:03 67168101                   /usr/lib64/libgnutls.so.28.41.16
 15726:20170914:124231.785 7fdd79bd9000-7fdd79dd8000 ---p 0012c000 08:03 67168101                   /usr/lib64/libgnutls.so.28.41.16
 15726:20170914:124231.785 7fdd79dd8000-7fdd79de2000 r--p 0012b000 08:03 67168101                   /usr/lib64/libgnutls.so.28.41.16
 15726:20170914:124231.785 7fdd79de2000-7fdd79de5000 rw-p 00135000 08:03 67168101                   /usr/lib64/libgnutls.so.28.41.16
 15726:20170914:124231.785 7fdd79de5000-7fdd79ece000 r-xp 00000000 08:03 67164526                   /usr/lib64/libstdc++.so.6.0.19
 15726:20170914:124231.785 7fdd79ece000-7fdd7a0ce000 ---p 000e9000 08:03 67164526                   /usr/lib64/libstdc++.so.6.0.19
 15726:20170914:124231.785 7fdd7a0ce000-7fdd7a0d7000 r--p 000e9000 08:03 67164526                   /usr/lib64/libstdc++.so.6.0.19
 15726:20170914:124231.785 7fdd7a0d7000-7fdd7a0d9000 rw-p 000f2000 08:03 67164526                   /usr/lib64/libstdc++.so.6.0.19
 15726:20170914:124231.785 7fdd7a0d9000-7fdd7a0ee000 rw-p 00000000 00:00 0 
 15726:20170914:124231.785 7fdd7a0ee000-7fdd7a105000 r-xp 00000000 08:03 67313149                   /usr/lib64/libpthread-2.17.so
 15726:20170914:124231.785 7fdd7a105000-7fdd7a304000 ---p 00017000 08:03 67313149                   /usr/lib64/libpthread-2.17.so
 15726:20170914:124231.785 7fdd7a304000-7fdd7a305000 r--p 00016000 08:03 67313149                   /usr/lib64/libpthread-2.17.so
 15726:20170914:124231.785 7fdd7a305000-7fdd7a306000 rw-p 00017000 08:03 67313149                   /usr/lib64/libpthread-2.17.so
 15726:20170914:124231.785 7fdd7a306000-7fdd7a30a000 rw-p 00000000 00:00 0 
 15726:20170914:124231.785 7fdd7a30a000-7fdd7a4c1000 r-xp 00000000 08:03 67163966                   /usr/lib64/libc-2.17.so
 15726:20170914:124231.785 7fdd7a4c1000-7fdd7a6c0000 ---p 001b7000 08:03 67163966                   /usr/lib64/libc-2.17.so
 15726:20170914:124231.785 7fdd7a6c0000-7fdd7a6c4000 r--p 001b6000 08:03 67163966                   /usr/lib64/libc-2.17.so
 15726:20170914:124231.785 7fdd7a6c4000-7fdd7a6c6000 rw-p 001ba000 08:03 67163966                   /usr/lib64/libc-2.17.so
 15726:20170914:124231.785 7fdd7a6c6000-7fdd7a6cb000 rw-p 00000000 00:00 0 
 15726:20170914:124231.785 7fdd7a6cb000-7fdd7a6e0000 r-xp 00000000 08:03 67164217                   /usr/lib64/libgcc_s-4.8.5-20150702.so.1
 15726:20170914:124231.785 7fdd7a6e0000-7fdd7a8df000 ---p 00015000 08:03 67164217                   /usr/lib64/libgcc_s-4.8.5-20150702.so.1
 15726:20170914:124231.785 7fdd7a8df000-7fdd7a8e0000 r--p 00014000 08:03 67164217                   /usr/lib64/libgcc_s-4.8.5-20150702.so.1
 15726:20170914:124231.785 7fdd7a8e0000-7fdd7a8e1000 rw-p 00015000 08:03 67164217                   /usr/lib64/libgcc_s-4.8.5-20150702.so.1
 15726:20170914:124231.786 7fdd7a8e1000-7fdd7a941000 r-xp 00000000 08:03 67487360                   /usr/lib64/libpcre.so.1.2.0
 15726:20170914:124231.786 7fdd7a941000-7fdd7ab40000 ---p 00060000 08:03 67487360                   /usr/lib64/libpcre.so.1.2.0
 15726:20170914:124231.786 7fdd7ab40000-7fdd7ab41000 r--p 0005f000 08:03 67487360                   /usr/lib64/libpcre.so.1.2.0
 15726:20170914:124231.786 7fdd7ab41000-7fdd7ab42000 rw-p 00060000 08:03 67487360                   /usr/lib64/libpcre.so.1.2.0
 15726:20170914:124231.786 7fdd7ab42000-7fdd7ab44000 r-xp 00000000 08:03 67168211                   /usr/lib64/libpcreposix.so.0.0.1
 15726:20170914:124231.786 7fdd7ab44000-7fdd7ad43000 ---p 00002000 08:03 67168211                   /usr/lib64/libpcreposix.so.0.0.1
 15726:20170914:124231.786 7fdd7ad43000-7fdd7ad44000 r--p 00001000 08:03 67168211                   /usr/lib64/libpcreposix.so.0.0.1
 15726:20170914:124231.786 7fdd7ad44000-7fdd7ad45000 rw-p 00002000 08:03 67168211                   /usr/lib64/libpcreposix.so.0.0.1
 15726:20170914:124231.786 7fdd7ad45000-7fdd7ad5b000 r-xp 00000000 08:03 67313153                   /usr/lib64/libresolv-2.17.so
 15726:20170914:124231.786 7fdd7ad5b000-7fdd7af5b000 ---p 00016000 08:03 67313153                   /usr/lib64/libresolv-2.17.so
 15726:20170914:124231.786 7fdd7af5b000-7fdd7af5c000 r--p 00016000 08:03 67313153                   /usr/lib64/libresolv-2.17.so
 15726:20170914:124231.786 7fdd7af5c000-7fdd7af5d000 rw-p 00017000 08:03 67313153                   /usr/lib64/libresolv-2.17.so
 15726:20170914:124231.786 7fdd7af5d000-7fdd7af5f000 rw-p 00000000 00:00 0 
 15726:20170914:124231.786 7fdd7af5f000-7fdd7af61000 r-xp 00000000 08:03 67240061                   /usr/lib64/libdl-2.17.so
 15726:20170914:124231.786 7fdd7af61000-7fdd7b161000 ---p 00002000 08:03 67240061                   /usr/lib64/libdl-2.17.so
 15726:20170914:124231.786 7fdd7b161000-7fdd7b162000 r--p 00002000 08:03 67240061                   /usr/lib64/libdl-2.17.so
 15726:20170914:124231.786 7fdd7b162000-7fdd7b163000 rw-p 00003000 08:03 67240061                   /usr/lib64/libdl-2.17.so
 15726:20170914:124231.786 7fdd7b163000-7fdd7b263000 r-xp 00000000 08:03 67240428                   /usr/lib64/libm-2.17.so
 15726:20170914:124231.786 7fdd7b263000-7fdd7b463000 ---p 00100000 08:03 67240428                   /usr/lib64/libm-2.17.so
 15726:20170914:124231.786 7fdd7b463000-7fdd7b464000 r--p 00100000 08:03 67240428                   /usr/lib64/libm-2.17.so
 15726:20170914:124231.786 7fdd7b464000-7fdd7b465000 rw-p 00101000 08:03 67240428                   /usr/lib64/libm-2.17.so
 15726:20170914:124231.786 7fdd7b465000-7fdd7b4ca000 r-xp 00000000 08:03 67166203                   /usr/lib64/libcurl.so.4.3.0
 15726:20170914:124231.786 7fdd7b4ca000-7fdd7b6ca000 ---p 00065000 08:03 67166203                   /usr/lib64/libcurl.so.4.3.0
 15726:20170914:124231.787 7fdd7b6ca000-7fdd7b6cc000 r--p 00065000 08:03 67166203                   /usr/lib64/libcurl.so.4.3.0
 15726:20170914:124231.787 7fdd7b6cc000-7fdd7b6cd000 rw-p 00067000 08:03 67166203                   /usr/lib64/libcurl.so.4.3.0
 15726:20170914:124231.787 7fdd7b6cd000-7fdd7b6ce000 rw-p 00000000 00:00 0 
 15726:20170914:124231.787 7fdd7b6ce000-7fdd7b6dc000 r-xp 00000000 08:03 67169411                   /usr/lib64/liblber-2.4.so.2.10.3
 15726:20170914:124231.787 7fdd7b6dc000-7fdd7b8db000 ---p 0000e000 08:03 67169411                   /usr/lib64/liblber-2.4.so.2.10.3
 15726:20170914:124231.787 7fdd7b8db000-7fdd7b8dc000 r--p 0000d000 08:03 67169411                   /usr/lib64/liblber-2.4.so.2.10.3
 15726:20170914:124231.787 7fdd7b8dc000-7fdd7b8dd000 rw-p 0000e000 08:03 67169411                   /usr/lib64/liblber-2.4.so.2.10.3
 15726:20170914:124231.787 7fdd7b8dd000-7fdd7b92d000 r-xp 00000000 08:03 67169412                   /usr/lib64/libldap-2.4.so.2.10.3
 15726:20170914:124231.787 7fdd7b92d000-7fdd7bb2c000 ---p 00050000 08:03 67169412                   /usr/lib64/libldap-2.4.so.2.10.3
 15726:20170914:124231.787 7fdd7bb2c000-7fdd7bb2f000 r--p 0004f000 08:03 67169412                   /usr/lib64/libldap-2.4.so.2.10.3
 15726:20170914:124231.787 7fdd7bb2f000-7fdd7bb30000 rw-p 00052000 08:03 67169412                   /usr/lib64/libldap-2.4.so.2.10.3
 15726:20170914:124231.787 7fdd7bb30000-7fdd7bcf0000 r-xp 00000000 08:03 67164213                   /usr/lib64/libcrypto.so.1.0.1e
 15726:20170914:124231.787 7fdd7bcf0000-7fdd7bef0000 ---p 001c0000 08:03 67164213                   /usr/lib64/libcrypto.so.1.0.1e
 15726:20170914:124231.787 7fdd7bef0000-7fdd7bf0a000 r--p 001c0000 08:03 67164213                   /usr/lib64/libcrypto.so.1.0.1e
 15726:20170914:124231.787 7fdd7bf0a000-7fdd7bf16000 rw-p 001da000 08:03 67164213                   /usr/lib64/libcrypto.so.1.0.1e
 15726:20170914:124231.787 7fdd7bf16000-7fdd7bf1a000 rw-p 00000000 00:00 0 
 15726:20170914:124231.787 7fdd7bf1a000-7fdd7bf7e000 r-xp 00000000 08:03 67168216                   /usr/lib64/libssl.so.1.0.1e
 15726:20170914:124231.787 7fdd7bf7e000-7fdd7c17d000 ---p 00064000 08:03 67168216                   /usr/lib64/libssl.so.1.0.1e
 15726:20170914:124231.787 7fdd7c17d000-7fdd7c181000 r--p 00063000 08:03 67168216                   /usr/lib64/libssl.so.1.0.1e
 15726:20170914:124231.787 7fdd7c181000-7fdd7c188000 rw-p 00067000 08:03 67168216                   /usr/lib64/libssl.so.1.0.1e
 15726:20170914:124231.787 7fdd7c188000-7fdd7c1ce000 r-xp 00000000 08:03 67551277                   /usr/lib64/libevent-2.0.so.5.1.9
 15726:20170914:124231.787 7fdd7c1ce000-7fdd7c3cd000 ---p 00046000 08:03 67551277                   /usr/lib64/libevent-2.0.so.5.1.9
 15726:20170914:124231.787 7fdd7c3cd000-7fdd7c3ce000 r--p 00045000 08:03 67551277                   /usr/lib64/libevent-2.0.so.5.1.9
 15726:20170914:124231.787 7fdd7c3ce000-7fdd7c3cf000 rw-p 00046000 08:03 67551277                   /usr/lib64/libevent-2.0.so.5.1.9
 15726:20170914:124231.787 7fdd7c3cf000-7fdd7c3d0000 rw-p 00000000 00:00 0 
 15726:20170914:124231.787 7fdd7c3d0000-7fdd7c3d5000 r-xp 00000000 08:03 67475717                   /usr/lib64/libOpenIPMIposix.so.0.0.1
 15726:20170914:124231.787 7fdd7c3d5000-7fdd7c5d4000 ---p 00005000 08:03 67475717                   /usr/lib64/libOpenIPMIposix.so.0.0.1
 15726:20170914:124231.787 7fdd7c5d4000-7fdd7c5d5000 r--p 00004000 08:03 67475717                   /usr/lib64/libOpenIPMIposix.so.0.0.1
 15726:20170914:124231.787 7fdd7c5d5000-7fdd7c5d6000 rw-p 00005000 08:03 67475717                   /usr/lib64/libOpenIPMIposix.so.0.0.1
 15726:20170914:124231.795 7fdd7c5d6000-7fdd7c6c3000 r-xp 00000000 08:03 69232243                   /usr/lib64/libOpenIPMI.so.0.0.5
 15726:20170914:124231.795 7fdd7c6c3000-7fdd7c8c3000 ---p 000ed000 08:03 69232243                   /usr/lib64/libOpenIPMI.so.0.0.5
 15726:20170914:124231.796 7fdd7c8c3000-7fdd7c8d9000 r--p 000ed000 08:03 69232243                   /usr/lib64/libOpenIPMI.so.0.0.5
 15726:20170914:124231.796 7fdd7c8d9000-7fdd7c8df000 rw-p 00103000 08:03 69232243                   /usr/lib64/libOpenIPMI.so.0.0.5
 15726:20170914:124231.796 7fdd7c8df000-7fdd7c8e3000 rw-p 00000000 00:00 0 
 15726:20170914:124231.796 7fdd7c8e3000-7fdd7c90b000 r-xp 00000000 08:03 67165226                   /usr/lib64/libssh2.so.1.0.1
 15726:20170914:124231.796 7fdd7c90b000-7fdd7cb0b000 ---p 00028000 08:03 67165226                   /usr/lib64/libssh2.so.1.0.1
 15726:20170914:124231.796 7fdd7cb0b000-7fdd7cb0c000 r--p 00028000 08:03 67165226                   /usr/lib64/libssh2.so.1.0.1
 15726:20170914:124231.796 7fdd7cb0c000-7fdd7cb0d000 rw-p 00029000 08:03 67165226                   /usr/lib64/libssh2.so.1.0.1
 15726:20170914:124231.796 7fdd7cb0d000-7fdd7cbd5000 r-xp 00000000 08:03 67165002                   /usr/lib64/libnetsnmp.so.31.0.2
 15726:20170914:124231.796 7fdd7cbd5000-7fdd7cdd4000 ---p 000c8000 08:03 67165002                   /usr/lib64/libnetsnmp.so.31.0.2
 15726:20170914:124231.796 7fdd7cdd4000-7fdd7cdd7000 r--p 000c7000 08:03 67165002                   /usr/lib64/libnetsnmp.so.31.0.2
 15726:20170914:124231.796 7fdd7cdd7000-7fdd7cdd9000 rw-p 000ca000 08:03 67165002                   /usr/lib64/libnetsnmp.so.31.0.2
 15726:20170914:124231.796 7fdd7cdd9000-7fdd7ce0f000 rw-p 00000000 00:00 0 
 15726:20170914:124231.796 7fdd7ce0f000-7fdd7ce6f000 r-xp 00000000 08:03 67551167                   /usr/lib64/libodbc.so.2.0.0
 15726:20170914:124231.796 7fdd7ce6f000-7fdd7d06e000 ---p 00060000 08:03 67551167                   /usr/lib64/libodbc.so.2.0.0
 15726:20170914:124231.796 7fdd7d06e000-7fdd7d06f000 r--p 0005f000 08:03 67551167                   /usr/lib64/libodbc.so.2.0.0
 15726:20170914:124231.796 7fdd7d06f000-7fdd7d076000 rw-p 00060000 08:03 67551167                   /usr/lib64/libodbc.so.2.0.0
 15726:20170914:124231.796 7fdd7d076000-7fdd7d077000 rw-p 00000000 00:00 0 
 15726:20170914:124231.796 7fdd7d077000-7fdd7d1d6000 r-xp 00000000 08:03 67168568                   /usr/lib64/libxml2.so.2.9.1
 15726:20170914:124231.796 7fdd7d1d6000-7fdd7d3d5000 ---p 0015f000 08:03 67168568                   /usr/lib64/libxml2.so.2.9.1
 15726:20170914:124231.796 7fdd7d3d5000-7fdd7d3dd000 r--p 0015e000 08:03 67168568                   /usr/lib64/libxml2.so.2.9.1
 15726:20170914:124231.796 7fdd7d3dd000-7fdd7d3df000 rw-p 00166000 08:03 67168568                   /usr/lib64/libxml2.so.2.9.1
 15726:20170914:124231.796 7fdd7d3df000-7fdd7d3e1000 rw-p 00000000 00:00 0 
 15726:20170914:124231.796 7fdd7d3e1000-7fdd7d3ee000 r-xp 00000000 08:03 69232231                   /usr/lib64/libiksemel.so.3.1.1
 15726:20170914:124231.796 7fdd7d3ee000-7fdd7d5ed000 ---p 0000d000 08:03 69232231                   /usr/lib64/libiksemel.so.3.1.1
 15726:20170914:124231.796 7fdd7d5ed000-7fdd7d5ee000 r--p 0000c000 08:03 69232231                   /usr/lib64/libiksemel.so.3.1.1
 15726:20170914:124231.796 7fdd7d5ee000-7fdd7d5ef000 rw-p 0000d000 08:03 69232231                   /usr/lib64/libiksemel.so.3.1.1
 15726:20170914:124231.797 7fdd7d5ef000-7fdd7d8ee000 r-xp 00000000 08:03 42350                      /usr/lib64/mysql/libmysqlclient.so.18.1.0
 15726:20170914:124231.797 7fdd7d8ee000-7fdd7daee000 ---p 002ff000 08:03 42350                      /usr/lib64/mysql/libmysqlclient.so.18.1.0
 15726:20170914:124231.797 7fdd7daee000-7fdd7daf4000 r--p 002ff000 08:03 42350                      /usr/lib64/mysql/libmysqlclient.so.18.1.0
 15726:20170914:124231.797 7fdd7daf4000-7fdd7dbc1000 rw-p 00305000 08:03 42350                      /usr/lib64/mysql/libmysqlclient.so.18.1.0
 15726:20170914:124231.797 7fdd7dbc1000-7fdd7dbc6000 rw-p 00000000 00:00 0 
 15726:20170914:124231.797 7fdd7dbc6000-7fdd7dbe6000 r-xp 00000000 08:03 67165213                   /usr/lib64/ld-2.17.so
 15726:20170914:124231.797 7fdd7dda2000-7fdd7ddbe000 rw-s 00000000 00:04 1376262                    /SYSV00000000 (deleted)
 15726:20170914:124231.797 7fdd7ddbe000-7fdd7dddb000 rw-p 00000000 00:00 0 
 15726:20170914:124231.797 7fdd7dde3000-7fdd7dde4000 rw-p 00000000 00:00 0 
 15726:20170914:124231.797 7fdd7dde4000-7fdd7dde5000 rw-p 00000000 00:00 0 
 15726:20170914:124231.797 7fdd7dde5000-7fdd7dde6000 rw-p 00000000 00:00 0 
 15726:20170914:124231.797 7fdd7dde6000-7fdd7dde7000 r--p 00020000 08:03 67165213                   /usr/lib64/ld-2.17.so
 15726:20170914:124231.797 7fdd7dde7000-7fdd7dde8000 rw-p 00021000 08:03 67165213                   /usr/lib64/ld-2.17.so
 15726:20170914:124231.797 7fdd7dde8000-7fdd7dde9000 rw-p 00000000 00:00 0 
 15726:20170914:124231.797 7fdd7dde9000-7fdd7dfe6000 r-xp 00000000 08:03 67168394                   /usr/sbin/zabbix_server_mysql
 15726:20170914:124231.797 7fdd7e1e5000-7fdd7e24d000 r--p 001fc000 08:03 67168394                   /usr/sbin/zabbix_server_mysql
 15726:20170914:124231.797 7fdd7e24d000-7fdd7e256000 rw-p 00264000 08:03 67168394                   /usr/sbin/zabbix_server_mysql
 15726:20170914:124231.797 7fdd7e256000-7fdd7e25e000 rw-p 00000000 00:00 0 
 15726:20170914:124231.797 7fdd7ef71000-7fdd7ef92000 rw-p 00000000 00:00 0                          [heap]
 15726:20170914:124231.797 7fdd7ef92000-7fdd7efde000 rw-p 00000000 00:00 0                          [heap]
 15726:20170914:124231.797 7fdd7efde000-7fdd7f66b000 rw-p 00000000 00:00 0                          [heap]
 15726:20170914:124231.797 7ffd82169000-7ffd8219f000 rw-p 00000000 00:00 0                          [stack]
 15726:20170914:124231.797 7ffd821fc000-7ffd821fe000 r-xp 00000000 00:00 0                          [vdso]
 15726:20170914:124231.797 ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0                  [vsyscall]
 15726:20170914:124231.797 ================================
Comment by Vladislavs Sokurenko [ 2017 Sep 14 ]

Thank you for your report, closing as a duplicate, please upgrade to the latest version.

Comment by Alain Ganuchaud [ 2017 Sep 27 ]

3.4.2 didn't solve the problem, I attach new server log (zabbix_server.log.2)
Thanks for help

[root]# rpm -qa | grep zabbix
zabbix-web-mysql-3.4.2-1.el7.noarch
zabbix-get-3.4.2-1.el7.x86_64
zabbix-proxy-sqlite3-3.4.2-1.el7.x86_64
zabbix-java-gateway-3.4.2-1.el7.x86_64
zabbix-proxy-mysql-3.4.2-1.el7.x86_64
zabbix-agent-3.4.2-1.el7.x86_64
zabbix-release-3.4-2.el7.noarch
zabbix-server-mysql-3.4.2-1.el7.x86_64
zabbix-web-3.4.2-1.el7.noarch

zabbix_server -V
zabbix_server (Zabbix) 3.4.2
Revision 72885 25 September 2017, compilation time: Sep 26 2017 08:14:41

Comment by Vladislavs Sokurenko [ 2017 Sep 27 ]

New backtrace for easier searching:

 27656:20170927:061506.138 server #359 started [preprocessing worker #22]
 27633:20170927:061506.165 server #353 started [preprocessing worker #16]
 27502:20170927:061509.308 Got signal [signal:11(SIGSEGV),reason:1,refaddr:(nil)]. Crashing ...
 27502:20170927:061509.308 ====== Fatal information: ======
 27502:20170927:061509.308 Program counter: 0x7fc414d58d47
 27502:20170927:061509.308 === Registers: ===
 27502:20170927:061509.308 r8      =     7fc4150da060 =      140480143532128 =      140480143532128
 27502:20170927:061509.308 r9      =               28 =                   40 =                   40
 27502:20170927:061509.309 r10     =                0 =                    0 =                    0
 27502:20170927:061509.309 r11     =                0 =                    0 =                    0
 27502:20170927:061509.309 r12     =              258 =                  600 =                  600
 27502:20170927:061509.309 r13     =                0 =                    0 =                    0
 27502:20170927:061509.309 r14     =                0 =                    0 =                    0
 27502:20170927:061509.309 r15     =                0 =                    0 =                    0
 27502:20170927:061509.309 rdi     =                0 =                    0 =                    0
 27502:20170927:061509.309 rsi     =                0 =                    0 =                    0
 27502:20170927:061509.309 rbp     =                0 =                    0 =                    0
 27502:20170927:061509.309 rbx     =                0 =                    0 =                    0
 27502:20170927:061509.310 rdx     =                a =                   10 =                   10
 27502:20170927:061509.310 rax     =     7fc4150d7260 =      140480143520352 =      140480143520352
 27502:20170927:061509.310 rcx     =                0 =                    0 =                    0
 27502:20170927:061509.310 rsp     =     7ffcb1a6e7f0 =      140723288991728 =      140723288991728
 27502:20170927:061509.310 rip     =     7fc414d58d47 =      140480139857223 =      140480139857223
 27502:20170927:061509.310 efl     =            10283 =                66179 =                66179
 27502:20170927:061509.310 csgsfs  =               33 =                   51 =                   51
 27502:20170927:061509.310 err     =                4 =                    4 =                    4
 27502:20170927:061509.310 trapno  =                e =                   14 =                   14
 27502:20170927:061509.310 oldmask =                0 =                    0 =                    0
 27502:20170927:061509.310 cr2     =                0 =                    0 =                    0
 27502:20170927:061509.311 === Backtrace: ===
 27502:20170927:061509.311 16: /usr/sbin/zabbix_server: trapper #8 [processing data](zbx_log_fatal_info+0x13c) [0x7fc4188e7d40]
 27502:20170927:061509.311 15: /usr/sbin/zabbix_server: trapper #8 [processing data](+0xea155) [0x7fc4188e8155]
 27502:20170927:061509.311 14: /lib64/libc.so.6(+0x35250) [0x7fc414d54250]
 27502:20170927:061509.311 13: /lib64/libc.so.6(+0x39d47) [0x7fc414d58d47]
 27502:20170927:061509.311 12: /lib64/libc.so.6(atoi+0x10) [0x7fc414d55750]
 27502:20170927:061509.311 11: /usr/sbin/zabbix_server: trapper #8 [processing data](zbx_tm_get_remote_tasks+0x247) [0x7fc4188844ef]
 27502:20170927:061509.311 10: /usr/sbin/zabbix_server: trapper #8 [processing data](zbx_send_proxy_data_respose+0x4b) [0x7fc41886a723]
 27502:20170927:061509.311 9: /usr/sbin/zabbix_server: trapper #8 [processing data](zbx_recv_proxy_data+0x1b2) [0x7fc41886a9fe]
 27502:20170927:061509.311 8: /usr/sbin/zabbix_server: trapper #8 [processing data](+0x6a6df) [0x7fc4188686df]
 27502:20170927:061509.311 7: /usr/sbin/zabbix_server: trapper #8 [processing data](+0x6ae47) [0x7fc418868e47]
 27502:20170927:061509.311 6: /usr/sbin/zabbix_server: trapper #8 [processing data](trapper_thread+0x21f) [0x7fc418869068]
 27502:20170927:061509.311 5: /usr/sbin/zabbix_server: trapper #8 [processing data](zbx_thread_start+0x37) [0x7fc4188f513b]
 27502:20170927:061509.311 4: /usr/sbin/zabbix_server: trapper #8 [processing data](MAIN_ZABBIX_ENTRY+0x8c8) [0x7fc41883e151]
 27502:20170927:061509.311 3: /usr/sbin/zabbix_server: trapper #8 [processing data](daemon_start+0x325) [0x7fc4188e7486]
 27502:20170927:061509.311 2: /usr/sbin/zabbix_server: trapper #8 [processing data](main+0x317) [0x7fc41883d887]
 27502:20170927:061509.311 1: /lib64/libc.so.6(__libc_start_main+0xf5) [0x7fc414d40b35]
 27502:20170927:061509.311 0: /usr/sbin/zabbix_server: trapper #8 [processing data](+0x33b99) [0x7fc418831b99]
Comment by Vladislavs Sokurenko [ 2017 Sep 27 ]

Could you please be so kind and provide objdump:

objdump -DSswx zabbix_server

Please also try this command

select t.taskid,t.type,t.clock,t.ttl,c.command_type,c.execute_on,c.port,c.authtype,c.username,c.password,c.publickey,c.privatekey,c.command,c.alertid,c.parent_taskid,c.hostid from task t left join task_remote_command c on t.taskid=c.taskid where t.status=1 and t.proxy_hostid !=0 and (command is NULL or alertid is NULL or parent_taskid is NULL or hostid is NULL) order by t.taskid;
Comment by Alain Ganuchaud [ 2017 Sep 27 ]

Thanks, I attached the requested files.

Comment by Andris Zeila [ 2017 Sep 28 ]

In your case for whatver reason there are no corresponding records in task_remote_command table for two tasks. This causes the crash. You can safely remove them while we are investigating the possible causes:

delete t from task t where type=2 and not exists (select null from task_remote_command c where t.taskid=c.taskid);

Regarding hunge queue - I understand you are using active proxies. In 3.4.1 there was a bug which caused unnecessary delays when sending proxy history. This might have been cause for delayed history data uploads and hence queue.

Comment by Alain Ganuchaud [ 2017 Sep 29 ]

Hi Andris, thank you for your help.

FYI, the server goes on crashing even if the tasks are now cleaned. I attach a new log (zabbix_server.log.3) in the hope it could help. We also had a regular (every 10s) job for cleaning those orphan tasks.

If those orphan tasks are the root cause, could you please tell me if it is possible to workaround this situation.
Thanks Alain

Comment by Andris Zeila [ 2017 Sep 29 ]

If those orphan tasks are the root cause

From the backtrace it would appear so. The following patch should fix it. However it's quite strange that orphaned tasks are created so regularly. 

Index: src/zabbix_server/taskmanager/proxy_tasks.c
===================================================================
--- src/zabbix_server/taskmanager/proxy_tasks.c	(revision 73037)
+++ src/zabbix_server/taskmanager/proxy_tasks.c	(working copy)
@@ -56,6 +56,7 @@
 			" where t.status=%d"
 				" and t.proxy_hostid=" ZBX_FS_UI64
 				" and t.clock+t.ttl>%d"
+				" and c.command_type is not null"
 			" order by t.taskid",
 			ZBX_TM_STATUS_NEW, proxy_hostid, time(NULL));
Comment by Andrea Biscuola (Inactive) [ 2017 Oct 04 ]

Succesfully tested

Comment by richlv [ 2017 Oct 04 ]

the last comment says "successfully tested", while there have been no comments on what has been identified and/or done.

this is extremely confusing. was the issue really ignored for 5 days ?

wiper There are comments, but...
Shortly - the task processing was improved to prevent/handle situations where there are records in task table without correspoding records in task data tables (task_remote_command, task_remote_command_result etc).

Comment by Andrea Biscuola (Inactive) [ 2017 Oct 05 ]

Hi richlv

Can you clarify? You mean that details regarding what was committed in the development branch are lacking? Actually for properly testing this it took a bit of time. In particular:

  • Test that the introduced code didn't cause regressions
  • Test that in all the possible conditions and combinations no orphaned records were generated.
  • Stress test for checking if the queue was growing as reported by the user.
  • Code review.

Or did you mean something else?

Comment by Andris Zeila [ 2017 Oct 05 ]

I changed the issue title to closer reflect the problem that was handled here. If there is still problem with huge pre-processor queue please create new issue, describing the setup in more details - number of values processed per second, commonly used item pre-processing operations etc.

Comment by Andris Zeila [ 2017 Oct 05 ]

Released in:

  • pre-3.4.3rc1 r73187
  • pre-4.0.0alpha1 r73188
Comment by Andris Zeila [ 2017 Oct 06 ]

(1) Fixed nested transactions in development branch svn://svn.zabbix.com/branches/dev/ZBX-12725

abs CLOSED

Comment by Andris Zeila [ 2017 Oct 06 ]

Released in:

  • pre-3.4.3rc1 r73271
  • pre-4.0.0alpha1 r73272
Comment by richlv [ 2017 Oct 06 ]

wiper, thank you a lot for the detail. when you say "There are comments", where can those be seen ?

abs, i meant the fact that there were no comments preceding "Succesfully tested" clarifying what will be done and no comment of any fix actually being available (with the link to the svn branch).

Comment by Andris Zeila [ 2017 Oct 11 ]

When processing huge amounts of data (normal situation with proxies after server downtime) 'change' preprocessing steps indeed could stall the manager, see ZBX-12857.

It's not related to the crashes described here.

Generated at Tue Apr 16 19:15:51 EEST 2024 using Jira 9.12.4#9120004-sha1:625303b708afdb767e17cb2838290c41888e9ff0.