[ZBX-12898] rare unknown crashes in poller Created: 2017 Oct 18  Updated: 2018 Mar 21  Resolved: 2018 Mar 21

Status: Closed
Project: ZABBIX BUGS AND ISSUES
Component/s: Proxy (P), Server (S)
Affects Version/s: 3.2.8
Fix Version/s: None

Type: Problem report Priority: Critical
Reporter: Oleksii Zagorskyi Assignee: Unassigned
Resolution: Duplicate Votes: 0
Labels: crash, odbc
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

Oracle DB backend.


Attachments: Zip Archive zabbix_server.dump.zip     Zip Archive zabbix_server_3.2.4_recompile.zip    
Issue Links:
Duplicate
duplicates ZBX-13010 Zabbix Server 3.4.3 Segmentation Faul... Closed
Sprint: Sprint 19, Sprint 20, Sprint 22

 Description   

It happened when we was on 3.2.4:

  6248:20170908:115505.750 Got signal [signal:11(SIGSEGV),reason:1,refaddr:(nil)]. Crashing ...
...
  6248:20170908:115505.751 === Backtrace: ===
  6248:20170908:115505.817 2: /usr/local/sbin/zabbix_server: poller #49 [got 18 values in 3.212184 sec, getting values](print_fatal_info+0xa6) [0x48bb66]
  6248:20170908:115505.817 1: /usr/local/sbin/zabbix_server: poller #49 [got 18 values in 3.212184 sec, getting values]() [0x48c293]
  6248:20170908:115505.817 0: /lib64/libc.so.6() [0x3a1cc32510]
*** glibc detected *** /usr/local/sbin/zabbix_server: poller #27 [got 7 values in 0.076096 sec, getting values]: double free or corruption (out): 0x000000000205f800 ***
======= Backtrace: =========
/lib64/libc.so.6[0x3a1cc75dee]
/lib64/libc.so.6[0x3a1cc78c80]
/usr/local/sbin/zabbix_server: poller #27 [got 7 values in 0.076096 sec, getting values](free_request+0x11)[0x44d621]
/usr/local/sbin/zabbix_server: poller #27 [got 7 values in 0.076096 sec, getting values](get_value_db+0x94)[0x435734]
/usr/local/sbin/zabbix_server: poller #27 [got 7 values in 0.076096 sec, getting values][0x42c7a8]
/usr/local/sbin/zabbix_server: poller #27 [got 7 values in 0.076096 sec, getting values](poller_thread+0xfe)[0x42c96e]
/usr/local/sbin/zabbix_server: poller #27 [got 7 values in 0.076096 sec, getting values](zbx_thread_start+0x62)[0x48c832]
/usr/local/sbin/zabbix_server: poller #27 [got 7 values in 0.076096 sec, getting values](MAIN_ZABBIX_ENTRY+0x38b)[0x41f02b]
/usr/local/sbin/zabbix_server: poller #27 [got 7 values in 0.076096 sec, getting values](daemon_start+0x1ae)[0x48ae6e]
/usr/local/sbin/zabbix_server: poller #27 [got 7 values in 0.076096 sec, getting values](main+0x368)[0x41f948]
/lib64/libc.so.6(__libc_start_main+0xfd)[0x3a1cc1ed1d]
/usr/local/sbin/zabbix_server: poller #27 [got 7 values in 0.076096 sec, getting values][0x416949]
...
 28171:20170921:093026.399 One child process died (PID:28203,exitcode/signal:6). Exiting ...
...
 28171:20170921:093032.304 Zabbix Server stopped. Zabbix 3.2.4 (revision 65975).

We upgraded to 3.2.8 and it happened again:

 17403:20171018:074946.215 Zabbix agent item "perf_counter[\2\18]" on host "kys" failed: first network error, wait for 20 seconds
...
 17501:20171018:093034.407 cannot send list of active checks to "1.12.1.1": host [KSI] not found
 17403:20171018:093034.620 Got signal [signal:11(SIGSEGV),reason:1,refaddr:0x7ffc001d2930]. Crashing ...
 17403:20171018:093034.620 ====== Fatal information: ======
 17403:20171018:093034.620 Program counter: 0x42b51a
 17403:20171018:093034.620 === Registers: ===
 17403:20171018:093034.620 r8      =         ffffffff =           4294967295 =           4294967295
 17403:20171018:093034.620 r9      =       3a1cf8e170 =         249594175856 =         249594175856
 17403:20171018:093034.621 r10     =       3a1cf8e170 =         249594175856 =         249594175856
 17403:20171018:093034.621 r11     =              246 =                  582 =                  582
 17403:20171018:093034.621 r12     =     7ffc28341300 =      140720982987520 =      140720982987520
 17403:20171018:093034.621 r13     =                1 =                    1 =                    1
 17403:20171018:093034.621 r14     =                0 =                    0 =                    0
 17403:20171018:093034.621 r15     =     7ffc001d06c0 =      140720310388416 =      140720310388416
 17403:20171018:093034.621 rdi     =                4 =                    4 =                    4
 17403:20171018:093034.621 rsi     =           4e62b8 =              5137080 =              5137080
 17403:20171018:093034.621 rbp     =                0 =                    0 =                    0
 17403:20171018:093034.621 rbx     =     7ffc001d06c0 =      140720310388416 =      140720310388416
 17403:20171018:093034.621 rdx     =           4e62a8 =              5137064 =              5137064
 17403:20171018:093034.621 rax     =                0 =                    0 =                    0
 17403:20171018:093034.621 rcx     =               80 =                  128 =                  128
 17403:20171018:093034.621 rsp     =     7ffc281d0610 =      140720981476880 =      140720981476880
 17403:20171018:093034.621 rip     =           42b51a =              4371738 =              4371738
 17403:20171018:093034.621 efl     =            10246 =                66118 =                66118
 17403:20171018:093034.621 csgsfs  =               33 =                   51 =                   51
 17403:20171018:093034.621 err     =                4 =                    4 =                    4
 17403:20171018:093034.621 trapno  =                e =                   14 =                   14
 17403:20171018:093034.621 oldmask =                0 =                    0 =                    0
 17403:20171018:093034.621 cr2     =     7ffc001d2930 =      140720310397232 =      140720310397232
 17403:20171018:093034.621 === Backtrace: ===
 17403:20171018:093034.734 10: /usr/local/sbin/zabbix_server: poller #33 [got 0 values in 0.000005 sec, getting values](print_fatal_info+0xa6) [0x48a656]
 17403:20171018:093034.735 9: /usr/local/sbin/zabbix_server: poller #33 [got 0 values in 0.000005 sec, getting values]() [0x48ad63]
 17403:20171018:093034.735 8: /lib64/libc.so.6() [0x3a1cc32510]
 17403:20171018:093034.735 7: /usr/local/sbin/zabbix_server: poller #33 [got 0 values in 0.000005 sec, getting values]() [0x42b51a]
 17403:20171018:093034.735 6: /usr/local/sbin/zabbix_server: poller #33 [got 0 values in 0.000005 sec, getting values](poller_thread+0xf6) [0x42bba6]
 17403:20171018:093034.735 5: /usr/local/sbin/zabbix_server: poller #33 [got 0 values in 0.000005 sec, getting values](zbx_thread_start+0x62) [0x48b2a2]
 17403:20171018:093034.735 4: /usr/local/sbin/zabbix_server: poller #33 [got 0 values in 0.000005 sec, getting values](MAIN_ZABBIX_ENTRY+0x37b) [0x41e1ab]
 17403:20171018:093034.735 3: /usr/local/sbin/zabbix_server: poller #33 [got 0 values in 0.000005 sec, getting values](daemon_start+0x1ae) [0x48995e]
 17403:20171018:093034.735 2: /usr/local/sbin/zabbix_server: poller #33 [got 0 values in 0.000005 sec, getting values](main+0x3c8) [0x41eb18]
 17403:20171018:093034.735 1: /lib64/libc.so.6(__libc_start_main+0xfd) [0x3a1cc1ed1d]
 17403:20171018:093034.735 0: /usr/local/sbin/zabbix_server: poller #33 [got 0 values in 0.000005 sec, getting values]() [0x415209]
 17403:20171018:093034.735 === Memory map: ===
 17403:20171018:093034.735 00400000-00589000 r-xp 00000000 fd:00 1268                               /usr/local/sbin/zabbix_server
 17403:20171018:093034.735 00788000-00790000 rw-p 00188000 fd:00 1268                               /usr/local/sbin/zabbix_server
 17403:20171018:093034.735 00790000-00796000 rw-p 00000000 00:00 0 
 17403:20171018:093034.735 01918000-01939000 rw-p 00000000 00:00 0 
 17403:20171018:093034.735 01939000-01ae7000 rw-p 00000000 00:00 0 
 17403:20171018:093034.735 01ae7000-02009000 rw-p 00000000 00:00 0 
 17403:20171018:093034.735 3a1c800000-3a1c820000 r-xp 00000000 fd:00 4266                           /lib64/ld-2.12.so
 17403:20171018:093034.735 3a1ca1f000-3a1ca21000 r--p 0001f000 fd:00 4266                           /lib64/ld-2.12.so
 17403:20171018:093034.735 3a1ca21000-3a1ca22000 rw-p 00021000 fd:00 4266                           /lib64/ld-2.12.so
 17403:20171018:093034.735 3a1ca22000-3a1ca23000 rw-p 00000000 00:00 0 
 17403:20171018:093034.735 3a1cc00000-3a1cd8a000 r-xp 00000000 fd:00 4267                           /lib64/libc-2.12.so
 17403:20171018:093034.735 3a1cd8a000-3a1cf8a000 ---p 0018a000 fd:00 4267                           /lib64/libc-2.12.so
 17403:20171018:093034.735 3a1cf8a000-3a1cf8e000 r--p 0018a000 fd:00 4267                           /lib64/libc-2.12.so
 17403:20171018:093034.736 3a1cf8e000-3a1cf90000 rw-p 0018e000 fd:00 4267                           /lib64/libc-2.12.so
 17403:20171018:093034.736 3a1cf90000-3a1cf94000 rw-p 00000000 00:00 0 
 17403:20171018:093034.736 3a1d000000-3a1d002000 r-xp 00000000 fd:00 4726                           /lib64/libdl-2.12.so
 17403:20171018:093034.736 3a1d002000-3a1d202000 ---p 00002000 fd:00 4726                           /lib64/libdl-2.12.so
 17403:20171018:093034.736 3a1d202000-3a1d203000 r--p 00002000 fd:00 4726                           /lib64/libdl-2.12.so
 17403:20171018:093034.736 3a1d203000-3a1d204000 rw-p 00003000 fd:00 4726                           /lib64/libdl-2.12.so
 17403:20171018:093034.736 3a1d400000-3a1d417000 r-xp 00000000 fd:00 4313                           /lib64/libpthread-2.12.so
 17403:20171018:093034.736 3a1d417000-3a1d617000 ---p 00017000 fd:00 4313                           /lib64/libpthread-2.12.so
 17403:20171018:093034.736 3a1d617000-3a1d618000 r--p 00017000 fd:00 4313                           /lib64/libpthread-2.12.so
 17403:20171018:093034.736 3a1d618000-3a1d619000 rw-p 00018000 fd:00 4313                           /lib64/libpthread-2.12.so
 17403:20171018:093034.736 3a1d619000-3a1d61d000 rw-p 00000000 00:00 0 
 17403:20171018:093034.736 3a1d800000-3a1d815000 r-xp 00000000 fd:00 4729                           /lib64/libz.so.1.2.3
 17403:20171018:093034.736 3a1d815000-3a1da14000 ---p 00015000 fd:00 4729                           /lib64/libz.so.1.2.3
 17403:20171018:093034.736 3a1da14000-3a1da15000 r--p 00014000 fd:00 4729                           /lib64/libz.so.1.2.3
 17403:20171018:093034.736 3a1da15000-3a1da16000 rw-p 00015000 fd:00 4729                           /lib64/libz.so.1.2.3
 17403:20171018:093034.736 3a1dc00000-3a1dc83000 r-xp 00000000 fd:00 4486                           /lib64/libm-2.12.so
 17403:20171018:093034.736 3a1dc83000-3a1de82000 ---p 00083000 fd:00 4486                           /lib64/libm-2.12.so
 17403:20171018:093034.736 3a1de82000-3a1de83000 r--p 00082000 fd:00 4486                           /lib64/libm-2.12.so
 17403:20171018:093034.736 3a1de83000-3a1de84000 rw-p 00083000 fd:00 4486                           /lib64/libm-2.12.so
 17403:20171018:093034.736 3a1e000000-3a1e05f000 r-xp 00000000 fd:00 15681                          /usr/lib64/libodbc.so.2.0.0
 17403:20171018:093034.736 3a1e05f000-3a1e25e000 ---p 0005f000 fd:00 15681                          /usr/lib64/libodbc.so.2.0.0
 17403:20171018:093034.736 3a1e25e000-3a1e266000 rw-p 0005e000 fd:00 15681                          /usr/lib64/libodbc.so.2.0.0
 17403:20171018:093034.736 3a1e266000-3a1e267000 rw-p 00000000 00:00 0 
 17403:20171018:093034.736 3a1e400000-3a1e4a1000 r-xp 00000000 fd:00 8251                           /usr/lib64/libnetsnmp.so.20.0.0
 17403:20171018:093034.736 3a1e4a1000-3a1e6a0000 ---p 000a1000 fd:00 8251                           /usr/lib64/libnetsnmp.so.20.0.0
 17403:20171018:093034.736 3a1e6a0000-3a1e6a3000 r--p 000a0000 fd:00 8251                           /usr/lib64/libnetsnmp.so.20.0.0
 17403:20171018:093034.736 3a1e6a3000-3a1e6a5000 rw-p 000a3000 fd:00 8251                           /usr/lib64/libnetsnmp.so.20.0.0
 17403:20171018:093034.736 3a1e6a5000-3a1e6db000 rw-p 00000000 00:00 0 
 17403:20171018:093034.736 3a1ec00000-3a1ece8000 r-xp 00000000 fd:00 10121                          /usr/lib64/libOpenIPMI.so.0.0.5
 17403:20171018:093034.736 3a1ece8000-3a1eee8000 ---p 000e8000 fd:00 10121                          /usr/lib64/libOpenIPMI.so.0.0.5
 17403:20171018:093034.736 3a1eee8000-3a1ef04000 rw-p 000e8000 fd:00 10121                          /usr/lib64/libOpenIPMI.so.0.0.5
 17403:20171018:093034.736 3a1ef04000-3a1ef08000 rw-p 00000000 00:00 0 
 17403:20171018:093034.736 3a1f000000-3a1f005000 r-xp 00000000 fd:00 10128                          /usr/lib64/libOpenIPMIposix.so.0.0.1
 17403:20171018:093034.737 3a1f005000-3a1f204000 ---p 00005000 fd:00 10128                          /usr/lib64/libOpenIPMIposix.so.0.0.1
 17403:20171018:093034.737 3a1f204000-3a1f205000 rw-p 00004000 fd:00 10128                          /usr/lib64/libOpenIPMIposix.so.0.0.1
 17403:20171018:093034.737 3a1f400000-3a1f416000 r-xp 00000000 fd:00 4728                           /lib64/libresolv-2.12.so
 17403:20171018:093034.737 3a1f416000-3a1f616000 ---p 00016000 fd:00 4728                           /lib64/libresolv-2.12.so
 17403:20171018:093034.737 3a1f616000-3a1f617000 r--p 00016000 fd:00 4728                           /lib64/libresolv-2.12.so
 17403:20171018:093034.737 3a1f617000-3a1f618000 rw-p 00017000 fd:00 4728                           /lib64/libresolv-2.12.so
 17403:20171018:093034.737 3a1f618000-3a1f61a000 rw-p 00000000 00:00 0 
 17403:20171018:093034.737 3a1f800000-3a1f816000 r-xp 00000000 fd:00 4781                           /lib64/libnsl-2.12.so
 17403:20171018:093034.737 3a1f816000-3a1fa15000 ---p 00016000 fd:00 4781                           /lib64/libnsl-2.12.so
 17403:20171018:093034.737 3a1fa15000-3a1fa16000 r--p 00015000 fd:00 4781                           /lib64/libnsl-2.12.so
 17403:20171018:093034.737 3a1fa16000-3a1fa17000 rw-p 00016000 fd:00 4781                           /lib64/libnsl-2.12.so
 17403:20171018:093034.737 3a1fa17000-3a1fa19000 rw-p 00000000 00:00 0 
 17403:20171018:093034.737 3a20000000-3a20032000 r-xp 00000000 fd:00 5117                           /lib64/libidn.so.11.6.1
 17403:20171018:093034.737 3a20032000-3a20231000 ---p 00032000 fd:00 5117                           /lib64/libidn.so.11.6.1
 17403:20171018:093034.737 3a20231000-3a20232000 rw-p 00031000 fd:00 5117                           /lib64/libidn.so.11.6.1
 17403:20171018:093034.737 3a20400000-3a20422000 r-xp 00000000 fd:00 10118                          /lib64/libncurses.so.5.7
 17403:20171018:093034.737 3a20422000-3a20621000 ---p 00022000 fd:00 10118                          /lib64/libncurses.so.5.7
 17403:20171018:093034.737 3a20621000-3a20622000 rw-p 00021000 fd:00 10118                          /lib64/libncurses.so.5.7
 17403:20171018:093034.737 3a20800000-3a20826000 r-xp 00000000 fd:00 7545                           /usr/lib64/libnssutil3.so
 17403:20171018:093034.737 3a20826000-3a20a25000 ---p 00026000 fd:00 7545                           /usr/lib64/libnssutil3.so
 17403:20171018:093034.737 3a20a25000-3a20a2c000 r--p 00025000 fd:00 7545                           /usr/lib64/libnssutil3.so
 17403:20171018:093034.737 3a20a2c000-3a20a2d000 rw-p 0002c000 fd:00 7545                           /usr/lib64/libnssutil3.so
 17403:20171018:093034.737 3a20c00000-3a20c04000 r-xp 00000000 fd:00 7539                           /lib64/libplc4.so
 17403:20171018:093034.737 3a20c04000-3a20e03000 ---p 00004000 fd:00 7539                           /lib64/libplc4.so
 17403:20171018:093034.737 3a20e03000-3a20e04000 r--p 00003000 fd:00 7539                           /lib64/libplc4.so
 17403:20171018:093034.737 3a20e04000-3a20e05000 rw-p 00004000 fd:00 7539                           /lib64/libplc4.so
 17403:20171018:093034.737 3a21000000-3a21003000 r-xp 00000000 fd:00 7544                           /lib64/libplds4.so
 17403:20171018:093034.737 3a21003000-3a21202000 ---p 00003000 fd:00 7544                           /lib64/libplds4.so
 17403:20171018:093034.737 3a21202000-3a21203000 r--p 00002000 fd:00 7544                           /lib64/libplds4.so
 17403:20171018:093034.737 3a21203000-3a21204000 rw-p 00003000 fd:00 7544                           /lib64/libplds4.so
 17403:20171018:093034.737 3a21400000-3a21407000 r-xp 00000000 fd:00 5096                           /lib64/libcrypt-2.12.so
 17403:20171018:093034.737 3a21407000-3a21607000 ---p 00007000 fd:00 5096                           /lib64/libcrypt-2.12.so
 17403:20171018:093034.738 3a21607000-3a21608000 r--p 00007000 fd:00 5096                           /lib64/libcrypt-2.12.so
 17403:20171018:093034.738 3a21608000-3a21609000 rw-p 00008000 fd:00 5096                           /lib64/libcrypt-2.12.so
 17403:20171018:093034.738 3a21609000-3a21637000 rw-p 00000000 00:00 0 
 17403:20171018:093034.738 3a21800000-3a21802000 r-xp 00000000 fd:00 5094                           /lib64/libfreebl3.so
 17403:20171018:093034.738 3a21802000-3a21a01000 ---p 00002000 fd:00 5094                           /lib64/libfreebl3.so
 17403:20171018:093034.738 3a21a01000-3a21a02000 r--p 00001000 fd:00 5094                           /lib64/libfreebl3.so
 17403:20171018:093034.738 3a21a02000-3a21a03000 rw-p 00002000 fd:00 5094                           /lib64/libfreebl3.so
 17403:20171018:093034.738 3a22000000-3a22028000 r-xp 00000000 fd:00 11287                          /usr/lib64/libsmime3.so
 17403:20171018:093034.738 3a22028000-3a22227000 ---p 00028000 fd:00 11287                          /usr/lib64/libsmime3.so
 17403:20171018:093034.738 3a22227000-3a2222b000 r--p 00027000 fd:00 11287                          /usr/lib64/libsmime3.so
 17403:20171018:093034.738 3a2222b000-3a2222c000 rw-p 0002b000 fd:00 11287                          /usr/lib64/libsmime3.so
 17403:20171018:093034.738 3a22800000-3a22847000 r-xp 00000000 fd:00 11134                          /usr/lib64/libssl3.so
 17403:20171018:093034.738 3a22847000-3a22a47000 ---p 00047000 fd:00 11134                          /usr/lib64/libssl3.so
 17403:20171018:093034.738 3a22a47000-3a22a4b000 r--p 00047000 fd:00 11134                          /usr/lib64/libssl3.so
 17403:20171018:093034.738 3a22a4b000-3a22a4c000 rw-p 0004b000 fd:00 11134                          /usr/lib64/libssl3.so
 17403:20171018:093034.738 3a22a4c000-3a22a4d000 rw-p 00000000 00:00 0 
 17403:20171018:093034.738 3a23800000-3a23949000 r-xp 00000000 fd:00 7286                           /usr/lib64/libxml2.so.2.7.6
 17403:20171018:093034.738 3a23949000-3a23b48000 ---p 00149000 fd:00 7286                           /usr/lib64/libxml2.so.2.7.6
 17403:20171018:093034.738 3a23b48000-3a23b52000 rw-p 00148000 fd:00 7286                           /usr/lib64/libxml2.so.2.7.6
 17403:20171018:093034.738 3a23b52000-3a23b53000 rw-p 00000000 00:00 0 
 17403:20171018:093034.738 3a23c00000-3a23dba000 r-xp 00000000 fd:00 8062                           /usr/lib64/libcrypto.so.1.0.1e
 17403:20171018:093034.738 3a23dba000-3a23fba000 ---p 001ba000 fd:00 8062                           /usr/lib64/libcrypto.so.1.0.1e
 17403:20171018:093034.738 3a23fba000-3a23fd5000 r--p 001ba000 fd:00 8062                           /usr/lib64/libcrypto.so.1.0.1e
 17403:20171018:093034.738 3a23fd5000-3a23fe1000 rw-p 001d5000 fd:00 8062                           /usr/lib64/libcrypto.so.1.0.1e
 17403:20171018:093034.738 3a23fe1000-3a23fe5000 rw-p 00000000 00:00 0 
 17403:20171018:093034.738 3a24000000-3a24019000 r-xp 00000000 fd:00 11346                          /usr/lib64/libsasl2.so.2.0.23
 17403:20171018:093034.738 3a24019000-3a24218000 ---p 00019000 fd:00 11346                          /usr/lib64/libsasl2.so.2.0.23
 17403:20171018:093034.738 3a24218000-3a24219000 r--p 00018000 fd:00 11346                          /usr/lib64/libsasl2.so.2.0.23
 17403:20171018:093034.738 3a24219000-3a2421a000 rw-p 00019000 fd:00 11346                          /usr/lib64/libsasl2.so.2.0.23
 17403:20171018:093034.738 3a24800000-3a24803000 r-xp 00000000 fd:00 11486                          /lib64/libcom_err.so.2.1
 17403:20171018:093034.739 3a24803000-3a24a02000 ---p 00003000 fd:00 11486                          /lib64/libcom_err.so.2.1
 17403:20171018:093034.739 3a24a02000-3a24a03000 r--p 00002000 fd:00 11486                          /lib64/libcom_err.so.2.1
 17403:20171018:093034.739 3a24a03000-3a24a04000 rw-p 00003000 fd:00 11486                          /lib64/libcom_err.so.2.1
 17403:20171018:093034.739 3a25400000-3a2544e000 r-xp 00000000 fd:00 11467                          /lib64/libldap-2.4.so.2.10.3
 17403:20171018:093034.739 3a2544e000-3a2564d000 ---p 0004e000 fd:00 11467                          /lib64/libldap-2.4.so.2.10.3
 17403:20171018:093034.739 3a2564d000-3a2564f000 r--p 0004d000 fd:00 11467                          /lib64/libldap-2.4.so.2.10.3
 17403:20171018:093034.739 3a2564f000-3a25651000 rw-p 0004f000 fd:00 11467                          /lib64/libldap-2.4.so.2.10.3
 17403:20171018:093034.739 3a25800000-3a25802000 r-xp 00000000 fd:00 11471                          /lib64/libkeyutils.so.1.3
 17403:20171018:093034.739 3a25802000-3a25a01000 ---p 00002000 fd:00 11471                          /lib64/libkeyutils.so.1.3
 17403:20171018:093034.739 3a25a01000-3a25a02000 r--p 00001000 fd:00 11471                          /lib64/libkeyutils.so.1.3
 17403:20171018:093034.739 3a25a02000-3a25a03000 rw-p 00002000 fd:00 11471                          /lib64/libkeyutils.so.1.3
 17403:20171018:093034.739 3a25c00000-3a25c0e000 r-xp 00000000 fd:00 11332                          /lib64/liblber-2.4.so.2.10.3
 17403:20171018:093034.739 3a25c0e000-3a25e0d000 ---p 0000e000 fd:00 11332                          /lib64/liblber-2.4.so.2.10.3
 17403:20171018:093034.739 3a25e0d000-3a25e0e000 r--p 0000d000 fd:00 11332                          /lib64/liblber-2.4.so.2.10.3
 17403:20171018:093034.739 3a25e0e000-3a25e0f000 rw-p 0000e000 fd:00 11332                          /lib64/liblber-2.4.so.2.10.3
 17403:20171018:093034.739 3a26800000-3a2680a000 r-xp 00000000 fd:00 11474                          /lib64/libkrb5support.so.0.1
 17403:20171018:093034.739 3a2680a000-3a26a09000 ---p 0000a000 fd:00 11474                          /lib64/libkrb5support.so.0.1
 17403:20171018:093034.739 3a26a09000-3a26a0a000 r--p 00009000 fd:00 11474                          /lib64/libkrb5support.so.0.1
 17403:20171018:093034.739 3a26a0a000-3a26a0b000 rw-p 0000a000 fd:00 11474                          /lib64/libkrb5support.so.0.1
 17403:20171018:093034.739 3a26c00000-3a26cdc000 r-xp 00000000 fd:00 11512                          /lib64/libkrb5.so.3.3
 17403:20171018:093034.739 3a26cdc000-3a26edb000 ---p 000dc000 fd:00 11512                          /lib64/libkrb5.so.3.3
 17403:20171018:093034.739 3a26edb000-3a26ee5000 r--p 000db000 fd:00 11512                          /lib64/libkrb5.so.3.3
 17403:20171018:093034.739 3a26ee5000-3a26ee7000 rw-p 000e5000 fd:00 11512                          /lib64/libkrb5.so.3.3
 17403:20171018:093034.739 3a27400000-3a27429000 r-xp 00000000 fd:00 11475                          /lib64/libk5crypto.so.3.1
 17403:20171018:093034.739 3a27429000-3a27629000 ---p 00029000 fd:00 11475                          /lib64/libk5crypto.so.3.1
 17403:20171018:093034.739 3a27629000-3a2762a000 r--p 00029000 fd:00 11475                          /lib64/libk5crypto.so.3.1
 17403:20171018:093034.739 3a2762a000-3a2762b000 rw-p 0002a000 fd:00 11475                          /lib64/libk5crypto.so.3.1
 17403:20171018:093034.739 3a2762b000-3a2762c000 rw-p 00000000 00:00 0 
 17403:20171018:093034.739 3a28400000-3a28441000 r-xp 00000000 fd:00 11518                          /lib64/libgssapi_krb5.so.2.2
 17403:20171018:093034.739 3a28441000-3a28641000 ---p 00041000 fd:00 11518                          /lib64/libgssapi_krb5.so.2.2
 17403:20171018:093034.740 3a28641000-3a28642000 r--p 00041000 fd:00 11518                          /lib64/libgssapi_krb5.so.2.2
 17403:20171018:093034.740 3a28642000-3a28644000 rw-p 00042000 fd:00 11518                          /lib64/libgssapi_krb5.so.2.2
 17403:20171018:093034.740 3a29400000-3a29462000 r-xp 00000000 fd:00 14408                          /usr/lib64/libssl.so.1.0.1e
 17403:20171018:093034.740 3a29462000-3a29662000 ---p 00062000 fd:00 14408                          /usr/lib64/libssl.so.1.0.1e
 17403:20171018:093034.740 3a29662000-3a29666000 r--p 00062000 fd:00 14408                          /usr/lib64/libssl.so.1.0.1e
 17403:20171018:093034.740 3a29666000-3a2966c000 rw-p 00066000 fd:00 14408                          /usr/lib64/libssl.so.1.0.1e
 17403:20171018:093034.740 3a29800000-3a29827000 r-xp 00000000 fd:00 2662                           /usr/lib64/libssh2.so.1.0.1
 17403:20171018:093034.740 3a29827000-3a29a26000 ---p 00027000 fd:00 2662                           /usr/lib64/libssh2.so.1.0.1
 17403:20171018:093034.740 3a29a26000-3a29a28000 rw-p 00026000 fd:00 2662                           /usr/lib64/libssh2.so.1.0.1
 17403:20171018:093034.740 3a29c00000-3a29c53000 r-xp 00000000 fd:00 22192                          /usr/lib64/libcurl.so.4.1.1
 17403:20171018:093034.740 3a29c53000-3a29e52000 ---p 00053000 fd:00 22192                          /usr/lib64/libcurl.so.4.1.1
 17403:20171018:093034.740 3a29e52000-3a29e55000 rw-p 00052000 fd:00 22192                          /usr/lib64/libcurl.so.4.1.1
 17403:20171018:093034.740 3fa1a00000-3fa1a01000 r-xp 00000000 fd:00 20966                          /lib64/libaio.so.1.0.1
 17403:20171018:093034.740 3fa1a01000-3fa1c00000 ---p 00001000 fd:00 20966                          /lib64/libaio.so.1.0.1
 17403:20171018:093034.740 3fa1c00000-3fa1c01000 rw-p 00000000 fd:00 20966                          /lib64/libaio.so.1.0.1
 17403:20171018:093034.740 7f776c6ca000-7f776c6e0000 r-xp 00000000 fd:00 4488                       /lib64/libgcc_s-4.4.7-20120601.so.1
 17403:20171018:093034.740 7f776c6e0000-7f776c8df000 ---p 00016000 fd:00 4488                       /lib64/libgcc_s-4.4.7-20120601.so.1
 17403:20171018:093034.740 7f776c8df000-7f776c8e0000 rw-p 00015000 fd:00 4488                       /lib64/libgcc_s-4.4.7-20120601.so.1
 17403:20171018:093034.740 7f776c8e0000-7f776c934000 r-xp 00000000 fd:00 30210                      /lib64/libldap_r-2.4.so.2.10.3
 17403:20171018:093034.740 7f776c934000-7f776cb34000 ---p 00054000 fd:00 30210                      /lib64/libldap_r-2.4.so.2.10.3
 17403:20171018:093034.740 7f776cb34000-7f776cb36000 r--p 00054000 fd:00 30210                      /lib64/libldap_r-2.4.so.2.10.3
 17403:20171018:093034.740 7f776cb36000-7f776cb38000 rw-p 00056000 fd:00 30210                      /lib64/libldap_r-2.4.so.2.10.3
 17403:20171018:093034.740 7f776cb38000-7f776cb3a000 rw-p 00000000 00:00 0 
 17403:20171018:093034.740 7f776cb3a000-7f776cb60000 r-xp 00000000 fd:00 7104                       /usr/lib64/libpq.so.5.2
 17403:20171018:093034.740 7f776cb60000-7f776cd5f000 ---p 00026000 fd:00 7104                       /usr/lib64/libpq.so.5.2
 17403:20171018:093034.740 7f776cd5f000-7f776cd62000 rw-p 00025000 fd:00 7104                       /usr/lib64/libpq.so.5.2
 17403:20171018:093034.740 7f776cd62000-7f776cddf000 r-xp 00000000 fd:00 16420                      /usr/lib64/psqlodbcw.so
 17403:20171018:093034.740 7f776cddf000-7f776cfde000 ---p 0007d000 fd:00 16420                      /usr/lib64/psqlodbcw.so
 17403:20171018:093034.740 7f776cfde000-7f776cfe1000 rw-p 0007c000 fd:00 16420                      /usr/lib64/psqlodbcw.so
 17403:20171018:093034.740 7f776cfe1000-7f776cfe2000 rw-p 00000000 00:00 0 
 17403:20171018:093034.740 7f776cfe2000-7f776d04d000 r-xp 00000000 fd:00 4449                       /usr/local/lib/libtdsodbc.so.0.0.0
 17403:20171018:093034.740 7f776d04d000-7f776d24d000 ---p 0006b000 fd:00 4449                       /usr/local/lib/libtdsodbc.so.0.0.0
 17403:20171018:093034.740 7f776d24d000-7f776d251000 rw-p 0006b000 fd:00 4449                       /usr/local/lib/libtdsodbc.so.0.0.0
 17403:20171018:093034.740 7f776d251000-7f776d277000 r-xp 00000000 fd:00 5040                       /usr/lib64/libnssdbm3.so
 17403:20171018:093034.741 7f776d277000-7f776d477000 ---p 00026000 fd:00 5040                       /usr/lib64/libnssdbm3.so
 17403:20171018:093034.741 7f776d477000-7f776d478000 r--p 00026000 fd:00 5040                       /usr/lib64/libnssdbm3.so
 17403:20171018:093034.741 7f776d478000-7f776d479000 rw-p 00027000 fd:00 5040                       /usr/lib64/libnssdbm3.so
 17403:20171018:093034.741 7f776d479000-7f776d49d000 r-xp 00000000 fd:00 30201                      /usr/lib64/libnsspem.so
 17403:20171018:093034.741 7f776d49d000-7f776d69d000 ---p 00024000 fd:00 30201                      /usr/lib64/libnsspem.so
 17403:20171018:093034.741 7f776d69d000-7f776d69f000 r--p 00024000 fd:00 30201                      /usr/lib64/libnsspem.so
 17403:20171018:093034.741 7f776d69f000-7f776d6a0000 rw-p 00026000 fd:00 30201                      /usr/lib64/libnsspem.so
 17403:20171018:093034.741 7f776d6a0000-7f776d6a2000 r-xp 00000000 fd:00 30199                      /usr/lib64/libnsssysinit.so
 17403:20171018:093034.741 7f776d6a2000-7f776d8a1000 ---p 00002000 fd:00 30199                      /usr/lib64/libnsssysinit.so
 17403:20171018:093034.741 7f776d8a1000-7f776d8a2000 r--p 00001000 fd:00 30199                      /usr/lib64/libnsssysinit.so
 17403:20171018:093034.741 7f776d8a2000-7f776d8a3000 rw-p 00002000 fd:00 30199                      /usr/lib64/libnsssysinit.so
 17403:20171018:093034.741 7f776d8a3000-7f776d915000 r-xp 00000000 fd:00 3192                       /lib64/libfreeblpriv3.so
 17403:20171018:093034.741 7f776d915000-7f776db15000 ---p 00072000 fd:00 3192                       /lib64/libfreeblpriv3.so
 17403:20171018:093034.741 7f776db15000-7f776db17000 r--p 00072000 fd:00 3192                       /lib64/libfreeblpriv3.so
 17403:20171018:093034.741 7f776db17000-7f776db18000 rw-p 00074000 fd:00 3192                       /lib64/libfreeblpriv3.so
 17403:20171018:093034.741 7f776db18000-7f776db1c000 rw-p 00000000 00:00 0 
 17403:20171018:093034.741 7f776db1c000-7f776dba7000 r-xp 00000000 fd:00 7152                       /usr/lib64/libsqlite3.so.0.8.6
 17403:20171018:093034.741 7f776dba7000-7f776dda7000 ---p 0008b000 fd:00 7152                       /usr/lib64/libsqlite3.so.0.8.6
 17403:20171018:093034.741 7f776dda7000-7f776ddaa000 rw-p 0008b000 fd:00 7152                       /usr/lib64/libsqlite3.so.0.8.6
 17403:20171018:093034.741 7f776ddaa000-7f776ddab000 rw-p 00000000 00:00 0 
 17403:20171018:093034.741 7f776ddab000-7f776dde9000 r-xp 00000000 fd:00 27381                      /usr/lib64/libsoftokn3.so
 17403:20171018:093034.741 7f776dde9000-7f776dfe8000 ---p 0003e000 fd:00 27381                      /usr/lib64/libsoftokn3.so
 17403:20171018:093034.741 7f776dfe8000-7f776dfea000 r--p 0003d000 fd:00 27381                      /usr/lib64/libsoftokn3.so
 17403:20171018:093034.741 7f776dfea000-7f776dfeb000 rw-p 0003f000 fd:00 27381                      /usr/lib64/libsoftokn3.so
 17403:20171018:093034.741 7f776dfeb000-7f776dff0000 r-xp 00000000 fd:00 1245                       /lib64/libnss_dns-2.12.so
 17403:20171018:093034.741 7f776dff0000-7f776e1ef000 ---p 00005000 fd:00 1245                       /lib64/libnss_dns-2.12.so
 17403:20171018:093034.741 7f776e1ef000-7f776e1f0000 r--p 00004000 fd:00 1245                       /lib64/libnss_dns-2.12.so
 17403:20171018:093034.741 7f776e1f0000-7f776e1f1000 rw-p 00005000 fd:00 1245                       /lib64/libnss_dns-2.12.so
 17403:20171018:093034.742 7f776e1f1000-7f776e201000 r-xp 00000000 fd:00 8261                       /usr/lib64/libodbcinst.so.2.0.0
 17403:20171018:093034.742 7f776e201000-7f776e401000 ---p 00010000 fd:00 8261                       /usr/lib64/libodbcinst.so.2.0.0
 17403:20171018:093034.742 7f776e401000-7f776e402000 rw-p 00010000 fd:00 8261                       /usr/lib64/libodbcinst.so.2.0.0
 17403:20171018:093034.742 7f776e402000-7f776e64b000 r-xp 00000000 fd:03 661608                     /app/oracle/12.1/lib/libsqora.so.12.1
 17403:20171018:093034.742 7f776e64b000-7f776e84b000 ---p 00249000 fd:03 661608                     /app/oracle/12.1/lib/libsqora.so.12.1
 17403:20171018:093034.742 7f776e84b000-7f776e870000 rw-p 00249000 fd:03 661608                     /app/oracle/12.1/lib/libsqora.so.12.1
 17403:20171018:093034.742 7f776e870000-7f776e873000 rw-p 00000000 00:00 0 
 17403:20171018:093034.742 7f776e873000-7f776e874000 r-xp 00000000 fd:00 3931                       /usr/lib64/gconv/ISO8859-1.so
 17403:20171018:093034.742 7f776e874000-7f776ea74000 ---p 00001000 fd:00 3931                       /usr/lib64/gconv/ISO8859-1.so
 17403:20171018:093034.742 7f776ea74000-7f776ea75000 r--p 00001000 fd:00 3931                       /usr/lib64/gconv/ISO8859-1.so
 17403:20171018:093034.742 7f776ea75000-7f776ea76000 rw-p 00002000 fd:00 3931                       /usr/lib64/gconv/ISO8859-1.so
 17403:20171018:093034.742 7f776ea76000-7f776ea7d000 r--s 00000000 fd:00 7357                       /usr/lib64/gconv/gconv-modules.cache
 17403:20171018:093034.742 7f776ea7d000-7f776ea7e000 rw-p 00000000 00:00 0 
 17403:20171018:093034.742 7f776ea7e000-7f7774913000 r--p 00000000 fd:00 27318                      /usr/lib/locale/locale-archive
 17403:20171018:093034.742 7f7774913000-7f7774923000 r-xp 00000000 fd:03 661406                     /app/oracle/12.1/lib/libnque12.so
 17403:20171018:093034.742 7f7774923000-7f7774b23000 ---p 00010000 fd:03 661406                     /app/oracle/12.1/lib/libnque12.so
 17403:20171018:093034.742 7f7774b23000-7f7774b24000 rw-p 00010000 fd:03 661406                     /app/oracle/12.1/lib/libnque12.so
 17403:20171018:093034.742 7f7774b24000-7f7774b25000 rw-p 00000000 00:00 0 
 17403:20171018:093034.742 7f7774b25000-7f7774b2f000 r-xp 00000000 fd:00 4569                       /usr/lib64/libnuma.so.1
 17403:20171018:093034.742 7f7774b2f000-7f7774d2f000 ---p 0000a000 fd:00 4569                       /usr/lib64/libnuma.so.1
 17403:20171018:093034.742 7f7774d2f000-7f7774d30000 rw-p 0000a000 fd:00 4569                       /usr/lib64/libnuma.so.1
 17403:20171018:093034.742 7f7774d30000-7f7774def000 r--p 00000000 fd:03 660874                     /app/oracle/12.1/oracore/zoneinfo/timezlrg_18.dat
 17403:20171018:093034.742 7f7774def000-7f7774e12000 rw-p 00000000 00:00 0 
 17403:20171018:093034.742 7f7774e12000-7f7779212000 rw-s 00000000 00:04 1839693860                 /SYSV760082cd (deleted)
 17403:20171018:093034.742 7f7779212000-7f7779221000 rw-s 00000000 00:04 1839661091                 /SYSV530082cd (deleted)
 17403:20171018:093034.742 7f7779221000-7f777ad21000 rw-s 00000000 00:04 1839628322                 /SYSV730082cd (deleted)
 17403:20171018:093034.743 7f777ad21000-7f7784621000 rw-s 00000000 00:04 1839595553                 /SYSV670082cd (deleted)
 17403:20171018:093034.743 7f7784621000-7f7788a21000 rw-s 00000000 00:04 1839562784                 /SYSV740082cd (deleted)
 17403:20171018:093034.743 7f7788a21000-7f778be21000 rw-s 00000000 00:04 1839530015                 /SYSV480082cd (deleted)
 17403:20171018:093034.743 7f778be21000-7f778f221000 rw-s 00000000 00:04 1839497236                 /SYSV680082cd (deleted)
 17403:20171018:093034.743 7f778f221000-7f778f222000 rw-p 00000000 00:00 0 
 17403:20171018:093034.743 7f778f222000-7f778fc0f000 r--s 00000000 fd:04 131129                     /var/lib/sss/mc/initgroups
 17403:20171018:093034.743 7f778fc0f000-7f778fc17000 r-xp 00000000 fd:00 7957                       /lib64/libnss_sss.so.2
 17403:20171018:093034.743 7f778fc17000-7f778fe16000 ---p 00008000 fd:00 7957                       /lib64/libnss_sss.so.2
 17403:20171018:093034.743 7f778fe16000-7f778fe17000 rw-p 00007000 fd:00 7957                       /lib64/libnss_sss.so.2
 17403:20171018:093034.743 7f778fe17000-7f778fe24000 r-xp 00000000 fd:00 27296                      /lib64/libnss_files-2.12.so
 17403:20171018:093034.743 7f778fe24000-7f7790023000 ---p 0000d000 fd:00 27296                      /lib64/libnss_files-2.12.so
 17403:20171018:093034.743 7f7790023000-7f7790024000 r--p 0000c000 fd:00 27296                      /lib64/libnss_files-2.12.so
 17403:20171018:093034.743 7f7790024000-7f7790025000 rw-p 0000d000 fd:00 27296                      /lib64/libnss_files-2.12.so
 17403:20171018:093034.743 7f7790025000-7f779002f000 rw-p 00000000 00:00 0 
 17403:20171018:093034.743 7f779002f000-7f779004c000 r-xp 00000000 fd:00 4767                       /lib64/libselinux.so.1
 17403:20171018:093034.743 7f779004c000-7f779024b000 ---p 0001d000 fd:00 4767                       /lib64/libselinux.so.1
 17403:20171018:093034.743 7f779024b000-7f779024c000 r--p 0001c000 fd:00 4767                       /lib64/libselinux.so.1
 17403:20171018:093034.744 7f779024c000-7f779024d000 rw-p 0001d000 fd:00 4767                       /lib64/libselinux.so.1
 17403:20171018:093034.744 7f779024d000-7f7790250000 rw-p 00000000 00:00 0 
 17403:20171018:093034.744 7f7790250000-7f779026d000 r-xp 00000000 fd:00 4882                       /lib64/libtinfo.so.5.7
 17403:20171018:093034.744 7f779026d000-7f779046c000 ---p 0001d000 fd:00 4882                       /lib64/libtinfo.so.5.7
 17403:20171018:093034.744 7f779046c000-7f7790470000 rw-p 0001c000 fd:00 4882                       /lib64/libtinfo.so.5.7
 17403:20171018:093034.744 7f7790470000-7f7790473000 rw-p 00000000 00:00 0 
 17403:20171018:093034.744 7f7790473000-7f77904ac000 r-xp 00000000 fd:00 7537                       /lib64/libnspr4.so
 17403:20171018:093034.744 7f77904ac000-7f77906ac000 ---p 00039000 fd:00 7537                       /lib64/libnspr4.so
 17403:20171018:093034.744 7f77906ac000-7f77906ad000 r--p 00039000 fd:00 7537                       /lib64/libnspr4.so
 17403:20171018:093034.744 7f77906ad000-7f77906af000 rw-p 0003a000 fd:00 7537                       /lib64/libnspr4.so
 17403:20171018:093034.744 7f77906af000-7f77906b3000 rw-p 00000000 00:00 0 
 17403:20171018:093034.744 7f77906b3000-7f77907ed000 r-xp 00000000 fd:00 7549                       /usr/lib64/libnss3.so
 17403:20171018:093034.744 7f77907ed000-7f77909ec000 ---p 0013a000 fd:00 7549                       /usr/lib64/libnss3.so
 17403:20171018:093034.744 7f77909ec000-7f77909f2000 r--p 00139000 fd:00 7549                       /usr/lib64/libnss3.so
 17403:20171018:093034.744 7f77909f2000-7f77909f4000 rw-p 0013f000 fd:00 7549                       /usr/lib64/libnss3.so
 17403:20171018:093034.744 7f77909f4000-7f77909f7000 rw-p 00000000 00:00 0 
 17403:20171018:093034.744 7f77909f7000-7f77909fd000 r-xp 00000000 fd:00 10124                      /usr/lib64/libgdbm.so.2.0.0
 17403:20171018:093034.744 7f77909fd000-7f7790bfc000 ---p 00006000 fd:00 10124                      /usr/lib64/libgdbm.so.2.0.0
 17403:20171018:093034.744 7f7790bfc000-7f7790bfd000 rw-p 00005000 fd:00 10124                      /usr/lib64/libgdbm.so.2.0.0
 17403:20171018:093034.744 7f7790bfd000-7f7790bfe000 rw-p 00000000 00:00 0 
 17403:20171018:093034.744 7f7790bfe000-7f7790c05000 r-xp 00000000 fd:00 10119                      /usr/lib64/libOpenIPMIutils.so.0.0.1
 17403:20171018:093034.744 7f7790c05000-7f7790e05000 ---p 00007000 fd:00 10119                      /usr/lib64/libOpenIPMIutils.so.0.0.1
 17403:20171018:093034.744 7f7790e05000-7f7790e06000 rw-p 00007000 fd:00 10119                      /usr/lib64/libOpenIPMIutils.so.0.0.1
 17403:20171018:093034.744 7f7790e06000-7f7790e07000 rw-p 00000000 00:00 0 
 17403:20171018:093034.744 7f7790e07000-7f7790e10000 r-xp 00000000 fd:00 5283                       /usr/lib64/libltdl.so.7.2.1
 17403:20171018:093034.744 7f7790e10000-7f779100f000 ---p 00009000 fd:00 5283                       /usr/lib64/libltdl.so.7.2.1
 17403:20171018:093034.744 7f779100f000-7f7791010000 rw-p 00008000 fd:00 5283                       /usr/lib64/libltdl.so.7.2.1
 17403:20171018:093034.744 7f7791010000-7f7791364000 r-xp 00000000 fd:03 660761                     /app/oracle/12.1/lib/libclntshcore.so.12.1
 17403:20171018:093034.744 7f7791364000-7f7791563000 ---p 00354000 fd:03 660761                     /app/oracle/12.1/lib/libclntshcore.so.12.1
 17403:20171018:093034.744 7f7791563000-7f779157e000 rw-p 00353000 fd:03 660761                     /app/oracle/12.1/lib/libclntshcore.so.12.1
 17403:20171018:093034.744 7f779157e000-7f7791584000 rw-p 00000000 00:00 0 
 17403:20171018:093034.744 7f7791584000-7f77915c8000 r-xp 00000000 fd:03 661376                     /app/oracle/12.1/lib/libons.so
 17403:20171018:093034.745 7f77915c8000-7f77917c7000 ---p 00044000 fd:03 661376                     /app/oracle/12.1/lib/libons.so
 17403:20171018:093034.745 7f77917c7000-7f77917c9000 rw-p 00043000 fd:03 661376                     /app/oracle/12.1/lib/libons.so
 17403:20171018:093034.745 7f77917c9000-7f779193f000 r-xp 00000000 fd:03 659437                     /app/oracle/12.1/lib/libipc1.so
 17403:20171018:093034.745 7f779193f000-7f7791b3f000 ---p 00176000 fd:03 659437                     /app/oracle/12.1/lib/libipc1.so
 17403:20171018:093034.745 7f7791b3f000-7f7791b42000 rw-p 00176000 fd:03 659437                     /app/oracle/12.1/lib/libipc1.so
 17403:20171018:093034.745 7f7791b42000-7f7791b47000 rw-p 00000000 00:00 0 
 17403:20171018:093034.745 7f7791b47000-7f7791bb9000 r-xp 00000000 fd:03 659435                     /app/oracle/12.1/lib/libmql1.so
 17403:20171018:093034.745 7f7791bb9000-7f7791db8000 ---p 00072000 fd:03 659435                     /app/oracle/12.1/lib/libmql1.so
 17403:20171018:093034.745 7f7791db8000-7f7791dbb000 rw-p 00071000 fd:03 659435                     /app/oracle/12.1/lib/libmql1.so
 17403:20171018:093034.745 7f7791dbb000-7f7791dbe000 rw-p 00000000 00:00 0 
 17403:20171018:093034.745 7f7791dbe000-7f7791dc5000 r-xp 00000000 fd:00 4716                       /lib64/librt-2.12.so
 17403:20171018:093034.745 7f7791dc5000-7f7791fc4000 ---p 00007000 fd:00 4716                       /lib64/librt-2.12.so
 17403:20171018:093034.745 7f7791fc4000-7f7791fc5000 r--p 00006000 fd:00 4716                       /lib64/librt-2.12.so
 17403:20171018:093034.745 7f7791fc5000-7f7791fc6000 rw-p 00007000 fd:00 4716                       /lib64/librt-2.12.so
 17403:20171018:093034.745 7f7791fc6000-7f7791fca000 rw-p 00000000 00:00 0 
 17403:20171018:093034.745 7f7791fca000-7f779248a000 r-xp 00000000 fd:03 661431                     /app/oracle/12.1/lib/libnnz12.so
 17403:20171018:093034.745 7f779248a000-7f779268a000 ---p 004c0000 fd:03 661431                     /app/oracle/12.1/lib/libnnz12.so
 17403:20171018:093034.745 7f779268a000-7f77926ed000 rw-p 004c0000 fd:03 661431                     /app/oracle/12.1/lib/libnnz12.so
 17403:20171018:093034.745 7f77926ed000-7f77926ef000 rw-p 00000000 00:00 0 
 17403:20171018:093034.745 7f77926ef000-7f77952ae000 r-xp 00000000 fd:03 662176                     /app/oracle/12.1/lib/libclntsh.so.12.1
 17403:20171018:093034.745 7f77952ae000-7f77954ae000 ---p 02bbf000 fd:03 662176                     /app/oracle/12.1/lib/libclntsh.so.12.1
 17403:20171018:093034.745 7f77954ae000-7f7795690000 rw-p 02bbf000 fd:03 662176                     /app/oracle/12.1/lib/libclntsh.so.12.1
 17403:20171018:093034.745 7f7795690000-7f77956b1000 rw-p 00000000 00:00 0 
 17403:20171018:093034.745 7ffc28164000-7ffc28345000 rwxp 00000000 00:00 0                          [stack]
 17403:20171018:093034.745 7ffc28345000-7ffc28346000 rw-p 00000000 00:00 0 
 17403:20171018:093034.745 7ffc28380000-7ffc28381000 r-xp 00000000 00:00 0                          [vdso]
 17403:20171018:093034.745 ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0                  [vsyscall]
 17403:20171018:093034.745 ================================
 17403:20171018:093034.745 Please consider attaching a disassembly listing to your bug report.
 17403:20171018:093034.745 This listing can be produced with, e.g., objdump -DSswx zabbix_server.
 17403:20171018:093034.746 ================================
 17326:20171018:093035.264 One child process died (PID:17403,exitcode/signal:1). Exiting ...
 17326:20171018:093037.422 syncing history data...
 17326:20171018:093037.463 syncing history data done
 17326:20171018:093037.463 syncing trend data...
 17326:20171018:093044.900 syncing trend data done
 17326:20171018:093044.902 Zabbix Server stopped. Zabbix 3.2.8 (revision 72884).

It happens randomly, cannot be predicted.

Objdump for 3.2.8 is attached.



 Comments   
Comment by Glebs Ivanovskis (Inactive) [ 2017 Oct 19 ]

I actually see three different crashes:

  • NULL dereference with suspiciously short backtrace;
  • double free in ODBC check;
  • segmentation fault with non-NULL pointer.

For that last case we have objdump and according to my analysis it shows that Zabbix crashes in activate_host() function trying to dereference mysteriously broken item:

static void	activate_host(DC_ITEM *item, zbx_timespec_t *ts)
{
	const char		*__function_name = "activate_host";
	zbx_host_availability_t	in, out;
	unsigned char		agent_type;

	zabbix_log(LOG_LEVEL_DEBUG, "In %s() hostid:" ZBX_FS_UI64 " itemid:" ZBX_FS_UI64 " type:%d",
			__function_name, item->host.hostid, item->itemid, (int)item->type);

Relevant piece of disassembly listing with annotations:

  42b507:	ba a8 62 4e 00       	mov    $0x4e62a8,%edx					<--- "activate_host"
  42b50c:	be b8 62 4e 00       	mov    $0x4e62b8,%esi					<--- "In %s() hostid:%lu itemid:%lu type:%d"
  42b511:	bf 04 00 00 00       	mov    $0x4,%edi
  42b516:	49 8d 1c 1f          	lea    (%r15,%rbx,1),%rbx
  42b51a:	44 0f b6 8b 70 22 00 00 	movzbl 0x2270(%rbx),%r9d			<--- item->type		<--- crashing here while trying to dereference item pointer
  42b522:	4c 8b 83 58 22 00 00 	mov    0x2258(%rbx),%r8					<--- item->itemid
  42b529:	48 8b 0b             	mov    (%rbx),%rcx					<--- item->host.hostid
  42b52c:	e8 cf 8a 02 00       	callq  454000 <__zbx_zabbix_log>			<--- zabbix_log(LOG_LEVEL_DEBUG, "In %s() hostid:" ZBX_FS_UI64 " itemid:" ZBX_FS_UI64 " type:%d",
  42b531:	48 8b 33             	mov    (%rbx),%rsi							__function_name, item->host.hostid, item->itemid, (int)item->type);

Offsets coincide with DC_ITEM structure for compilation on 64-bit platform without TLS support.

Since crashes are rare, unpredictable and symptoms vary greatly I'd like you to rule out any hardware issue before I go any further.

Comment by Oleksii Zagorskyi [ 2017 Oct 19 ]

ODBC is heavily used:

# grep Driver odbc.ini |sort|uniq -c
     31 Driver      = FreeTDS
     56 Driver      = Oracle12g
      2 Driver = PostgreSQL

No patches.

Objdump for recompiled 3.2.4 is attached. Used configure:

configure --with-oracle --with-oracle-include=/app/oracle/12.1/ --with-oracle-lib=/app/oracle/12.1/lib --with-net-snmp --enable-server --enable-agent --with-libxml2 --with-libcurl --with-unixodbc --with-ssh2 --with-ldap --with-openipmi --enable-java 

glebs.ivanovskis Thank you! Additional thanks for debug info compiled in. Unfortunately, objdump does not match backtraces.

From backtrace:

/usr/local/sbin/zabbix_server: poller #27 [got 7 values in 0.076096 sec, getting values](get_value_db+0x94)[0x435734]

From objdump:

0000000000433d70 <get_value_db>:

433d70‍16 + 94‍16 = 433e04‍16 ≠ 435734‍16

Nevermind, I wasn't hoping to get much from it. Seems that Zabbix crashes because of memory corruption and memory corruption happens long before crash, so knowing exact place of crash does not help much...

Comment by Oleksii Zagorskyi [ 2017 Oct 19 ]

It just happened again:

 24645:20171019:145505.763 ====== Fatal information: ======
 24645:20171019:145505.763 Program counter: 0x44cd58
 24645:20171019:145505.763 === Registers: ===
 24645:20171019:145505.763 r8      =              440 =                 1088 =                 1088
 24645:20171019:145505.763 r9      =                2 =                    2 =                    2
 24645:20171019:145505.763 r10     =               c0 =                  192 =                  192
 24645:20171019:145505.763 r11     = d1bed080d1bed000 = 15113746651195297792 = -3332997422514253824
 24645:20171019:145505.763 r12     =                0 =                    0 =                    0
 24645:20171019:145505.763 r13     =                2 =                    2 =                    2
 24645:20171019:145505.763 r14     =     7ffcfba557b4 =      140724530403252 =      140724530403252
 24645:20171019:145505.763 r15     =     7ffcfb8e23a0 =      140724528882592 =      140724528882592
 24645:20171019:145505.763 rdi     =          13b21da =             20652506 =             20652506
 24645:20171019:145505.763 rsi     =          13b21da =             20652506 =             20652506
 24645:20171019:145505.763 rbp     =          13b1990 =             20650384 =             20650384
 24645:20171019:145505.763 rbx     =     7ffcfb002fa0 =      140724519579552 =      140724519579552
 24645:20171019:145505.763 rdx     =                2 =                    2 =                    2
 24645:20171019:145505.763 rax     =                0 =                    0 =                    0
 24645:20171019:145505.763 rcx     =                0 =                    0 =                    0
 24645:20171019:145505.763 rsp     =     7ffcfb8e2220 =      140724528882208 =      140724528882208
 24645:20171019:145505.763 rip     =           44cd58 =              4509016 =              4509016
 24645:20171019:145505.763 efl     =            10246 =                66118 =                66118
 24645:20171019:145505.763 csgsfs  = 5f82000000000033 =  6882063180575539251 =  6882063180575539251
 24645:20171019:145505.763 err     =                4 =                    4 =                    4
 24645:20171019:145505.763 trapno  =                e =                   14 =                   14
 24645:20171019:145505.763 oldmask =                0 =                    0 =                    0
 24645:20171019:145505.764 cr2     =     7ffcfb002fd0 =      140724519579600 =      140724519579600
 24645:20171019:145505.764 === Backtrace: ===
 24645:20171019:145505.871 12: /usr/local/sbin/zabbix_server: poller #57 [got 14 values in 0.078198 sec, getting values](print_fatal_info+0xa6) [0x48a656]
 24645:20171019:145505.871 11: /usr/local/sbin/zabbix_server: poller #57 [got 14 values in 0.078198 sec, getting values]() [0x48ad63]
 24645:20171019:145505.871 10: /lib64/libc.so.6() [0x3a1cc32510]
 24645:20171019:145505.871 9: /usr/local/sbin/zabbix_server: poller #57 [got 14 values in 0.078198 sec, getting values](set_result_type+0x128) [0x44cd58]
 24645:20171019:145505.871 8: /usr/local/sbin/zabbix_server: poller #57 [got 14 values in 0.078198 sec, getting values](get_value_db+0x169) [0x434a59]
 24645:20171019:145505.871 7: /usr/local/sbin/zabbix_server: poller #57 [got 14 values in 0.078198 sec, getting values]() [0x42b9e8]
 24645:20171019:145505.871 6: /usr/local/sbin/zabbix_server: poller #57 [got 14 values in 0.078198 sec, getting values](poller_thread+0xf6) [0x42bba6]
 24645:20171019:145505.871 5: /usr/local/sbin/zabbix_server: poller #57 [got 14 values in 0.078198 sec, getting values](zbx_thread_start+0x62) [0x48b2a2]
 24645:20171019:145505.871 4: /usr/local/sbin/zabbix_server: poller #57 [got 14 values in 0.078198 sec, getting values](MAIN_ZABBIX_ENTRY+0x37b) [0x41e1ab]
 24645:20171019:145505.871 3: /usr/local/sbin/zabbix_server: poller #57 [got 14 values in 0.078198 sec, getting values](daemon_start+0x1ae) [0x48995e]
 24645:20171019:145505.871 2: /usr/local/sbin/zabbix_server: poller #57 [got 14 values in 0.078198 sec, getting values](main+0x3c8) [0x41eb18]
 24645:20171019:145505.871 1: /lib64/libc.so.6(__libc_start_main+0xfd) [0x3a1cc1ed1d]
 24645:20171019:145505.871 0: /usr/local/sbin/zabbix_server: poller #57 [got 14 values in 0.078198 sec, getting values]() [0x415209] 

the same 3.2.8 binary, which was provided as zabbix_server.dump.rar

glebs.ivanovskis I would love to see registers too, if possible.
zalex_ua Added above.

Comment by Glebs Ivanovskis (Inactive) [ 2017 Oct 20 ]

First backtace led me to a place where DC_ITEM * pointer is dereferenced, second one – to the place where AGENT_RESULT * was dereferenced. If we trace where these pointers come from, here is what we see:

static int	get_values(unsigned char poller_type, int *nextcheck)
{
	const char		*__function_name = "get_values";
	DC_ITEM			items[MAX_POLLER_ITEMS];
	AGENT_RESULT		results[MAX_POLLER_ITEMS];

If attempts to dereference items or results allocated on stack cause segmentation fault, then the root cause of all problems is stack corruption.

Comment by Glebs Ivanovskis (Inactive) [ 2017 Nov 09 ]

Similar symptoms in ZBX-13010.

Comment by Glebs Ivanovskis (Inactive) [ 2018 Mar 21 ]

It looks like this issue, ZBX-13010 and ZBX-13624 are all caused by a bug in Oracle ODBC driver causing buffer overflow, stack corruption and all sorts of miracles.

Closing as Duplicate of ZBX-13010 since it contains more information and a final conclusion.

Generated at Fri Mar 29 17:50:17 EET 2024 using Jira 9.12.4#9120004-sha1:625303b708afdb767e17cb2838290c41888e9ff0.