[ZBX-10410] Zabbix server crash when doing history cache sync during shutdown Created: 2016 Feb 18  Updated: 2017 May 30  Resolved: 2016 Apr 06

Status: Closed
Project: ZABBIX BUGS AND ISSUES
Component/s: Proxy (P), Server (S)
Affects Version/s: 3.0.0
Fix Version/s: 3.0.2rc1, 3.2.0alpha1

Type: Incident report Priority: Critical
Reporter: Andris Zeila Assignee: Unassigned
Resolution: Fixed Votes: 0
Labels: crash
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: File hostcreator_api.py    

 Description   

Zabbix server crashed when exiting because of low history index cache size:

  31283:20160218:160105.927 __mem_malloc: skipped 3 asked 66304 kip_min 1728 skip_max 22608
  31283:20160218:160105.927 [file:dbcache.c,line:2561] zbx_mem_realloc(): out of memory (requested 66304 bytes)
  31283:20160218:160105.927 [file:dbcache.c,line:2561] zbx_mem_realloc(): please increase HistoryIndexCacheSize configuration parameter

with the following stack trace:

  31277:20160218:160107.937 === Backtrace: ===
  31277:20160218:160107.952 17: 
/home/monitoring/zabbix-bin/sbin/zabbix_server(print_fatal_info+0x19d) [0x46bcfd]
  31277:20160218:160107.952 16: /home/monitoring/zabbix-bin/sbin/zabbix_server() [0x46c058]
  31277:20160218:160107.952 15: /lib/x86_64-linux-gnu/libc.so.6(+0x36d40) [0x7f6b8315ad40]
  
  31277:20160218:160107.952 14: /home/monitoring/zabbix-bin/sbin/zabbix_server() [0x44cb23]
	return zbx_timespec_compare(&item1->tail->ts, &item2->tail->ts); 
  
  31277:20160218:160107.952 13: /home/monitoring/zabbix-bin/sbin/zabbix_server() [0x4648fb]
	if (heap->compare_func(&heap->elems[(index - 1) / 2], &heap->elems[index]) <= 0)
  
    31277:20160218:160107.952 12: /home/monitoring/zabbix-bin/sbin/zabbix_server(zbx_binary_heap_insert+0xbe) [0x464c1c] 
	index = __binary_heap_bubble_up(heap, index);
  
  31277:20160218:160107.953 11: /home/monitoring/zabbix-bin/sbin/zabbix_server() [0x44cb5c]
	static void	hc_queue_item(zbx_hc_item_t *item)
	{
		...
		zbx_binary_heap_insert(&cache->history_queue, &elem);
	}
	
  31277:20160218:160107.953 10: /home/monitoring/zabbix-bin/sbin/zabbix_server(DCsync_history+0xf0) [0x44df64]
	if (ZBX_HC_ITEM_STATUS_QUEUED != item->status)
		hc_queue_item(item);
	
  31277:20160218:160107.953 9: /home/monitoring/zabbix-bin/sbin/zabbix_server(free_database_cache+0x40) [0x4511d0]
  31277:20160218:160107.953 8: /home/monitoring/zabbix-bin/sbin/zabbix_server(zbx_on_exit+0xe6) [0x417d01]
  31277:20160218:160107.953 7: /home/monitoring/zabbix-bin/sbin/zabbix_server() [0x46c310] 
  31277:20160218:160107.953 6: /lib/x86_64-linux-gnu/libc.so.6(+0x36d40)

Apparently during hc_update_history_queue() the history item hashset contained items without data (NULL tail/head values). It's not clear yet how it could have happened. The initial error (insufficient history index cache size) can't be direct cause.

Current theory is that after the exit of the initial process server sent termination signal to the rest processes and one was aborted in the middle of new item adding to history cache, however unlikely it would seem:

		item = hc_get_item(item_value->itemid);

		/* new items must be inserted in queue */
		update_queue = (NULL == item->tail);

		if (NULL == item->head)
			item->tail = data;
		else
			item->head->next = data;

		item->head = data;

If a process was killed right after item = hc_get_item(item_value->itemid); then we would have history item with NULL tail/head values.



 Comments   
Comment by Andris Zeila [ 2016 Feb 19 ]

Fixed in development branch svn://svn.zabbix.com/branches/dev/ZBX-10410

Comment by Sandis Neilands (Inactive) [ 2016 Mar 17 ]

Noticed a server crash during testing.

Test scenario:

  • 10000 hosts;
  • hosts have 1 item (agent.ping) from the linked template;
  • HistoryIndexCacheSize=256K (you might have to change this a bit);
  • after changing the interval from 30s to 5s the server crashed.
  • the scenario for a proxy is symmetrical.
 21377:20160317:174620.314 Starting Zabbix Server. Zabbix 3.0.1rc1 (revision {ZABBIX_REVISION}).
 21377:20160317:174620.314 ****** Enabled features ******
 21377:20160317:174620.314 SNMP monitoring:           YES
 21377:20160317:174620.314 IPMI monitoring:           YES
 21377:20160317:174620.314 Web monitoring:            YES
 21377:20160317:174620.314 VMware monitoring:         YES
 21377:20160317:174620.314 SMTP authentication:       YES
 21377:20160317:174620.314 Jabber notifications:      YES
 21377:20160317:174620.314 Ez Texting notifications:  YES
 21377:20160317:174620.314 ODBC:                      YES
 21377:20160317:174620.314 SSH2 support:              YES
 21377:20160317:174620.314 IPv6 support:              YES
 21377:20160317:174620.314 TLS support:               YES
 21377:20160317:174620.314 ******************************
 21377:20160317:174620.314 using configuration file: /home/sandis/ZBX/ZBX-10410/ZBX-10410/usr/etc/zabbix_server.conf
 21377:20160317:174620.316 current database version (mandatory/optional): 03000000/03000000
 21377:20160317:174620.316 required mandatory version: 03000000
 21377:20160317:174620.400 server #0 started [main process]
 21379:20160317:174620.400 server #1 started [configuration syncer #1]
 21380:20160317:174620.400 server #2 started [db watchdog #1]
 21381:20160317:174620.401 server #3 started [poller #1]
 21382:20160317:174620.401 server #4 started [poller #2]
 21383:20160317:174620.401 server #5 started [poller #3]
 21384:20160317:174620.402 server #6 started [poller #4]
 21389:20160317:174620.405 server #11 started [trapper #3]
 21396:20160317:174620.405 server #18 started [http poller #1]
 21403:20160317:174620.406 server #25 started [proxy poller #1]
 21402:20160317:174620.406 server #24 started [escalator #1]
 21404:20160317:174620.407 server #26 started [self-monitoring #1]
 21395:20160317:174620.408 server #17 started [timer #1]
 21392:20160317:174620.409 server #14 started [icmp pinger #1]
 21398:20160317:174620.409 server #20 started [history syncer #1]
 21394:20160317:174620.409 server #16 started [housekeeper #1]
 21399:20160317:174620.409 server #21 started [history syncer #2]
 21400:20160317:174620.414 server #22 started [history syncer #3]
 21391:20160317:174620.418 server #13 started [trapper #5]
 21393:20160317:174620.420 server #15 started [alerter #1]
 21388:20160317:174620.420 server #10 started [trapper #2]
 21387:20160317:174620.421 server #9 started [trapper #1]
 21390:20160317:174620.422 server #12 started [trapper #4]
 21401:20160317:174620.423 server #23 started [history syncer #4]
 21386:20160317:174620.423 server #8 started [unreachable poller #1]
 21385:20160317:174620.423 server #7 started [poller #5]
 21397:20160317:174620.426 server #19 started [discoverer #1]
 21381:20160317:180734.572 __mem_malloc: skipped 2 asked 19648 skip_min 464 skip_max 17160
 21382:20160317:180735.605 __mem_malloc: skipped 2 asked 29472 skip_min 464 skip_max 12856
 21382:20160317:180735.605 [file:dbcache.c,line:2561] zbx_mem_realloc(): out of memory (requested 29472 bytes)
 21382:20160317:180735.605 [file:dbcache.c,line:2561] zbx_mem_realloc(): please increase HistoryIndexCacheSize configuration parameter
 21377:20160317:180735.607 One child process died (PID:21382,exitcode/signal:1). Exiting ...
 21377:20160317:180737.608 Got signal [signal:11(SIGSEGV),reason:1,refaddr:0x647c]. Crashing ...
 21377:20160317:180737.608 ====== Fatal information: ======
 21377:20160317:180737.608 Program counter: 0x48baea
 21377:20160317:180737.608 === Registers: ===
 21377:20160317:180737.608 r8      =                4 =                    4 =                    4
 21377:20160317:180737.608 r9      =                0 =                    0 =                    0
 21377:20160317:180737.608 r10     =               7f =                  127 =                  127
 21377:20160317:180737.608 r11     =              282 =                  642 =                  642
 21377:20160317:180737.608 r12     =           4165a0 =              4285856 =              4285856
 21377:20160317:180737.608 r13     =     7ffd15d9d720 =      140724970051360 =      140724970051360
 21377:20160317:180737.609 r14     =                0 =                    0 =                    0
 21377:20160317:180737.609 r15     =                0 =                    0 =                    0
 21377:20160317:180737.609 rdi     =                0 =                    0 =                    0
 21377:20160317:180737.609 rsi     =     7ffd15d9c068 =      140724970045544 =      140724970045544
 21377:20160317:180737.609 rbp     =     7ffd15d9c080 =      140724970045568 =      140724970045568
 21377:20160317:180737.609 rbx     =                0 =                    0 =                    0
 21377:20160317:180737.609 rdx     =                2 =                    2 =                    2
 21377:20160317:180737.609 rax     =                2 =                    2 =                    2
 21377:20160317:180737.609 rcx     =             6474 =                25716 =                25716
 21377:20160317:180737.609 rsp     =     7ffd15d9c050 =      140724970045520 =      140724970045520
 21377:20160317:180737.609 rip     =           48baea =              4766442 =              4766442
 21377:20160317:180737.609 efl     =            10293 =                66195 =                66195
 21377:20160317:180737.609 csgsfs  = 749b000000000033 =  8402309529789792307 =  8402309529789792307
 21377:20160317:180737.609 err     =                4 =                    4 =                    4
 21377:20160317:180737.609 trapno  =                e =                   14 =                   14
 21377:20160317:180737.609 oldmask =            10000 =                65536 =                65536
 21377:20160317:180737.609 cr2     =             647c =                25724 =                25724
 21377:20160317:180737.609 === Backtrace: ===
 21377:20160317:180737.610 15: ./zabbix_server(print_fatal_info+0x111) [0x4be4b1]
 21377:20160317:180737.610 14: ./zabbix_server() [0x4befa7]
 21377:20160317:180737.610 13: /lib/x86_64-linux-gnu/libc.so.6(+0x36d40) [0x7f98634e7d40]
 21377:20160317:180737.610 12: ./zabbix_server() [0x48baea]
 21377:20160317:180737.610 11: ./zabbix_server(DCsync_history+0xb5) [0x487185]
 21377:20160317:180737.611 10: ./zabbix_server() [0x48a51a]
 21377:20160317:180737.611 9: ./zabbix_server(free_database_cache+0x31) [0x489fc1]
 21377:20160317:180737.611 8: ./zabbix_server(zbx_on_exit+0x17a) [0x41ebaa]
 21377:20160317:180737.611 7: ./zabbix_server() [0x4bf233]
 21377:20160317:180737.611 6: /lib/x86_64-linux-gnu/libc.so.6(+0x36d40) [0x7f98634e7d40]
 21377:20160317:180737.611 5: /lib/x86_64-linux-gnu/libc.so.6(wait+0x65) [0x7f9863571af5]
 21377:20160317:180737.611 4: ./zabbix_server(MAIN_ZABBIX_ENTRY+0x903) [0x41e993]
 21377:20160317:180737.611 3: ./zabbix_server(daemon_start+0x3a3) [0x4bd1c3]
 21377:20160317:180737.611 2: ./zabbix_server(main+0x369) [0x41e019]
 21377:20160317:180737.611 1: /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf5) [0x7f98634d2ec5]
 21377:20160317:180737.611 0: ./zabbix_server() [0x4165c9]
 21377:20160317:180737.611 === Memory map: ===
 21377:20160317:180737.611 00400000-005e2000 r-xp 00000000 08:02 3572791                            /home/sandis/ZBX/ZBX-10410/ZBX-10410/usr/sbin/zabbix_server
 21377:20160317:180737.611 007e1000-007ea000 rw-p 001e1000 08:02 3572791                            /home/sandis/ZBX/ZBX-10410/ZBX-10410/usr/sbin/zabbix_server
 21377:20160317:180737.611 007ea000-007f0000 rw-p 00000000 00:00 0 
 21377:20160317:180737.611 027c5000-027e6000 rw-p 00000000 00:00 0                                  [heap]
 21377:20160317:180737.611 027e6000-02821000 rw-p 00000000 00:00 0                                  [heap]
 21377:20160317:180737.611 7f9857dea000-7f9857e00000 r-xp 00000000 08:02 137709                     /lib/x86_64-linux-gnu/libgcc_s.so.1
 21377:20160317:180737.611 7f9857e00000-7f9857fff000 ---p 00016000 08:02 137709                     /lib/x86_64-linux-gnu/libgcc_s.so.1
 21377:20160317:180737.611 7f9857fff000-7f9858000000 rw-p 00015000 08:02 137709                     /lib/x86_64-linux-gnu/libgcc_s.so.1
 21377:20160317:180737.611 7f9858000000-7f9858021000 rw-p 00000000 00:00 0 
 21377:20160317:180737.611 7f9858021000-7f985c000000 ---p 00000000 00:00 0 
 21377:20160317:180737.611 7f985c17a000-7f985c17b000 ---p 00000000 00:00 0 
 21377:20160317:180737.611 7f985c17b000-7f985c97b000 rw-p 00000000 00:00 0 
 21377:20160317:180737.611 7f985c97b000-7f985d17b000 rw-s 00000000 00:04 261619835                  /SYSV76028405 (deleted)
 21377:20160317:180737.611 7f985d17b000-7f985d3e2000 rw-s 00000000 00:04 261554216                  /SYSV73028405 (deleted)
 21377:20160317:180737.612 7f985d3e2000-7f985e17c000 rw-s 00000000 00:04 261521444                  /SYSV67028405 (deleted)
 21377:20160317:180737.612 7f985e17c000-7f985e57c000 rw-s 00000000 00:04 261488675                  /SYSV74028405 (deleted)
 21377:20160317:180737.612 7f985e57c000-7f985e587000 r-xp 00000000 08:02 137750                     /lib/x86_64-linux-gnu/libnss_files-2.19.so
 21377:20160317:180737.612 7f985e587000-7f985e786000 ---p 0000b000 08:02 137750                     /lib/x86_64-linux-gnu/libnss_files-2.19.so
 21377:20160317:180737.612 7f985e786000-7f985e787000 r--p 0000a000 08:02 137750                     /lib/x86_64-linux-gnu/libnss_files-2.19.so
 21377:20160317:180737.612 7f985e787000-7f985e788000 rw-p 0000b000 08:02 137750                     /lib/x86_64-linux-gnu/libnss_files-2.19.so
 21377:20160317:180737.612 7f985e788000-7f985e793000 r-xp 00000000 08:02 137734                     /lib/x86_64-linux-gnu/libnss_nis-2.19.so
 21377:20160317:180737.612 7f985e793000-7f985e992000 ---p 0000b000 08:02 137734                     /lib/x86_64-linux-gnu/libnss_nis-2.19.so
 21377:20160317:180737.612 7f985e992000-7f985e993000 r--p 0000a000 08:02 137734                     /lib/x86_64-linux-gnu/libnss_nis-2.19.so
 21377:20160317:180737.612 7f985e993000-7f985e994000 rw-p 0000b000 08:02 137734                     /lib/x86_64-linux-gnu/libnss_nis-2.19.so
 21377:20160317:180737.612 7f985e994000-7f985e9ab000 r-xp 00000000 08:02 137724                     /lib/x86_64-linux-gnu/libnsl-2.19.so
 21377:20160317:180737.612 7f985e9ab000-7f985ebaa000 ---p 00017000 08:02 137724                     /lib/x86_64-linux-gnu/libnsl-2.19.so
 21377:20160317:180737.612 7f985ebaa000-7f985ebab000 r--p 00016000 08:02 137724                     /lib/x86_64-linux-gnu/libnsl-2.19.so
 21377:20160317:180737.612 7f985ebab000-7f985ebac000 rw-p 00017000 08:02 137724                     /lib/x86_64-linux-gnu/libnsl-2.19.so
 21377:20160317:180737.612 7f985ebac000-7f985ebae000 rw-p 00000000 00:00 0 
 21377:20160317:180737.612 7f985ebae000-7f985ebb7000 r-xp 00000000 08:02 137723                     /lib/x86_64-linux-gnu/libnss_compat-2.19.so
 21377:20160317:180737.612 7f985ebb7000-7f985edb6000 ---p 00009000 08:02 137723                     /lib/x86_64-linux-gnu/libnss_compat-2.19.so
 21377:20160317:180737.612 7f985edb6000-7f985edb7000 r--p 00008000 08:02 137723                     /lib/x86_64-linux-gnu/libnss_compat-2.19.so
 21377:20160317:180737.612 7f985edb7000-7f985edb8000 rw-p 00009000 08:02 137723                     /lib/x86_64-linux-gnu/libnss_compat-2.19.so
 21377:20160317:180737.612 7f985edb8000-7f985edba000 r-xp 00000000 08:02 135365                     /lib/x86_64-linux-gnu/libkeyutils.so.1.4
 21377:20160317:180737.612 7f985edba000-7f985efba000 ---p 00002000 08:02 135365                     /lib/x86_64-linux-gnu/libkeyutils.so.1.4
 21377:20160317:180737.612 7f985efba000-7f985efbb000 r--p 00002000 08:02 135365                     /lib/x86_64-linux-gnu/libkeyutils.so.1.4
 21377:20160317:180737.612 7f985efbb000-7f985efbc000 rw-p 00003000 08:02 135365                     /lib/x86_64-linux-gnu/libkeyutils.so.1.4
 21377:20160317:180737.612 7f985efbc000-7f985efc5000 r-xp 00000000 08:02 137717                     /lib/x86_64-linux-gnu/libcrypt-2.19.so
 21377:20160317:180737.612 7f985efc5000-7f985f1c5000 ---p 00009000 08:02 137717                     /lib/x86_64-linux-gnu/libcrypt-2.19.so
 21377:20160317:180737.612 7f985f1c5000-7f985f1c6000 r--p 00009000 08:02 137717                     /lib/x86_64-linux-gnu/libcrypt-2.19.so
 21377:20160317:180737.612 7f985f1c6000-7f985f1c7000 rw-p 0000a000 08:02 137717                     /lib/x86_64-linux-gnu/libcrypt-2.19.so
 21377:20160317:180737.613 7f985f1c7000-7f985f1f5000 rw-p 00000000 00:00 0 
 21377:20160317:180737.613 7f985f1f5000-7f985f2a9000 r-xp 00000000 08:02 14680360                   /usr/lib/x86_64-linux-gnu/libsqlite3.so.0.8.6
 21377:20160317:180737.613 7f985f2a9000-7f985f4a9000 ---p 000b4000 08:02 14680360                   /usr/lib/x86_64-linux-gnu/libsqlite3.so.0.8.6
 21377:20160317:180737.613 7f985f4a9000-7f985f4ab000 r--p 000b4000 08:02 14680360                   /usr/lib/x86_64-linux-gnu/libsqlite3.so.0.8.6
 21377:20160317:180737.613 7f985f4ab000-7f985f4ad000 rw-p 000b6000 08:02 14680360                   /usr/lib/x86_64-linux-gnu/libsqlite3.so.0.8.6
 21377:20160317:180737.613 7f985f4ad000-7f985f4ae000 rw-p 00000000 00:00 0 
 21377:20160317:180737.613 7f985f4ae000-7f985f4f3000 r-xp 00000000 08:02 14689108                   /usr/lib/x86_64-linux-gnu/libhx509.so.5.0.0
 21377:20160317:180737.613 7f985f4f3000-7f985f6f2000 ---p 00045000 08:02 14689108                   /usr/lib/x86_64-linux-gnu/libhx509.so.5.0.0
 21377:20160317:180737.613 7f985f6f2000-7f985f6f4000 r--p 00044000 08:02 14689108                   /usr/lib/x86_64-linux-gnu/libhx509.so.5.0.0
 21377:20160317:180737.613 7f985f6f4000-7f985f6f6000 rw-p 00046000 08:02 14689108                   /usr/lib/x86_64-linux-gnu/libhx509.so.5.0.0
 21377:20160317:180737.613 7f985f6f6000-7f985f6f7000 rw-p 00000000 00:00 0 
 21377:20160317:180737.613 7f985f6f7000-7f985f704000 r-xp 00000000 08:02 14689100                   /usr/lib/x86_64-linux-gnu/libheimbase.so.1.0.0
 21377:20160317:180737.613 7f985f704000-7f985f903000 ---p 0000d000 08:02 14689100                   /usr/lib/x86_64-linux-gnu/libheimbase.so.1.0.0
 21377:20160317:180737.613 7f985f903000-7f985f904000 r--p 0000c000 08:02 14689100                   /usr/lib/x86_64-linux-gnu/libheimbase.so.1.0.0
 21377:20160317:180737.613 7f985f904000-7f985f905000 rw-p 0000d000 08:02 14689100                   /usr/lib/x86_64-linux-gnu/libheimbase.so.1.0.0
 21377:20160317:180737.613 7f985f905000-7f985f92c000 r-xp 00000000 08:02 14689662                   /usr/lib/x86_64-linux-gnu/libwind.so.0.0.0
 21377:20160317:180737.613 7f985f92c000-7f985fb2c000 ---p 00027000 08:02 14689662                   /usr/lib/x86_64-linux-gnu/libwind.so.0.0.0
 21377:20160317:180737.613 7f985fb2c000-7f985fb2d000 r--p 00027000 08:02 14689662                   /usr/lib/x86_64-linux-gnu/libwind.so.0.0.0
 21377:20160317:180737.613 7f985fb2d000-7f985fb2e000 rw-p 00028000 08:02 14689662                   /usr/lib/x86_64-linux-gnu/libwind.so.0.0.0
 21377:20160317:180737.613 7f985fb2e000-7f985fb35000 r-xp 00000000 08:02 14681204                   /usr/lib/x86_64-linux-gnu/libffi.so.6.0.1
 21377:20160317:180737.613 7f985fb35000-7f985fd34000 ---p 00007000 08:02 14681204                   /usr/lib/x86_64-linux-gnu/libffi.so.6.0.1
 21377:20160317:180737.613 7f985fd34000-7f985fd35000 r--p 00006000 08:02 14681204                   /usr/lib/x86_64-linux-gnu/libffi.so.6.0.1
 21377:20160317:180737.613 7f985fd35000-7f985fd36000 rw-p 00007000 08:02 14681204                   /usr/lib/x86_64-linux-gnu/libffi.so.6.0.1
 21377:20160317:180737.613 7f985fd36000-7f985fd40000 r-xp 00000000 08:02 14686733                   /usr/lib/x86_64-linux-gnu/libkrb5support.so.0.1
 21377:20160317:180737.613 7f985fd40000-7f985ff3f000 ---p 0000a000 08:02 14686733                   /usr/lib/x86_64-linux-gnu/libkrb5support.so.0.1
 21377:20160317:180737.613 7f985ff3f000-7f985ff40000 r--p 00009000 08:02 14686733                   /usr/lib/x86_64-linux-gnu/libkrb5support.so.0.1
 21377:20160317:180737.613 7f985ff40000-7f985ff41000 rw-p 0000a000 08:02 14686733                   /usr/lib/x86_64-linux-gnu/libkrb5support.so.0.1
 21377:20160317:180737.613 7f985ff41000-7f985ff6d000 r-xp 00000000 08:02 14686729                   /usr/lib/x86_64-linux-gnu/libk5crypto.so.3.1
 21377:20160317:180737.614 7f985ff6d000-7f986016c000 ---p 0002c000 08:02 14686729                   /usr/lib/x86_64-linux-gnu/libk5crypto.so.3.1
 21377:20160317:180737.614 7f986016c000-7f986016e000 r--p 0002b000 08:02 14686729                   /usr/lib/x86_64-linux-gnu/libk5crypto.so.3.1
 21377:20160317:180737.614 7f986016e000-7f986016f000 rw-p 0002d000 08:02 14686729                   /usr/lib/x86_64-linux-gnu/libk5crypto.so.3.1
 21377:20160317:180737.614 7f986016f000-7f9860170000 rw-p 00000000 00:00 0 
 21377:20160317:180737.614 7f9860170000-7f986022c000 r-xp 00000000 08:02 14686731                   /usr/lib/x86_64-linux-gnu/libkrb5.so.3.3
 21377:20160317:180737.614 7f986022c000-7f986042c000 ---p 000bc000 08:02 14686731                   /usr/lib/x86_64-linux-gnu/libkrb5.so.3.3
 21377:20160317:180737.614 7f986042c000-7f9860439000 r--p 000bc000 08:02 14686731                   /usr/lib/x86_64-linux-gnu/libkrb5.so.3.3
 21377:20160317:180737.614 7f9860439000-7f986043b000 rw-p 000c9000 08:02 14686731                   /usr/lib/x86_64-linux-gnu/libkrb5.so.3.3
 21377:20160317:180737.614 7f986043b000-7f986044f000 r-xp 00000000 08:02 14689458                   /usr/lib/x86_64-linux-gnu/libroken.so.18.1.0
 21377:20160317:180737.614 7f986044f000-7f986064e000 ---p 00014000 08:02 14689458                   /usr/lib/x86_64-linux-gnu/libroken.so.18.1.0
 21377:20160317:180737.614 7f986064e000-7f986064f000 r--p 00013000 08:02 14689458                   /usr/lib/x86_64-linux-gnu/libroken.so.18.1.0
 21377:20160317:180737.614 7f986064f000-7f9860650000 rw-p 00014000 08:02 14689458                   /usr/lib/x86_64-linux-gnu/libroken.so.18.1.0
 21377:20160317:180737.614 7f9860650000-7f9860680000 r-xp 00000000 08:02 14689096                   /usr/lib/x86_64-linux-gnu/libhcrypto.so.4.1.0
 21377:20160317:180737.614 7f9860680000-7f9860880000 ---p 00030000 08:02 14689096                   /usr/lib/x86_64-linux-gnu/libhcrypto.so.4.1.0
 21377:20160317:180737.614 7f9860880000-7f9860881000 r--p 00030000 08:02 14689096                   /usr/lib/x86_64-linux-gnu/libhcrypto.so.4.1.0
 21377:20160317:180737.614 7f9860881000-7f9860882000 rw-p 00031000 08:02 14689096                   /usr/lib/x86_64-linux-gnu/libhcrypto.so.4.1.0
 21377:20160317:180737.614 7f9860882000-7f9860883000 rw-p 00000000 00:00 0 
 21377:20160317:180737.614 7f9860883000-7f9860886000 r-xp 00000000 08:02 136663                     /lib/x86_64-linux-gnu/libcom_err.so.2.1
 21377:20160317:180737.614 7f9860886000-7f9860a85000 ---p 00003000 08:02 136663                     /lib/x86_64-linux-gnu/libcom_err.so.2.1
 21377:20160317:180737.614 7f9860a85000-7f9860a86000 r--p 00002000 08:02 136663                     /lib/x86_64-linux-gnu/libcom_err.so.2.1
 21377:20160317:180737.614 7f9860a86000-7f9860a87000 rw-p 00003000 08:02 136663                     /lib/x86_64-linux-gnu/libcom_err.so.2.1
 21377:20160317:180737.614 7f9860a87000-7f9860b24000 r-xp 00000000 08:02 14688546                   /usr/lib/x86_64-linux-gnu/libasn1.so.8.0.0
 21377:20160317:180737.614 7f9860b24000-7f9860d24000 ---p 0009d000 08:02 14688546                   /usr/lib/x86_64-linux-gnu/libasn1.so.8.0.0
 21377:20160317:180737.614 7f9860d24000-7f9860d25000 r--p 0009d000 08:02 14688546                   /usr/lib/x86_64-linux-gnu/libasn1.so.8.0.0
 21377:20160317:180737.614 7f9860d25000-7f9860d28000 rw-p 0009e000 08:02 14688546                   /usr/lib/x86_64-linux-gnu/libasn1.so.8.0.0
 21377:20160317:180737.614 7f9860d28000-7f9860daa000 r-xp 00000000 08:02 14689168                   /usr/lib/x86_64-linux-gnu/libkrb5.so.26.0.0
 21377:20160317:180737.615 7f9860daa000-7f9860fa9000 ---p 00082000 08:02 14689168                   /usr/lib/x86_64-linux-gnu/libkrb5.so.26.0.0
 21377:20160317:180737.615 7f9860fa9000-7f9860fac000 r--p 00081000 08:02 14689168                   /usr/lib/x86_64-linux-gnu/libkrb5.so.26.0.0
 21377:20160317:180737.615 7f9860fac000-7f9860faf000 rw-p 00084000 08:02 14689168                   /usr/lib/x86_64-linux-gnu/libkrb5.so.26.0.0
 21377:20160317:180737.615 7f9860faf000-7f9860fb0000 rw-p 00000000 00:00 0 
 21377:20160317:180737.615 7f9860fb0000-7f9860fb8000 r-xp 00000000 08:02 14689102                   /usr/lib/x86_64-linux-gnu/libheimntlm.so.0.1.0
 21377:20160317:180737.615 7f9860fb8000-7f98611b7000 ---p 00008000 08:02 14689102                   /usr/lib/x86_64-linux-gnu/libheimntlm.so.0.1.0
 21377:20160317:180737.615 7f98611b7000-7f98611b8000 r--p 00007000 08:02 14689102                   /usr/lib/x86_64-linux-gnu/libheimntlm.so.0.1.0
 21377:20160317:180737.615 7f98611b8000-7f98611b9000 rw-p 00008000 08:02 14689102                   /usr/lib/x86_64-linux-gnu/libheimntlm.so.0.1.0
 21377:20160317:180737.615 7f98611b9000-7f98611bd000 r-xp 00000000 08:02 135355                     /lib/x86_64-linux-gnu/libgpg-error.so.0.10.0
 21377:20160317:180737.615 7f98611bd000-7f98613bc000 ---p 00004000 08:02 135355                     /lib/x86_64-linux-gnu/libgpg-error.so.0.10.0
 21377:20160317:180737.615 7f98613bc000-7f98613bd000 r--p 00003000 08:02 135355                     /lib/x86_64-linux-gnu/libgpg-error.so.0.10.0
 21377:20160317:180737.615 7f98613bd000-7f98613be000 rw-p 00004000 08:02 135355                     /lib/x86_64-linux-gnu/libgpg-error.so.0.10.0
 21377:20160317:180737.615 7f98613be000-7f98613f9000 r-xp 00000000 08:02 14689363                   /usr/lib/x86_64-linux-gnu/libp11-kit.so.0.0.0
 21377:20160317:180737.615 7f98613f9000-7f98615f8000 ---p 0003b000 08:02 14689363                   /usr/lib/x86_64-linux-gnu/libp11-kit.so.0.0.0
 21377:20160317:180737.615 7f98615f8000-7f98615fe000 r--p 0003a000 08:02 14689363                   /usr/lib/x86_64-linux-gnu/libp11-kit.so.0.0.0
 21377:20160317:180737.615 7f98615fe000-7f9861600000 rw-p 00040000 08:02 14689363                   /usr/lib/x86_64-linux-gnu/libp11-kit.so.0.0.0
 21377:20160317:180737.615 7f9861600000-7f9861612000 r-xp 00000000 08:02 14689554                   /usr/lib/x86_64-linux-gnu/libtasn1.so.6.2.0
 21377:20160317:180737.615 7f9861612000-7f9861812000 ---p 00012000 08:02 14689554                   /usr/lib/x86_64-linux-gnu/libtasn1.so.6.2.0
 21377:20160317:180737.615 7f9861812000-7f9861813000 r--p 00012000 08:02 14689554                   /usr/lib/x86_64-linux-gnu/libtasn1.so.6.2.0
 21377:20160317:180737.615 7f9861813000-7f9861814000 rw-p 00013000 08:02 14689554                   /usr/lib/x86_64-linux-gnu/libtasn1.so.6.2.0
 21377:20160317:180737.615 7f9861814000-7f9861858000 r-xp 00000000 08:02 14687353                   /usr/lib/x86_64-linux-gnu/libgssapi_krb5.so.2.2
 21377:20160317:180737.615 7f9861858000-7f9861a58000 ---p 00044000 08:02 14687353                   /usr/lib/x86_64-linux-gnu/libgssapi_krb5.so.2.2
 21377:20160317:180737.615 7f9861a58000-7f9861a59000 r--p 00044000 08:02 14687353                   /usr/lib/x86_64-linux-gnu/libgssapi_krb5.so.2.2
 21377:20160317:180737.615 7f9861a59000-7f9861a5b000 rw-p 00045000 08:02 14687353                   /usr/lib/x86_64-linux-gnu/libgssapi_krb5.so.2.2
 21377:20160317:180737.615 7f9861a5b000-7f9861a74000 r-xp 00000000 08:02 14689472                   /usr/lib/x86_64-linux-gnu/librtmp.so.0
 21377:20160317:180737.615 7f9861a74000-7f9861c73000 ---p 00019000 08:02 14689472                   /usr/lib/x86_64-linux-gnu/librtmp.so.0
 21377:20160317:180737.615 7f9861c73000-7f9861c74000 r--p 00018000 08:02 14689472                   /usr/lib/x86_64-linux-gnu/librtmp.so.0
 21377:20160317:180737.615 7f9861c74000-7f9861c75000 rw-p 00019000 08:02 14689472                   /usr/lib/x86_64-linux-gnu/librtmp.so.0
 21377:20160317:180737.615 7f9861c75000-7f9861ca6000 r-xp 00000000 08:02 14689128                   /usr/lib/x86_64-linux-gnu/libidn.so.11.6.11
 21377:20160317:180737.615 7f9861ca6000-7f9861ea6000 ---p 00031000 08:02 14689128                   /usr/lib/x86_64-linux-gnu/libidn.so.11.6.11
 21377:20160317:180737.615 7f9861ea6000-7f9861ea7000 r--p 00031000 08:02 14689128                   /usr/lib/x86_64-linux-gnu/libidn.so.11.6.11
 21377:20160317:180737.615 7f9861ea7000-7f9861ea8000 rw-p 00032000 08:02 14689128                   /usr/lib/x86_64-linux-gnu/libidn.so.11.6.11
 21377:20160317:180737.616 7f9861ea8000-7f9861ee2000 r-xp 00000000 08:02 14688971                   /usr/lib/x86_64-linux-gnu/libgssapi.so.3.0.0
 21377:20160317:180737.616 7f9861ee2000-7f98620e2000 ---p 0003a000 08:02 14688971                   /usr/lib/x86_64-linux-gnu/libgssapi.so.3.0.0
 21377:20160317:180737.616 7f98620e2000-7f98620e3000 r--p 0003a000 08:02 14688971                   /usr/lib/x86_64-linux-gnu/libgssapi.so.3.0.0
 21377:20160317:180737.616 7f98620e3000-7f98620e5000 rw-p 0003b000 08:02 14688971                   /usr/lib/x86_64-linux-gnu/libgssapi.so.3.0.0
 21377:20160317:180737.616 7f98620e5000-7f98620e6000 rw-p 00000000 00:00 0 
 21377:20160317:180737.616 7f98620e6000-7f98620ff000 r-xp 00000000 08:02 14689489                   /usr/lib/x86_64-linux-gnu/libsasl2.so.2.0.25
 21377:20160317:180737.616 7f98620ff000-7f98622ff000 ---p 00019000 08:02 14689489                   /usr/lib/x86_64-linux-gnu/libsasl2.so.2.0.25
 21377:20160317:180737.616 7f98622ff000-7f9862300000 r--p 00019000 08:02 14689489                   /usr/lib/x86_64-linux-gnu/libsasl2.so.2.0.25
 21377:20160317:180737.616 7f9862300000-7f9862301000 rw-p 0001a000 08:02 14689489                   /usr/lib/x86_64-linux-gnu/libsasl2.so.2.0.25
 21377:20160317:180737.616 7f9862301000-7f9862306000 r-xp 00000000 08:02 14688881                   /usr/lib/x86_64-linux-gnu/libgdbm.so.3.0.0
 21377:20160317:180737.616 7f9862306000-7f9862505000 ---p 00005000 08:02 14688881                   /usr/lib/x86_64-linux-gnu/libgdbm.so.3.0.0
 21377:20160317:180737.616 7f9862505000-7f9862506000 r--p 00004000 08:02 14688881                   /usr/lib/x86_64-linux-gnu/libgdbm.so.3.0.0
 21377:20160317:180737.616 7f9862506000-7f9862507000 rw-p 00005000 08:02 14688881                   /usr/lib/x86_64-linux-gnu/libgdbm.so.3.0.0
 21377:20160317:180737.616 7f9862507000-7f986250e000 r-xp 00000000 08:02 14687270                   /usr/lib/libOpenIPMIutils.so.0.0.1
 21377:20160317:180737.616 7f986250e000-7f986270e000 ---p 00007000 08:02 14687270                   /usr/lib/libOpenIPMIutils.so.0.0.1
 21377:20160317:180737.616 7f986270e000-7f986270f000 r--p 00007000 08:02 14687270                   /usr/lib/libOpenIPMIutils.so.0.0.1
 21377:20160317:180737.616 7f986270f000-7f9862710000 rw-p 00008000 08:02 14687270                   /usr/lib/libOpenIPMIutils.so.0.0.1
 21377:20160317:180737.616 7f9862710000-7f986278c000 r-xp 00000000 08:02 132271                     /lib/x86_64-linux-gnu/libgcrypt.so.11.8.2
 21377:20160317:180737.616 7f986278c000-7f986298c000 ---p 0007c000 08:02 132271                     /lib/x86_64-linux-gnu/libgcrypt.so.11.8.2
 21377:20160317:180737.616 7f986298c000-7f986298d000 r--p 0007c000 08:02 132271                     /lib/x86_64-linux-gnu/libgcrypt.so.11.8.2
 21377:20160317:180737.616 7f986298d000-7f9862990000 rw-p 0007d000 08:02 132271                     /lib/x86_64-linux-gnu/libgcrypt.so.11.8.2
 21377:20160317:180737.616 7f9862990000-7f9862999000 r-xp 00000000 08:02 14689190                   /usr/lib/x86_64-linux-gnu/libltdl.so.7.3.0
 21377:20160317:180737.616 7f9862999000-7f9862b98000 ---p 00009000 08:02 14689190                   /usr/lib/x86_64-linux-gnu/libltdl.so.7.3.0
 21377:20160317:180737.616 7f9862b98000-7f9862b99000 r--p 00008000 08:02 14689190                   /usr/lib/x86_64-linux-gnu/libltdl.so.7.3.0
 21377:20160317:180737.616 7f9862b99000-7f9862b9a000 rw-p 00009000 08:02 14689190                   /usr/lib/x86_64-linux-gnu/libltdl.so.7.3.0
 21377:20160317:180737.616 7f9862b9a000-7f9862bbb000 r-xp 00000000 08:02 135370                     /lib/x86_64-linux-gnu/liblzma.so.5.0.0
 21377:20160317:180737.616 7f9862bbb000-7f9862dba000 ---p 00021000 08:02 135370                     /lib/x86_64-linux-gnu/liblzma.so.5.0.0
 21377:20160317:180737.616 7f9862dba000-7f9862dbb000 r--p 00020000 08:02 135370                     /lib/x86_64-linux-gnu/liblzma.so.5.0.0
 21377:20160317:180737.616 7f9862dbb000-7f9862dbc000 rw-p 00021000 08:02 135370                     /lib/x86_64-linux-gnu/liblzma.so.5.0.0
 21377:20160317:180737.616 7f9862dbc000-7f9862e73000 r-xp 00000000 08:02 14685133                   /usr/lib/x86_64-linux-gnu/libgnutls.so.26.22.6
 21377:20160317:180737.616 7f9862e73000-7f9863072000 ---p 000b7000 08:02 14685133                   /usr/lib/x86_64-linux-gnu/libgnutls.so.26.22.6
 21377:20160317:180737.617 7f9863072000-7f9863078000 r--p 000b6000 08:02 14685133                   /usr/lib/x86_64-linux-gnu/libgnutls.so.26.22.6
 21377:20160317:180737.617 7f9863078000-7f9863079000 rw-p 000bc000 08:02 14685133                   /usr/lib/x86_64-linux-gnu/libgnutls.so.26.22.6
 21377:20160317:180737.617 7f9863079000-7f986307a000 rw-p 00000000 00:00 0 
 21377:20160317:180737.617 7f986307a000-7f9863093000 r-xp 00000000 08:02 137725                     /lib/x86_64-linux-gnu/libpthread-2.19.so
 21377:20160317:180737.617 7f9863093000-7f9863292000 ---p 00019000 08:02 137725                     /lib/x86_64-linux-gnu/libpthread-2.19.so
 21377:20160317:180737.617 7f9863292000-7f9863293000 r--p 00018000 08:02 137725                     /lib/x86_64-linux-gnu/libpthread-2.19.so
 21377:20160317:180737.617 7f9863293000-7f9863294000 rw-p 00019000 08:02 137725                     /lib/x86_64-linux-gnu/libpthread-2.19.so
 21377:20160317:180737.617 7f9863294000-7f9863298000 rw-p 00000000 00:00 0 
 21377:20160317:180737.617 7f9863298000-7f98632b0000 r-xp 00000000 08:02 135484                     /lib/x86_64-linux-gnu/libz.so.1.2.8
 21377:20160317:180737.617 7f98632b0000-7f98634af000 ---p 00018000 08:02 135484                     /lib/x86_64-linux-gnu/libz.so.1.2.8
 21377:20160317:180737.617 7f98634af000-7f98634b0000 r--p 00017000 08:02 135484                     /lib/x86_64-linux-gnu/libz.so.1.2.8
 21377:20160317:180737.617 7f98634b0000-7f98634b1000 rw-p 00018000 08:02 135484                     /lib/x86_64-linux-gnu/libz.so.1.2.8
 21377:20160317:180737.617 7f98634b1000-7f986366c000 r-xp 00000000 08:02 137744                     /lib/x86_64-linux-gnu/libc-2.19.so
 21377:20160317:180737.617 7f986366c000-7f986386b000 ---p 001bb000 08:02 137744                     /lib/x86_64-linux-gnu/libc-2.19.so
 21377:20160317:180737.617 7f986386b000-7f986386f000 r--p 001ba000 08:02 137744                     /lib/x86_64-linux-gnu/libc-2.19.so
 21377:20160317:180737.617 7f986386f000-7f9863871000 rw-p 001be000 08:02 137744                     /lib/x86_64-linux-gnu/libc-2.19.so
 21377:20160317:180737.617 7f9863871000-7f9863876000 rw-p 00000000 00:00 0 
 21377:20160317:180737.617 7f9863876000-7f986388d000 r-xp 00000000 08:02 135799                     /lib/x86_64-linux-gnu/libresolv-2.19.so
 21377:20160317:180737.617 7f986388d000-7f9863a8d000 ---p 00017000 08:02 135799                     /lib/x86_64-linux-gnu/libresolv-2.19.so
 21377:20160317:180737.617 7f9863a8d000-7f9863a8e000 r--p 00017000 08:02 135799                     /lib/x86_64-linux-gnu/libresolv-2.19.so
 21377:20160317:180737.617 7f9863a8e000-7f9863a8f000 rw-p 00018000 08:02 135799                     /lib/x86_64-linux-gnu/libresolv-2.19.so
 21377:20160317:180737.617 7f9863a8f000-7f9863a91000 rw-p 00000000 00:00 0 
 21377:20160317:180737.617 7f9863a91000-7f9863a94000 r-xp 00000000 08:02 137716                     /lib/x86_64-linux-gnu/libdl-2.19.so
 21377:20160317:180737.617 7f9863a94000-7f9863c93000 ---p 00003000 08:02 137716                     /lib/x86_64-linux-gnu/libdl-2.19.so
 21377:20160317:180737.617 7f9863c93000-7f9863c94000 r--p 00002000 08:02 137716                     /lib/x86_64-linux-gnu/libdl-2.19.so
 21377:20160317:180737.617 7f9863c94000-7f9863c95000 rw-p 00003000 08:02 137716                     /lib/x86_64-linux-gnu/libdl-2.19.so
 21377:20160317:180737.617 7f9863c95000-7f9863d9a000 r-xp 00000000 08:02 137714                     /lib/x86_64-linux-gnu/libm-2.19.so
 21377:20160317:180737.617 7f9863d9a000-7f9863f99000 ---p 00105000 08:02 137714                     /lib/x86_64-linux-gnu/libm-2.19.so
 21377:20160317:180737.617 7f9863f99000-7f9863f9a000 r--p 00104000 08:02 137714                     /lib/x86_64-linux-gnu/libm-2.19.so
 21377:20160317:180737.617 7f9863f9a000-7f9863f9b000 rw-p 00105000 08:02 137714                     /lib/x86_64-linux-gnu/libm-2.19.so
 21377:20160317:180737.617 7f9863f9b000-7f9863ffa000 r-xp 00000000 08:02 14685159                   /usr/lib/x86_64-linux-gnu/libcurl-gnutls.so.4.3.0
 21377:20160317:180737.618 7f9863ffa000-7f98641fa000 ---p 0005f000 08:02 14685159                   /usr/lib/x86_64-linux-gnu/libcurl-gnutls.so.4.3.0
 21377:20160317:180737.618 7f98641fa000-7f98641fc000 r--p 0005f000 08:02 14685159                   /usr/lib/x86_64-linux-gnu/libcurl-gnutls.so.4.3.0
 21377:20160317:180737.618 7f98641fc000-7f98641fd000 rw-p 00061000 08:02 14685159                   /usr/lib/x86_64-linux-gnu/libcurl-gnutls.so.4.3.0
 21377:20160317:180737.618 7f98641fd000-7f986420a000 r-xp 00000000 08:02 14687363                   /usr/lib/x86_64-linux-gnu/liblber-2.4.so.2.8.3
 21377:20160317:180737.618 7f986420a000-7f986440a000 ---p 0000d000 08:02 14687363                   /usr/lib/x86_64-linux-gnu/liblber-2.4.so.2.8.3
 21377:20160317:180737.618 7f986440a000-7f986440b000 r--p 0000d000 08:02 14687363                   /usr/lib/x86_64-linux-gnu/liblber-2.4.so.2.8.3
 21377:20160317:180737.618 7f986440b000-7f986440c000 rw-p 0000e000 08:02 14687363                   /usr/lib/x86_64-linux-gnu/liblber-2.4.so.2.8.3
 21377:20160317:180737.618 7f986440c000-7f9864459000 r-xp 00000000 08:02 14687362                   /usr/lib/x86_64-linux-gnu/libldap_r-2.4.so.2.8.3
 21377:20160317:180737.618 7f9864459000-7f9864658000 ---p 0004d000 08:02 14687362                   /usr/lib/x86_64-linux-gnu/libldap_r-2.4.so.2.8.3
 21377:20160317:180737.618 7f9864658000-7f986465a000 r--p 0004c000 08:02 14687362                   /usr/lib/x86_64-linux-gnu/libldap_r-2.4.so.2.8.3
 21377:20160317:180737.618 7f986465a000-7f986465b000 rw-p 0004e000 08:02 14687362                   /usr/lib/x86_64-linux-gnu/libldap_r-2.4.so.2.8.3
 21377:20160317:180737.618 7f986465b000-7f986465d000 rw-p 00000000 00:00 0 
 21377:20160317:180737.618 7f986465d000-7f9864810000 r-xp 00000000 08:02 132171                     /lib/x86_64-linux-gnu/libcrypto.so.1.0.0
 21377:20160317:180737.618 7f9864810000-7f9864a0f000 ---p 001b3000 08:02 132171                     /lib/x86_64-linux-gnu/libcrypto.so.1.0.0
 21377:20160317:180737.618 7f9864a0f000-7f9864a2a000 r--p 001b2000 08:02 132171                     /lib/x86_64-linux-gnu/libcrypto.so.1.0.0
 21377:20160317:180737.618 7f9864a2a000-7f9864a35000 rw-p 001cd000 08:02 132171                     /lib/x86_64-linux-gnu/libcrypto.so.1.0.0
 21377:20160317:180737.618 7f9864a35000-7f9864a39000 rw-p 00000000 00:00 0 
 21377:20160317:180737.618 7f9864a39000-7f9864a8e000 r-xp 00000000 08:02 132170                     /lib/x86_64-linux-gnu/libssl.so.1.0.0
 21377:20160317:180737.618 7f9864a8e000-7f9864c8e000 ---p 00055000 08:02 132170                     /lib/x86_64-linux-gnu/libssl.so.1.0.0
 21377:20160317:180737.618 7f9864c8e000-7f9864c91000 r--p 00055000 08:02 132170                     /lib/x86_64-linux-gnu/libssl.so.1.0.0
 21377:20160317:180737.618 7f9864c91000-7f9864c98000 rw-p 00058000 08:02 132170                     /lib/x86_64-linux-gnu/libssl.so.1.0.0
 21377:20160317:180737.618 7f9864c98000-7f9864c9d000 r-xp 00000000 08:02 14687272                   /usr/lib/libOpenIPMIposix.so.0.0.1
 21377:20160317:180737.618 7f9864c9d000-7f9864e9c000 ---p 00005000 08:02 14687272                   /usr/lib/libOpenIPMIposix.so.0.0.1
 21377:20160317:180737.618 7f9864e9c000-7f9864e9d000 r--p 00004000 08:02 14687272                   /usr/lib/libOpenIPMIposix.so.0.0.1
 21377:20160317:180737.618 7f9864e9d000-7f9864e9e000 rw-p 00005000 08:02 14687272                   /usr/lib/libOpenIPMIposix.so.0.0.1
 21377:20160317:180737.618 7f9864e9e000-7f9864f88000 r-xp 00000000 08:02 14687267                   /usr/lib/libOpenIPMI.so.0.0.5
 21377:20160317:180737.618 7f9864f88000-7f9865188000 ---p 000ea000 08:02 14687267                   /usr/lib/libOpenIPMI.so.0.0.5
 21377:20160317:180737.618 7f9865188000-7f986519e000 r--p 000ea000 08:02 14687267                   /usr/lib/libOpenIPMI.so.0.0.5
 21377:20160317:180737.618 7f986519e000-7f98651a4000 rw-p 00100000 08:02 14687267                   /usr/lib/libOpenIPMI.so.0.0.5
 21377:20160317:180737.619 7f98651a4000-7f98651a9000 rw-p 00000000 00:00 0 
 21377:20160317:180737.619 7f98651a9000-7f98651d0000 r-xp 00000000 08:02 14689540                   /usr/lib/x86_64-linux-gnu/libssh2.so.1.0.1
 21377:20160317:180737.619 7f98651d0000-7f98653cf000 ---p 00027000 08:02 14689540                   /usr/lib/x86_64-linux-gnu/libssh2.so.1.0.1
 21377:20160317:180737.619 7f98653cf000-7f98653d0000 r--p 00026000 08:02 14689540                   /usr/lib/x86_64-linux-gnu/libssh2.so.1.0.1
 21377:20160317:180737.619 7f98653d0000-7f98653d1000 rw-p 00027000 08:02 14689540                   /usr/lib/x86_64-linux-gnu/libssh2.so.1.0.1
 21377:20160317:180737.619 7f98653d1000-7f9865472000 r-xp 00000000 08:02 14687022                   /usr/lib/x86_64-linux-gnu/libnetsnmp.so.30.0.2
 21377:20160317:180737.619 7f9865472000-7f9865672000 ---p 000a1000 08:02 14687022                   /usr/lib/x86_64-linux-gnu/libnetsnmp.so.30.0.2
 21377:20160317:180737.619 7f9865672000-7f9865673000 r--p 000a1000 08:02 14687022                   /usr/lib/x86_64-linux-gnu/libnetsnmp.so.30.0.2
 21377:20160317:180737.619 7f9865673000-7f9865675000 rw-p 000a2000 08:02 14687022                   /usr/lib/x86_64-linux-gnu/libnetsnmp.so.30.0.2
 21377:20160317:180737.619 7f9865675000-7f98656ab000 rw-p 00000000 00:00 0 
 21377:20160317:180737.619 7f98656ab000-7f986570a000 r-xp 00000000 08:02 14687891                   /usr/lib/x86_64-linux-gnu/libodbc.so.1.0.0
 21377:20160317:180737.619 7f986570a000-7f9865909000 ---p 0005f000 08:02 14687891                   /usr/lib/x86_64-linux-gnu/libodbc.so.1.0.0
 21377:20160317:180737.619 7f9865909000-7f986590a000 r--p 0005e000 08:02 14687891                   /usr/lib/x86_64-linux-gnu/libodbc.so.1.0.0
 21377:20160317:180737.619 7f986590a000-7f9865911000 rw-p 0005f000 08:02 14687891                   /usr/lib/x86_64-linux-gnu/libodbc.so.1.0.0
 21377:20160317:180737.619 7f9865911000-7f9865912000 rw-p 00000000 00:00 0 
 21377:20160317:180737.619 7f9865912000-7f9865a6e000 r-xp 00000000 08:02 14681806                   /usr/lib/x86_64-linux-gnu/libxml2.so.2.9.1
 21377:20160317:180737.619 7f9865a6e000-7f9865c6e000 ---p 0015c000 08:02 14681806                   /usr/lib/x86_64-linux-gnu/libxml2.so.2.9.1
 21377:20160317:180737.619 7f9865c6e000-7f9865c76000 r--p 0015c000 08:02 14681806                   /usr/lib/x86_64-linux-gnu/libxml2.so.2.9.1
 21377:20160317:180737.619 7f9865c76000-7f9865c78000 rw-p 00164000 08:02 14681806                   /usr/lib/x86_64-linux-gnu/libxml2.so.2.9.1
 21377:20160317:180737.619 7f9865c78000-7f9865c79000 rw-p 00000000 00:00 0 
 21377:20160317:180737.619 7f9865c79000-7f9865c85000 r-xp 00000000 08:02 14687433                   /usr/lib/libiksemel.so.3.0.0
 21377:20160317:180737.619 7f9865c85000-7f9865e85000 ---p 0000c000 08:02 14687433                   /usr/lib/libiksemel.so.3.0.0
 21377:20160317:180737.619 7f9865e85000-7f9865e86000 r--p 0000c000 08:02 14687433                   /usr/lib/libiksemel.so.3.0.0
 21377:20160317:180737.619 7f9865e86000-7f9865e87000 rw-p 0000d000 08:02 14687433                   /usr/lib/libiksemel.so.3.0.0
 21377:20160317:180737.619 7f9865e87000-7f9866137000 r-xp 00000000 08:02 14686044                   /usr/lib/x86_64-linux-gnu/libmysqlclient.so.18.0.0
 21377:20160317:180737.619 7f9866137000-7f9866337000 ---p 002b0000 08:02 14686044                   /usr/lib/x86_64-linux-gnu/libmysqlclient.so.18.0.0
 21377:20160317:180737.619 7f9866337000-7f986633c000 r--p 002b0000 08:02 14686044                   /usr/lib/x86_64-linux-gnu/libmysqlclient.so.18.0.0
 21377:20160317:180737.619 7f986633c000-7f98663ba000 rw-p 002b5000 08:02 14686044                   /usr/lib/x86_64-linux-gnu/libmysqlclient.so.18.0.0
 21377:20160317:180737.619 7f98663ba000-7f98663bf000 rw-p 00000000 00:00 0 
 21377:20160317:180737.619 7f98663bf000-7f98663e2000 r-xp 00000000 08:02 137730                     /lib/x86_64-linux-gnu/ld-2.19.so
 21377:20160317:180737.619 7f9866584000-7f98665a4000 rw-s 00000000 00:04 261455901                  /SYSV48028405 (deleted)
 21377:20160317:180737.619 7f98665a4000-7f98665bb000 rw-p 00000000 00:00 0 
 21377:20160317:180737.620 7f98665bb000-7f98665bc000 rw-p 00000000 00:00 0 
 21377:20160317:180737.620 7f98665bc000-7f98665be000 rw-s 00000000 00:04 261587064                  /SYSV53028405 (deleted)
 21377:20160317:180737.620 7f98665be000-7f98665de000 rw-s 00000000 00:04 261423122                  /SYSV68028405 (deleted)
 21377:20160317:180737.620 7f98665de000-7f98665df000 rw-p 00000000 00:00 0 
 21377:20160317:180737.620 7f98665df000-7f98665e1000 rw-p 00000000 00:00 0 
 21377:20160317:180737.620 7f98665e1000-7f98665e2000 r--p 00022000 08:02 137730                     /lib/x86_64-linux-gnu/ld-2.19.so
 21377:20160317:180737.620 7f98665e2000-7f98665e3000 rw-p 00023000 08:02 137730                     /lib/x86_64-linux-gnu/ld-2.19.so
 21377:20160317:180737.620 7f98665e3000-7f98665e4000 rw-p 00000000 00:00 0 
 21377:20160317:180737.620 7ffd15d7e000-7ffd15d9f000 rw-p 00000000 00:00 0                          [stack]
 21377:20160317:180737.620 7ffd15dac000-7ffd15dae000 r-xp 00000000 00:00 0                          [vdso]
 21377:20160317:180737.620 7ffd15dae000-7ffd15db0000 r--p 00000000 00:00 0                          [vvar]
 21377:20160317:180737.620 ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0                  [vsyscall]
 21377:20160317:180737.620 ================================
 21377:20160317:180737.620 Please consider attaching a disassembly listing to your bug report.
 21377:20160317:180737.620 This listing can be produced with, e.g., objdump -DSswx zabbix_server.
 21377:20160317:180737.620 ================================
Comment by Sandis Neilands (Inactive) [ 2016 Mar 31 ]

The cause of the crash is an error in __binary_heap_ensure_free_space(). It stores the number of allocated binary heap elements before the allocation. If the allocation fails then other processes can read or write past the allocated shared memory for the binary heap. In our case this happened with history queue which is stored as binary heap in history index cache shared memory. This issue has been latent since the initial implementation of binary heap back in 2010 but it was discovered only now after implementing ZBXNEXT-3071.

However there is a more serious problem. If the server has run out of history index cache memory then the final synchronization is not possible as well. This is a regression of ZBXNEXT-3071.

The solution for the crash is trivial.

The solution for the final synchronization problem is non-trivial. Perhaps the simplest approach is to store the history queue in heap memory instead of history index cache's shared memory during the final synchronization.

Comment by Sandis Neilands (Inactive) [ 2016 Apr 05 ]

Fixed a memory corruption caused by failed memory allocation when inserting element in a binary heap in r59205.

Reduced the risk of failed final history synchronization due to exhausted history index cache memory in r59285.

The remaining failure mode - the main server/proxy process runs out of heap memory during final synchronization. The maximum HistoryIndexCacheSize=2G.

Comment by Sandis Neilands (Inactive) [ 2016 Apr 05 ]

The attached python script adds 10000 hosts via API. You'll have to adapt the program to suit your environment.

Comment by Andris Zeila [ 2016 Apr 06 ]

Successfully tested, pleased review minor code restructuring in r59304

sandis.neilands Reviewed, CLOSED.

Comment by Andris Zeila [ 2016 Apr 06 ]

Released in:

  • pre-3.0.2rc1 r59306
  • pre-3.1.0 r59307
Comment by Sandis Neilands (Inactive) [ 2016 Apr 06 ]

New issues related to this problem:

  • ZBX-10630 - Protect cache integrity by blocking signals for critical sections;
  • ZBXNEXT-3233 - More controlled exit (server, proxy).
Generated at Fri Apr 19 09:40:34 EEST 2024 using Jira 9.12.4#9120004-sha1:625303b708afdb767e17cb2838290c41888e9ff0.