[ZBX-11017] zabbix_server crashes on start (Got signal [signal:11(SIGSEGV),reason:1,refaddr:0x1000000000]. Crashing ...) Created: 2016 Jul 22  Updated: 2018 Jan 02  Resolved: 2018 Jan 02

Status: Closed
Project: ZABBIX BUGS AND ISSUES
Component/s: Server (S)
Affects Version/s: 3.0.3
Fix Version/s: None

Type: Incident report Priority: Blocker
Reporter: André Burkhardt Assignee: Unassigned
Resolution: Workaround proposed Votes: 0
Labels: crash
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

SLES 11 SP4, PostgreSQL 9.4.8 on dedicated Server

Attachments: File zabbix_server.objdump-new.gz     File zabbix_server.tar.gz    
Issue Links:
Sub-task
depends on ZBX-12159 Resolving TNS names via LDAP crash on... Confirmed

 Description   

We want upgrade our Zabbix 2.4.7 Environment to Zabbix 3.0.3. All preparation is done but zabbix_server won't start. It crashes with "Got signal [signal:11(SIGSEGV),reason:1,refaddr:0x1000000000]. Crashing ..." The zabbix_server.log and an .objdump-file is attached.

The same problem exists on separate Testsystem and when using an empty fresh database.

Can you help? Thx

 Comments   
Comment by Glebs Ivanovskis (Inactive) [ 2016 Jul 22 ]

Server is crashing synchronously in all poller processes, unreachable poller process and one anonymous process (I suspect #19 discoverer, if you are using default Start* settings) after in init_snmp() call. Example backtrace for easier searching:

 28715:20160722:075908.242 === Backtrace: ===
 28715:20160722:075908.243 17: /usr/local/zabbix-server/sbin/zabbix_server(print_fatal_info+0xe2) [0x480482]
 28715:20160722:075908.243 16: /usr/local/zabbix-server/sbin/zabbix_server() [0x480d4d]
 28715:20160722:075908.243 15: /lib64/libc.so.6(+0x32910) [0x7fb99fe96910]
 28715:20160722:075908.244 14: /usr/lib64/libcrypto.so.1.0.0(EVP_PKEY_CTX_free+0x9) [0x7fb9a126cd19]
 28715:20160722:075908.244 13: /usr/lib64/libcrypto.so.1.0.0(EVP_MD_CTX_cleanup+0x3f) [0x7fb9a125f34f]
 28715:20160722:075908.244 12: /usr/lib64/libnetsnmp.so.15(sc_hash+0x166) [0x7fb9a1f4a8e6]
 28715:20160722:075908.244 11: /usr/lib64/libnetsnmp.so.15(hash_engineID+0x8e) [0x7fb9a1f491be]
 28715:20160722:075908.245 10: /usr/lib64/libnetsnmp.so.15(search_enginetime_list+0x31) [0x7fb9a1f492e1]
 28715:20160722:075908.245 9: /usr/lib64/libnetsnmp.so.15(set_enginetime+0x56) [0x7fb9a1f497e6]
 28715:20160722:075908.245 8: /usr/lib64/libnetsnmp.so.15(init_snmpv3_post_config+0x94) [0x7fb9a1f46614]
 28715:20160722:075908.245 7: /usr/lib64/libnetsnmp.so.15(snmp_call_callbacks+0x15c) [0x7fb9a1f4cfbc]
 28715:20160722:075908.246 6: /usr/local/zabbix-server/sbin/zabbix_server(poller_thread+0x25d) [0x42a3fd]
 28715:20160722:075908.246 5: /usr/local/zabbix-server/sbin/zabbix_server(zbx_thread_start+0x46) [0x4812d6]
 28715:20160722:075908.246 4: /usr/local/zabbix-server/sbin/zabbix_server(MAIN_ZABBIX_ENTRY+0x36b) [0x41cbdb]
 28715:20160722:075908.246 3: /usr/local/zabbix-server/sbin/zabbix_server(daemon_start+0x1ae) [0x47f71e]
 28715:20160722:075908.247 2: /usr/local/zabbix-server/sbin/zabbix_server(main+0x391) [0x41d411]
 28715:20160722:075908.247 1: /lib64/libc.so.6(__libc_start_main+0xe6) [0x7fb99fe82c36]
 28715:20160722:075908.247 0: /usr/local/zabbix-server/sbin/zabbix_server() [0x417b29]

Interestingly, memory map reveals that two versions of both libssl and libcrypto are pulled as dependencies. They are very likely to be incompatible because major so-versions are changed for a reason.

 28715:20160722:075908.248 === Memory map: ===
...
 28715:20160722:075908.281 7fb99f86d000-7fb99f9e1000 r-xp 00000000 ca:00 246302                     /usr/lib64/libcrypto.so.0.9.8
 28715:20160722:075908.282 7fb99f9e1000-7fb99fbe0000 ---p 00174000 ca:00 246302                     /usr/lib64/libcrypto.so.0.9.8
 28715:20160722:075908.282 7fb99fbe0000-7fb99fbf0000 r--p 00173000 ca:00 246302                     /usr/lib64/libcrypto.so.0.9.8
 28715:20160722:075908.282 7fb99fbf0000-7fb99fc09000 rw-p 00183000 ca:00 246302                     /usr/lib64/libcrypto.so.0.9.8
...
 28715:20160722:075908.283 7fb99fc0d000-7fb99fc5d000 r-xp 00000000 ca:00 246327                     /usr/lib64/libssl.so.0.9.8
 28715:20160722:075908.283 7fb99fc5d000-7fb99fe5c000 ---p 00050000 ca:00 246327                     /usr/lib64/libssl.so.0.9.8
 28715:20160722:075908.284 7fb99fe5c000-7fb99fe5e000 r--p 0004f000 ca:00 246327                     /usr/lib64/libssl.so.0.9.8
 28715:20160722:075908.286 7fb99fe5e000-7fb99fe64000 rw-p 00051000 ca:00 246327                     /usr/lib64/libssl.so.0.9.8
...
 28715:20160722:075908.301 7fb9a1147000-7fb9a131c000 r-xp 00000000 ca:00 247114                     /usr/lib64/libcrypto.so.1.0.0
 28715:20160722:075908.301 7fb9a131c000-7fb9a151c000 ---p 001d5000 ca:00 247114                     /usr/lib64/libcrypto.so.1.0.0
 28715:20160722:075908.301 7fb9a151c000-7fb9a1537000 r--p 001d5000 ca:00 247114                     /usr/lib64/libcrypto.so.1.0.0
 28715:20160722:075908.302 7fb9a1537000-7fb9a1542000 rw-p 001f0000 ca:00 247114                     /usr/lib64/libcrypto.so.1.0.0
...
 28715:20160722:075908.302 7fb9a1546000-7fb9a15a7000 r-xp 00000000 ca:00 247349                     /usr/lib64/libssl.so.1.0.0
 28715:20160722:075908.304 7fb9a15a7000-7fb9a17a6000 ---p 00061000 ca:00 247349                     /usr/lib64/libssl.so.1.0.0
 28715:20160722:075908.305 7fb9a17a6000-7fb9a17aa000 r--p 00060000 ca:00 247349                     /usr/lib64/libssl.so.1.0.0
 28715:20160722:075908.305 7fb9a17aa000-7fb9a17b1000 rw-p 00064000 ca:00 247349                     /usr/lib64/libssl.so.1.0.0

Looks very similar to ZBX-10656. Switching to a different version of NetSNMP or OpenSSL libraries may help as a workaround.

Comment by Glebs Ivanovskis (Inactive) [ 2016 Jul 25 ]

Dear AnB!

Where did you get Zabbix server binary from? Was it from packages or compiled from source? And NetSNMP library? Can you show the output of ldd /usr/lib64/libnetsnmp.so.15.1.2?

Comment by André Burkhardt [ 2016 Jul 25 ]

Dear Glebs Ivanovskis,

thanks for your quick response! The zabbix binaries were compiled from source. In SLES 11 there is a special Update-Repository (SLE11-Security-Module) which provides openssl1 beside openssl0_9_8. So both libraries exists. The netsnmp Package from Distro depends on libcrypto.so.0.9.8.

An apparent working solution is to uninstall the netsnmp-Package from Distro and compile the netsnmp-package from source. This seems working for me. No more errors in zabbix_server.log.

output netsnmp from distro:

node:~ # ldd /usr/lib64/libnetsnmp.so.15.1.2
	linux-vdso.so.1 =>  (0x00007ffcf69d9000)
	libcrypto.so.0.9.8 => /usr/lib64/libcrypto.so.0.9.8 (0x00007f2ae2e17000)
	libc.so.6 => /lib64/libc.so.6 (0x00007f2ae2a9b000)
	libdl.so.2 => /lib64/libdl.so.2 (0x00007f2ae2896000)
	libz.so.1 => /lib64/libz.so.1 (0x00007f2ae2680000)
	/lib64/ld-linux-x86-64.so.2 (0x00007f2ae34a5000)

output netsnmp from source:

node:~ # ldd /usr/local/net-snmp/lib/libnetsnmp.so.30.0.3
	linux-vdso.so.1 =>  (0x00007ffc8f0d7000)
	librt.so.1 => /lib64/librt.so.1 (0x00007f730eca5000)
	libcrypto.so.1.0.0 => /usr/lib64/libcrypto.so.1.0.0 (0x00007f730e8a6000)
	libc.so.6 => /lib64/libc.so.6 (0x00007f730e529000)
	libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f730e30c000)
	/lib64/ld-linux-x86-64.so.2 (0x00007f730f19a000)
	libdl.so.2 => /lib64/libdl.so.2 (0x00007f730e108000)
	libz.so.1 => /lib64/libz.so.1 (0x00007f730def1000)

A new objdump-file is attached.

Whats your opinion?

Comment by Glebs Ivanovskis (Inactive) [ 2016 Jul 26 ]

I think you should be fine now.

We have recognized the problem. However, this is not something that can be fixed easily. The very least we can do is to document this case.

Generated at Fri Apr 04 03:20:35 EEST 2025 using Jira 9.12.4#9120004-sha1:625303b708afdb767e17cb2838290c41888e9ff0.