Basically I could reopen ZBX-8733 but decided to create separate issue to provide clean description.

In netsnmp.m4 we have such a line:

_full_libnetsnmp_libs="`$_libnetsnmp_config --libs` -lcrypto"

Note hard coded " -lcrypto".
I several times was not very sure is it correct approach or not (I investigated the m4 script previously for different cases).
Now I faced with a problem and I believe the unconditional linking "-lcrypto" is incorrect.

On an ARM powered small board I have problems with running zabbix proxy when it compiled with net-snmp.
It just crashes during start.
In log file I don't see that pollers were started, backtrace shows nothing useful.

Proxy binary with snmp support (which crashes):

# ldd ./zabbix_proxy_test
        libsqlite3.so.0 => /usr/lib/libsqlite3.so.0 (0x4014f000)
        libnetsnmp.so.20 => /usr/lib/libnetsnmp.so.20 (0x401c8000)
        libcrypto.so.1.0.0 => /usr/lib/libcrypto.so.1.0.0 (0x40273000)
        libm.so.6 => /lib/libm.so.6 (0x403ca000)
        libdl.so.2 => /lib/libdl.so.2 (0x400dd000)
        librt.so.1 => /lib/librt.so.1 (0x4001e000)
        libresolv.so.2 => /lib/libresolv.so.2 (0x4009b000)
        libgcc_s.so.1 => /lib/libgcc_s.so.1 (0x400e8000)
        libc.so.6 => /lib/libc.so.6 (0x4043d000)
        libpthread.so.0 => /lib/libpthread.so.0 (0x400fa000)
        libcrypto.so.0.9.8 => /usr/lib/libcrypto.so.0.9.8 (0x40564000)
        libssl.so.0.9.8 => /usr/lib/libssl.so.0.9.8 (0x4068b000)
        /lib/ld-linux.so.3 (0x400b7000)

note two different libcrypto linked here: libcrypto.so.1.0.0 and libcrypto.so.0.9.8

without:

# ldd ./zabbix_proxy
        libsqlite3.so.0 => /usr/lib/libsqlite3.so.0 (0x400cb000)
        libm.so.6 => /lib/libm.so.6 (0x40144000)
        libdl.so.2 => /lib/libdl.so.2 (0x401b7000)
        librt.so.1 => /lib/librt.so.1 (0x401c2000)
        libresolv.so.2 => /lib/libresolv.so.2 (0x40013000)
        libgcc_s.so.1 => /lib/libgcc_s.so.1 (0x4007b000)
        libc.so.6 => /lib/libc.so.6 (0x401d1000)
        libpthread.so.0 => /lib/libpthread.so.0 (0x402f8000)
        /lib/ld-linux.so.3 (0x400a5000)

note both libraries are missing.

On my own (another device/distro) raspberry pi:

root@kot2:~# ldd ./zabbix_proxy
        /usr/lib/arm-linux-gnueabihf/libcofi_rpi.so (0xb6fa4000)
        libsqlite3.so.0 => /usr/lib/arm-linux-gnueabihf/libsqlite3.so.0 (0xb6f03000)
        libnetsnmp.so.15 => /usr/lib/libnetsnmp.so.15 (0xb6e4e000)
        libm.so.6 => /lib/arm-linux-gnueabihf/libm.so.6 (0xb6ddd000)
        libdl.so.2 => /lib/arm-linux-gnueabihf/libdl.so.2 (0xb6dd2000)
        librt.so.1 => /lib/arm-linux-gnueabihf/librt.so.1 (0xb6dc3000)
        libresolv.so.2 => /lib/arm-linux-gnueabihf/libresolv.so.2 (0xb6daf000)
        libc.so.6 => /lib/arm-linux-gnueabihf/libc.so.6 (0xb6c7f000)
        libpthread.so.0 => /lib/arm-linux-gnueabihf/libpthread.so.0 (0xb6c60000)                                                                                                                                                
        libgcc_s.so.1 => /lib/arm-linux-gnueabihf/libgcc_s.so.1 (0xb6c38000)                                                                                                                                                    
        /lib/ld-linux-armhf.so.3 (0xb6fb2000)
        libcrypto.so.1.0.0 => /usr/lib/arm-linux-gnueabihf/libcrypto.so.1.0.0 (0xb6ad4000)
        libz.so.1 => /lib/arm-linux-gnueabihf/libz.so.1 (0xb6ab6000)            

note only one library.

The ARM device is:

# cat /etc/angstrom-version
Angstrom 2010.7-test-20110215
Built from branch: org.openembedded.dev
Revision: d4f2f04
Target system: arm-angstrom-linux-gnueabi

OpenSSL packages on this host:

# opkg list-installed | grep ssl
libssl0.9.8 - 0.9.8m-r12.0.5
libssl1.0.0 - 1.0.0d-r14.0.6
openssl - 1.0.0d-r14.0.6
openssl-conf - 1.0.0d-r14.0.6
openssl-dev - 1.0.0d-r14.0.6

Related packages info:

# opkg info libssl0.9.8
Package: libssl0.9.8
Version: 0.9.8m-r12.0.5
Depends: libcrypto0.9.8, libc6
Provides:
Status: install ok installed
Section: libs/network
Architecture: armv7a
Maintainer: Angstrom Developers <[email protected]>
MD5Sum: 3334bd369453470367d1d3bca0a5cc62
Size: 124918
Filename: libssl0.9.8_0.9.8m-r12.0.5_armv7a.ipk
Description: Secure Socket Layer (SSL) binary and related cryptographic tools.
Installed-Time: 1340084421

# opkg info libssl1.0.0
Package: libssl1.0.0
Version: 1.0.0d-r14.0.6
Depends: libcrypto1.0.0 (>= 1.0.0d), libc6 (>= 2.9)
Provides:
Status: install ok installed
Section: libs/network
Architecture: armv7a
Maintainer: Angstrom Developers <[email protected]>
MD5Sum: e61d0bb495714c3f2d89a290eb16a332
Size: 138864
Filename: libssl1.0.0_1.0.0d-r14.0.6_armv7a.ipk
Description: Secure Socket Layer (SSL) binary and related cryptographic tools.
Installed-Time: 1322731543

Let's check why are these libraries for:

# opkg whatdepends libssl0.9.8
Root set:
  libssl0.9.8
What depends on root set
        net-snmp-server svn-r5.0.5      depends on libssl0.9.8
        libnetsnmp20 svn-r5.0.5 depends on libssl0.9.8
        net-snmp-client svn-r5.0.5      depends on libnetsnmp20
        net-snmp-dev svn-r5.0.5 depends on net-snmp-client

# opkg whatdepends libssl1.0.0
Root set:
  libssl1.0.0
What depends on root set
        openssl 1.0.0d-r14.0.6  depends on libssl1.0.0 (>= 1.0.0d)
        openssl-dev 1.0.0d-r14.0.6      depends on openssl (= 1.0.0d-r14.0.6)
        python-io 2.6.6-ml12.2.6        depends on libssl1.0.0 (>= 1.0.0d)
        python-textutils 2.6.6-ml12.2.6 depends on python-io
        python-crypt 2.6.6-ml12.2.6     depends on libssl1.0.0 (>= 1.0.0d)
        python-debugger 2.6.6-ml12.2.6  depends on python-io
        python-email 2.6.6-ml12.2.6     depends on python-io
        python-pyserial 2.4-ml2.6       depends on python-io
        python-mmap 2.6.6-ml12.2.6      depends on python-io
        python-smtpd 2.6.6-ml12.2.6     depends on python-email
        python-sqlite3 2.6.6-ml12.2.6   depends on python-crypt
        python-mime 2.6.6-ml12.2.6      depends on python-io
        python-sqlite3-tests 2.6.6-ml12.2.6     depends on python-sqlite3
        python-multiprocessing 2.6.6-ml12.2.6   depends on python-io
        python-logging 2.6.6-ml12.2.6   depends on python-io
        python-subprocess 2.6.6-ml12.2.6        depends on python-io
        python-modules 2.6.6-ml12.2.6   depends on python-crypt
        python-profile 2.6.6-ml12.2.6   depends on python-textutils
        python-pickle 2.6.6-ml12.2.6    depends on python-io
        python-terminal 2.6.6-ml12.2.6  depends on python-io
        python-doctest 2.6.6-ml12.2.6   depends on python-io

So both libraries required on the ARM host, the are many specific python scripts for monitoring.

Then on the ARM host I performed experiments with a dummy 5-lines C code (test.c).
Trying to link libraries in different ways:

# gcc test.c
# ldd ./a.out
        libc.so.6 => /lib/libc.so.6 (0x40201000)
        /lib/ld-linux.so.3 (0x40051000)

# gcc test.c -lcrypto
# ldd ./a.out
        libcrypto.so.1.0.0 => /usr/lib/libcrypto.so.1.0.0 (0x400a3000)
        libc.so.6 => /lib/libc.so.6 (0x401fa000)
        libdl.so.2 => /lib/libdl.so.2 (0x4008f000)
        libgcc_s.so.1 => /lib/libgcc_s.so.1 (0x4006f000)
        /lib/ld-linux.so.3 (0x4001b000)

# gcc test.c -lnetsnmp
# ldd ./a.out
        libnetsnmp.so.20 => /usr/lib/libnetsnmp.so.20 (0x401b1000)
        libc.so.6 => /lib/libc.so.6 (0x4025c000)
        libcrypto.so.0.9.8 => /usr/lib/libcrypto.so.0.9.8 (0x40383000)
        libssl.so.0.9.8 => /usr/lib/libssl.so.0.9.8 (0x404aa000)
        /lib/ld-linux.so.3 (0x40022000)
        libdl.so.2 => /lib/libdl.so.2 (0x400b0000)
        libgcc_s.so.1 => /lib/libgcc_s.so.1 (0x400c7000)

So far everything as expected.

Let's try to link both libraries:

# gcc test.c -lcrypto -lnetsnmp
/usr/lib/gcc/arm-angstrom-linux-gnueabi/4.3.3/../../../../arm-angstrom-linux-gnueabi/bin/ld: warning: libcrypto.so.0.9.8, needed by /usr/lib/gcc/arm-angstrom-linux-gnueabi/4.3.3/../../../libnetsnmp.so, may conflict with libcrypto.so.1.0.0

We see an important warning.
The same warning we see when compiling zabbix proxy - for last gcc invocation (when linking).
But the line with gcc invocation is very-very long and the warning is almost NOT noticeable, unfortunately.

Some extra detailed info

# ls -l /usr/lib/libcrypto*
lrwxrwxrwx 1 root root      18 Sep 30 01:18 /usr/lib/libcrypto.so -> libcrypto.so.1.0.0
-rwxr-xr-x 1 root root 1163896 Mar 31  2010 /usr/lib/libcrypto.so.0.9.8
-rwxr-xr-x 1 root root 1360252 Sep  2 20:54 /usr/lib/libcrypto.so.1.0.0

# ldd /usr/lib/libnetsnmp.so.20
        libcrypto.so.0.9.8 => /usr/lib/libcrypto.so.0.9.8 (0x40269000)
        libssl.so.0.9.8 => /usr/lib/libssl.so.0.9.8 (0x40170000)
        libc.so.6 => /lib/libc.so.6 (0x40390000)
        libdl.so.2 => /lib/libdl.so.2 (0x401b7000)
        libgcc_s.so.1 => /lib/libgcc_s.so.1 (0x40098000)
        /lib/ld-linux.so.3 (0x40072000)

Then I tried to remove " -lcrypto" from configure script (or m4 script) and proxy was compiled successfully and operates well.

I'm not sure why " -lcrypto" is there in m4 script.
Maybe it was required for some distro some time ago or for some specific platform like AIX etc.

My investigation shows that the " -lcrypto" originally was used only for testing in AC_CHECK_LIB macro, but not for linking.
First time it was added for linking was in r4051, a line became:
SNMP_LIBS="$SNMP_LIBS -lcrypto -lnetsnmp"

Then it has been removed in r4054, then again added in r4055.
And the line was not changed after r4055 and looks like I already mentioned:
_full_libnetsnmp_libs="`$_libnetsnmp_config --libs` -lcrypto"

Please consider to remove " -lcrypto" from netsnmp.m4

Some changes regarding "lcrypto" also are in revs 216, 256.
They may help to understand why it was needed at all.

Commit message for r4051 says:

$ svn log -v -c 4051
------------------------------------------------------------------------
r4051 | alex | 2007-04-23 19:22:06 +0300 (Mon, 23 Apr 2007) | 2 lines
Changed paths:
   M /trunk/configure.in
   M /trunk/m4/netsnmp.m4

More updates for static compile.

------------------------------------------------------------------------
$ svn di -c 4051
Index: m4/netsnmp.m4
===================================================================
--- m4/netsnmp.m4       (revision 4050)
+++ m4/netsnmp.m4       (revision 4051)
@@ -73,7 +73,7 @@
                fi
 
                AC_CHECK_LIB(netsnmp, main, , AC_MSG_ERROR([Not found netsnmp library]))
-               SNMP_LIBS="$SNMP_LIBS -lnetsnmp"
+               SNMP_LIBS="$SNMP_LIBS -lcrypto -lnetsnmp"
 
                _full_libnetsnmp_cflags=`$_libnetsnmp_config --cflags`
                for i in $_full_libnetsnmp_cflags; do
Index: configure.in
===================================================================
--- configure.in        (revision 4050)
+++ configure.in        (revision 4051)
@@ -638,7 +638,8 @@
        fi
 fi
 CFLAGS="${CFLAGS} ${SNMP_CPPFLAGS}"
-LDFLAGS="${LDFLAGS} ${SNMP_LDFLAGS}"
+LDFLAGS="${LDFLAGS}"
+LIBS="${LIBS} ${SNMP_LDFLAGS}"
 
 dnl Check for UCD-SNMP [by default - skip]
 LIBSNMP_CHECK_CONFIG([no])

Apparently, it had to do with static compilation.

I have the same issue as mentioned in this issue, my issue can be seen in ZBX-8845. I tried the following changes on my FreeBSD server but its does not work. I still have two libcrypto libraries linked to zabbix_server binary and it still crashes.

Riaan, this is probably because "net-snmp-config --libs" contains "-lcrypto". So even if we remove "-lcrypto" that is hardcoded into our m4 script, this will not solve your issue.

For instance, this is what it gives on our FreeBSD 9.2:

$ net-snmp-config --libs
-L/usr/lib -lm -lkvm -ldevstat -lpkg -L/usr/local/lib -L/usr/local/lib -lnetsnmp -lcrypto -lelf -lssp_nonshared

In r4053, the line was like this:

SNMP_LIBS="$SNMP_LIBS -lcrypto -lnetsnmp"

In r4055, it became like this (and was only used for static compilation):

_full_libnetsnmp_libs="`$_libnetsnmp_config --libs` -lcrypto"

After that it started to be used for regular compilation, too.

Unfortunately, it is hard for Alexei to remember what the issue with static compilation was about. However, note that r4053 does not use "net-snmp-config --libs", whereas r4055 does.

We can make two guesses here: (a) one is that "net-snmp-config --libs" did not output "-lcrypto" at the time, but it was required, and (b) that "net-snmp-config --libs" did output "-lcrypto" and the unnecessarily hardcoded "-lcrypto" is due to copy-paste.

Here is the output on our Solaris 10:

$ net-snmp-config --version
5.0.9
$ net-snmp-config --libs
-R../lib -L/usr/sfw/lib -lnetsnmp -lgen -lpkcs11 -lkstat -lelf -lm -ldl -lnsl -lsocket -ladm

$ ./net-snmp-config --version
5.7.2
$ ./net-snmp-config --libs
-L/export/home/zabbix/net-snmp-5.7.2-bin/lib -lnetsnmp -lrt -lkstat -lcrypto -lnsl -lsocket -lelf -lm

Recent versions of Net-SNMP, even Net-SNMP 5.0.11.2 from 2008 (see http://www.net-snmp.org/download.html), do output "-lcrypto", at least on my machine. So it seems that nowadays hardcoded "-lcrypto" is not needed, regardless of which reason above was true.

Thus, we can remove the hardcoded "-lcrypto", and that is available in development branch svn://svn.zabbix.com/branches/dev/ZBX-8849 . However, as noted above, that is not going to solve everyone's problem.

Thanks for the reply. I did update configure and netsnmp.m4 to remove the hard-coded -lcrypto and recompiled but zabbix_server still contains two libcrypto libraries. I tested and found that one version is required by openssl and the other by libnetsnmp.

ldd /usr/local/lib/libssl.so
/usr/local/lib/libssl.so:
libcrypto.so.8 => /usr/local/lib/libcrypto.so.8 (0x801268000)
libthr.so.3 => /lib/libthr.so.3 (0x801642000)
libc.so.7 => /lib/libc.so.7 (0x80081b000)

ldd /usr/local/lib/libnetsnmp.so.30
/usr/local/lib/libnetsnmp.so.30:
libm.so.5 => /lib/libm.so.5 (0x8012da000)
libkvm.so.5 => /lib/libkvm.so.5 (0x8014fb000)
libdevstat.so.7 => /lib/libdevstat.so.7 (0x801703000)
libpkg.so.3 => /usr/local/lib/libpkg.so.3 (0x801908000)
libcrypto.so.6 => /lib/libcrypto.so.6 (0x801c8f000)
libc.so.7 => /lib/libc.so.7 (0x80081b000)
libelf.so.1 => /usr/lib/libelf.so.1 (0x802037000)
libfetch.so.6 => /usr/lib/libfetch.so.6 (0x802250000)
libutil.so.9 => /lib/libutil.so.9 (0x802462000)
libssl.so.6 => /usr/lib/libssl.so.6 (0x802675000)
libjail.so.1 => /lib/libjail.so.1 (0x8028cb000)
libarchive.so.5 => /usr/lib/libarchive.so.5 (0x802ad0000)
libz.so.6 => /lib/libz.so.6 (0x802d11000)
libbz2.so.4 => /usr/lib/libbz2.so.4 (0x802f25000)
liblzma.so.5 => /usr/lib/liblzma.so.5 (0x803135000)
libbsdxml.so.4 => /lib/libbsdxml.so.4 (0x803358000)

Riaan, note that if you edit netsnmp.m4 then you must to run bootstrap.sh script (which available in svn, but missing in sources tar.gz) which will regenerate configure script and that's enough.
Otherwise you need to just edit existing configure script (do the same changes).
Then you can recompile zabbix server.

Also, show please output of command "net-snmp-config --libs"

On my Debian/testing x64 it provides "-lnetsnmp -lcrypto -lm"
On Debian/stable 7.5 on RasberryPi it provides "-lnetsnmp" only.

The output of the command as requested:
net-snmp-config --libs
-L/usr/lib -lm -lkvm -ldevstat -lpkg -L/usr/local/lib -L/usr/local/lib -lnetsnmp -lcrypto -lelf -lssp_nonshared

Riaan, I'm attaching hackish_netsnmp.m4.
Rename it to netsnmp.m4 and place it to m4 folder. Then run bootstrap.sh and then configure, compile as usually.
This dirty hacked file don't use net-snmp-config command at all, I just removed some block of code in that file. Custom path for --with-net-snmp configure option is ignored.
I had to prepare such hack in the past to work around a cross compilation configure issue (will be posted in future as a ZBX improvement request).

In such way -lcrypto will not be included for linking stage and I hope you will get working zabbix binary.

My guess is that "net-snmp-config --libs" outputs libraries needed for static compilation. Indeed, why otherwise would it mention -lcrypto and -lm?

By looking at the available options to "net-snmp-config", it seems that "net-snmp-config --netsnmp-libs" outputs just what is required to link with dynamic Net-SNMP library.

In fact, if we look at m4/libcurl.m4, then we will see that a similar distinction is made for cURL: we use "curl-config --libs" for dynamic compilation and add "curl-config --static-libs" for static compilation.

Usage of "net-snmp-config --netsnmp-libs", as well as a slight cleaning up, is available in development branch svn://svn.zabbix.com/branches/dev/ZBX-8849 .

Note that "--netsnmp-libs" option to "net-snmp-config" is available in 5.0.11.2 (from year 2008), and maybe older versions, too.

Successfully tested

Fixed in pre-2.2.8 r50303, pre-2.4.2 r50304, and pre-2.5.0 (trunk) r50305.