[ZBX-12159] Resolving TNS names via LDAP crash on Oracle ODBC monitoring Created: 2017 May 10  Updated: 2022 Oct 08

Status: Confirmed
Project: ZABBIX BUGS AND ISSUES
Component/s: Proxy (P), Server (S)
Affects Version/s: 3.2.6
Fix Version/s: None

Type: Problem report Priority: Trivial
Reporter: Alexey Pustovalov Assignee: Alexei Vladishev
Resolution: Unresolved Votes: 3
Labels: ldap, odbc, oracle
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

Oracle ODBC


Attachments: File binding_withot_ldap.log     File bindings.log     File ldap.ora     File odbc.ini     File odbcinst.ini     File oracle-instant-client.sh     Text File proof_of_concep_bindings.txt     File sqlnet.ora     File tnsnames.ora     File zabbix_server.tar.gz    
Issue Links:
Duplicate
is duplicated by ZBX-9166 zabbix server poller crashes Closed
is duplicated by ZBX-13010 Zabbix Server 3.4.3 Segmentation Faul... Closed
Sub-task
part of ZBXNEXT-3388 Load libraries in runtime with RTLD_L... Open
part of ZBX-9082 zabbix_server crash after create Data... Open
part of ZBX-12367 zabbix_proxy: poller #3 invalid pointer Open
part of ZBX-11017 zabbix_server crashes on start (Got s... Closed
part of ZBX-7665 Zabbix 2.2.1 ODBC monitor crash - in ... Closed
part of ZBX-10656 One child process died Closed
part of ZBX-10962 Crash/exit in poller performing ODBC ... Closed
part of ZBX-9655 Zabbix Server CRASH Closed
part of ZBX-10897 crash using unixODBC with an Oracle d... Closed
Team: Team A
Sprint: Sprint 7, Sprint 8, Sprint 9, Sprint 10, Sprint 11, Sprint 12, Sprint 13, Sprint 14, Sprint 15, Sprint 16, Sprint 17, Sprint 18, Sprint 20, Sprint 21, Sprint 26, Sprint 27, Sprint 28, Sprint 29, Sprint 30, Sprint 31, Sprint 37

 Description   

Zabbix server with Oracle ODBC monitoring and enabled resolving tns names entries via LDAP causes crash in Oracle ODBC library:

getentry.c:35: ldap_first_entry: Assertion `( (ld)->ldc->ldc_options.ldo_valid == 0x2 )' failed.

Current workaround is to compile proxy without OpenLDAP and curl, then do unixODBC checks from this proxy.



 Comments   
Comment by Glebs Ivanovskis (Inactive) [ 2017 May 10 ]

I would check for symbol conflicts in Zabbix server binary, whichever libraries linked to it and Oracle ODBC library.

Comment by Alexey Pustovalov [ 2017 May 10 ]

libldap is linked to both.

Comment by Glebs Ivanovskis (Inactive) [ 2017 May 10 ]

Different versions? What does ldd show for server and Oracle ODBC driver?

Comment by Glebs Ivanovskis (Inactive) [ 2017 May 11 ]

Few related links from the Internet (one, two, three) blame Oracle. I guess we should do the same, nothing to fix in Zabbix this time.

Comment by Andrea Biscuola (Inactive) [ 2017 May 11 ]

Checking the dependencies of the oracle client libraries. The only library that have a symbol named 'ldap_first_entry' is libclntsh.so.12.1.

$ readelf -s libclntsh.so.12.1|grep ldap_first_entry
 11768: 00000000014bdf60   112 FUNC    GLOBAL DEFAULT   11 ldap_first_entry
 44932: 00000000014bdf60   112 FUNC    GLOBAL DEFAULT   11 ldap_first_entry
 

And this library is not linked to libldap in any way:

$ readelf -d libclntsh.so.12.1

Dynamic section at offset 0x2bbb068 contains 33 entries:
  Tag        Type                         Name/Value
 0x0000000000000001 (NEEDED)             Shared library: [libmql1.so]
 0x0000000000000001 (NEEDED)             Shared library: [libipc1.so]
 0x0000000000000001 (NEEDED)             Shared library: [libnnz12.so]
 0x0000000000000001 (NEEDED)             Shared library: [libons.so]
 0x0000000000000001 (NEEDED)             Shared library: [libdl.so.2]
 0x0000000000000001 (NEEDED)             Shared library: [libm.so.6]
 0x0000000000000001 (NEEDED)             Shared library: [libpthread.so.0]
 0x0000000000000001 (NEEDED)             Shared library: [libnsl.so.1]
 0x0000000000000001 (NEEDED)             Shared library: [librt.so.1]
 0x0000000000000001 (NEEDED)             Shared library: [libc.so.6]
 0x0000000000000001 (NEEDED)             Shared library: [ld-linux-x86-64.so.2]
 0x0000000000000001 (NEEDED)             Shared library: [libaio.so.1]
 0x0000000000000001 (NEEDED)             Shared library: [libclntshcore.so.12.1]
 0x000000000000000e (SONAME)             Library soname: [libclntsh.so.12.1]
 0x000000000000000c (INIT)               0x4abd38
 0x000000000000000d (FINI)               0x2400390
 0x000000006ffffef5 (GNU_HASH)           0x190
 0x0000000000000005 (STRTAB)             0xcd5b8
 0x0000000000000006 (SYMTAB)             0x30f70
 0x000000000000000a (STRSZ)              411734 (bytes)
 0x000000000000000b (SYMENT)             24 (bytes)
 0x0000000000000003 (PLTGOT)             0x2dbe088
 0x0000000000000002 (PLTRELSZ)           380016 (bytes)
 0x0000000000000014 (PLTREL)             RELA
 0x0000000000000017 (JMPREL)             0x44f0c8
 0x0000000000000007 (RELA)               0x13efb8
 0x0000000000000008 (RELASZ)             3211536 (bytes)
 0x0000000000000009 (RELAENT)            24 (bytes)
 0x000000006ffffffe (VERNEED)            0x13ee98
 0x000000006fffffff (VERNEEDNUM)         6
 0x000000006ffffff0 (VERSYM)             0x131e0e
 0x000000006ffffff9 (RELACOUNT)          125813
 0x0000000000000000 (NULL)               0x0
 

Going further, libsqora.so.12.1 is linked with the previous library:

$ readelf -d libsqora.so.12.1

Dynamic section at offset 0x249040 contains 29 entries:
  Tag        Type                         Name/Value
 0x0000000000000001 (NEEDED)             Shared library: [libdl.so.2]
 0x0000000000000001 (NEEDED)             Shared library: [libm.so.6]
 0x0000000000000001 (NEEDED)             Shared library: [libpthread.so.0]
 0x0000000000000001 (NEEDED)             Shared library: [libnsl.so.1]
 0x0000000000000001 (NEEDED)             Shared library: [librt.so.1]
 0x0000000000000001 (NEEDED)             Shared library: [libclntsh.so.12.1]
 0x0000000000000001 (NEEDED)             Shared library: [libodbcinst.so.2]
 0x0000000000000001 (NEEDED)             Shared library: [libc.so.6]
 0x0000000000000001 (NEEDED)             Shared library: [ld-linux-x86-64.so.2]
 0x000000000000000e (SONAME)             Library soname: [libsqora.so.12.1]
 0x000000000000000c (INIT)               0x52ff0
 0x000000000000000d (FINI)               0x1cad00
 0x000000006ffffef5 (GNU_HASH)           0x190
 0x0000000000000005 (STRTAB)             0x13628
 0x0000000000000006 (SYMTAB)             0x4ad8
 0x000000000000000a (STRSZ)              28675 (bytes)
 0x000000000000000b (SYMENT)             24 (bytes)
 0x0000000000000003 (PLTGOT)             0x4499e8
 0x0000000000000002 (PLTRELSZ)           37392 (bytes)
 0x0000000000000014 (PLTREL)             RELA
 0x0000000000000017 (JMPREL)             0x49de0
 0x0000000000000007 (RELA)               0x1baa8
 0x0000000000000008 (RELASZ)             189240 (bytes)
 0x0000000000000009 (RELAENT)            24 (bytes)
 0x000000006ffffffe (VERNEED)            0x1b9c8
 0x000000006fffffff (VERNEEDNUM)         6
 0x000000006ffffff0 (VERSYM)             0x1a62c
 0x000000006ffffff9 (RELACOUNT)          7249
 0x0000000000000000 (NULL)               0x0
 

This confirm that the oracle driver have it's own ldap built-in functions. IMHO, when the zabbix server is started, the loader override the function address of the oracle driver built-in function with the one coming from libldap when resolving the dynamic linking.
For being sure i need to see the output of:

readelf -s ./zabbix_server

and

readelf -d ./zabbix_server

And after that as a solution we can see if we can write a linker script for resolving this conflict.

Comment by Glebs Ivanovskis (Inactive) [ 2017 May 11 ]

Another idea was to use objcopy with option --redefine-sym old=new to resolve symbol conflict.

Comment by Vladislavs Sokurenko [ 2017 May 26 ]

Unfortunately OpenLDAP and Oracle ODBC driver have same LDAP functions that conflict with each other, so they cannot be used at the same time at the moment.
However configuration entry --with-ldap is only needed to monitor LDAP availability, it is not needed by Oracle ODBC driver.

This means that current workaround is to compile server without --with-ldap flag if LDAP availability monitoring is not used.
If LDAP availability monitoring is needed then compile proxy --with-ldap, create additional host monitored by proxy and monitor LDAP availability via that proxy.

Comment by Glebs Ivanovskis (Inactive) [ 2017 May 29 ]

Proof of concept solution for trunk implemented in development branch svn://svn.zabbix.com/branches/dev/ZBX-12159 revision 68633. Let's see if it solves the problem of using LDAP and Oracle database monitoring simultaneously. vso, would you be so kind to test it in your environment?

Then we would need to solve the following questions:

  • Should we keep the ability to link with LDAP statically?

vso It does not work, there is a bug report ZBX-2076. Ideally I think we should fix it for the agent. So I suggest we remove this ability since it does not work and fix in ZBX-2076 if it is required.

  • Should we allow starting Zabbix without LDAP library if it was not found? (Currently loading LDAP is a must if Zabbix was configured and compiled --with-ldap.)

vso Current behavior with shared libraries is that when application starts it searches for shared libraries, I believe in common paths, if not found then error is shown.
I think that what must happen, also this would mean that for user nothing would change, still libraries are shared and loaded transparently for him. So ideally it's 100% same behavior as before and nothing breaks or changes from user point of view ?

Alternative is that we can mimic unixODBC behavior, so once LDAP is used then it becomes unsupported if library cannot be found and appropriate message in frontend is shown. There should be no flag --with-ldap. But this solution is bad since user will only know that library is missing when trying to use the library.

glebs.ivanovskis Yes, currently it's very much what linker does automatically on the start, but postponed a bit and with hiding of loaded library symbols from others. That we will still have two versions of a binary (2‍number of "on/off" dependencies, actually): with and without LDAP. Good thing is that we can compile without LDAP headers. If we do like UnixODBC (and I personally would prefer this approach) we will have one version of a binary for all possible enabled/disabled features (in the long run). But we would need all the headers to compile it, mandatory.

  • Should we let runtime linker load libldap.so and liblber.so from wherever path it looks into or should we specify exact path provided in configuration file or in --with-ldap option?

vso Same as previous step I believe we should try to make it work 100 % as before.

  • How to manage different "so"-versions of the library?

vso the same way it is currently done when building package. we can specify path from where to take library during compile time so user experience is the same. ( those are just some ideas about direction we should take/investigate)

glebs.ivanovskis It's a bit trickier than you think. libsomelib.so is usually a symlink to the highest version of all libsomelib.so.x available on the system. And in some cases libsomelib.so.x is in turn a symlink to libsomelib.so.x.y. NEEDED section of a binary usually stores libsomelib.so.x (check ldd output) in order not to link with binary incompatible libsomelib.so.z. If we simply try to load libsomelib.so we are risking loading binary incompatible version of a library and potentially crashing.

vso Does not crash anymore, bindings attached

see:

     32609:	binding file ./sbin/zabbix_server [0] to /lib/x86_64-linux-gnu/libdl.so.2 [0]: normal symbol `dlsym' [GLIBC_2.2.5]
     32609:	binding file /lib/x86_64-linux-gnu/libdl.so.2 [0] to /lib/x86_64-linux-gnu/libc.so.6 [0]: normal symbol `_dl_sym' [GLIBC_PRIVATE]
     32609:	binding file /usr/lib/x86_64-linux-gnu/libldap.so [0] to /usr/lib/x86_64-linux-gnu/libldap.so [0]: normal symbol `ldap_init'
     32609:	binding file /usr/lib/x86_64-linux-gnu/libldap.so [0] to /usr/lib/x86_64-linux-gnu/libldap.so [0]: normal symbol `ldap_search_s'
     32609:	binding file /usr/lib/x86_64-linux-gnu/libldap.so [0] to /usr/lib/x86_64-linux-gnu/libldap.so [0]: normal symbol `ldap_first_entry'
     32609:	binding file /usr/lib/x86_64-linux-gnu/libldap.so [0] to /usr/lib/x86_64-linux-gnu/libldap.so [0]: normal symbol `ldap_first_attribute'
     32609:	binding file /usr/lib/x86_64-linux-gnu/libldap.so [0] to /usr/lib/x86_64-linux-gnu/libldap.so [0]: normal symbol `ldap_get_values'
     32609:	binding file /usr/lib/x86_64-linux-gnu/libldap.so [0] to /usr/lib/x86_64-linux-gnu/libldap.so [0]: normal symbol `ldap_value_free'
     32609:	binding file /usr/lib/x86_64-linux-gnu/libldap.so [0] to /usr/lib/x86_64-linux-gnu/libldap.so [0]: normal symbol `ldap_memfree'
     32609:	binding file /usr/lib/x86_64-linux-gnu/libldap.so [0] to /usr/lib/x86_64-linux-gnu/libldap.so [0]: normal symbol `ldap_msgfree'
     32609:	binding file /usr/lib/x86_64-linux-gnu/libldap.so [0] to /usr/lib/x86_64-linux-gnu/libldap.so [0]: normal symbol `ldap_unbind'
     32609:	binding file /usr/lib/x86_64-linux-gnu/libldap.so [0] to /usr/lib/x86_64-linux-gnu/libldap.so [0]: normal symbol `ldap_err2string'
     32609:	binding file /usr/lib/x86_64-linux-gnu/liblber-2.4.so.2 [0] to /usr/lib/x86_64-linux-gnu/liblber-2.4.so.2 [0]: normal symbol `ber_free'

glebs.ivanovskis Most importantly:

$ cat proof_of_concep_bindings.txt | grep libclntsh | grep ldap
     32629:	binding file /usr/lib/oracle/12.2/client64/lib/libclntsh.so.12.1 [0] to /usr/lib/oracle/12.2/client64/lib/libclntsh.so.12.1 [0]: normal symbol `nzsupplfc_ldap_fetch_crl'
     32629:	binding file /usr/lib/oracle/12.2/client64/lib/libclntsh.so.12.1 [0] to /usr/lib/oracle/12.2/client64/lib/libclntsh.so.12.1 [0]: normal symbol `nzsuppliu_ldap_is_url'
     32629:	binding file /usr/lib/oracle/12.2/client64/lib/libclntsh.so.12.1 [0] to /usr/lib/oracle/12.2/client64/lib/libclntsh.so.12.1 [0]: normal symbol `nzsupplfc_ldap_fetch_crldp'
     32629:	binding file /usr/lib/oracle/12.2/client64/lib/libclntsh.so.12.1 [0] to /usr/lib/oracle/12.2/client64/lib/libclntsh.so.12.1 [0]: normal symbol `ldap_debug'
     32629:	binding file /usr/lib/oracle/12.2/client64/lib/libclntsh.so.12.1 [0] to /usr/lib/oracle/12.2/client64/lib/libclntsh.so.12.1 [0]: normal symbol `nlpauseldap'
     32629:	binding file /usr/lib/oracle/12.2/client64/lib/libclntsh.so.12.1 [0] to /usr/lib/oracle/12.2/client64/lib/libclntsh.so.12.1 [0]: normal symbol `ora_ldap_discover'
     32629:	binding file /usr/lib/oracle/12.2/client64/lib/libclntsh.so.12.1 [0] to /usr/lib/oracle/12.2/client64/lib/libclntsh.so.12.1 [0]: normal symbol `ldap_memfree'
     32629:	binding file /usr/lib/oracle/12.2/client64/lib/libclntsh.so.12.1 [0] to /usr/lib/oracle/12.2/client64/lib/libclntsh.so.12.1 [0]: normal symbol `ora_ldap_memfree'
     32629:	binding file /usr/lib/oracle/12.2/client64/lib/libclntsh.so.12.1 [0] to /usr/lib/oracle/12.2/client64/lib/libclntsh.so.12.1 [0]: normal symbol `ldap_open'
     32629:	binding file /usr/lib/oracle/12.2/client64/lib/libclntsh.so.12.1 [0] to /usr/lib/oracle/12.2/client64/lib/libclntsh.so.12.1 [0]: normal symbol `ora_ldap_open'
     32629:	binding file /usr/lib/oracle/12.2/client64/lib/libclntsh.so.12.1 [0] to /usr/lib/oracle/12.2/client64/lib/libclntsh.so.12.1 [0]: normal symbol `ldap_bind_s'
     32629:	binding file /usr/lib/oracle/12.2/client64/lib/libclntsh.so.12.1 [0] to /usr/lib/oracle/12.2/client64/lib/libclntsh.so.12.1 [0]: normal symbol `ora_ldap_bind_s'
     32629:	binding file /usr/lib/oracle/12.2/client64/lib/libclntsh.so.12.1 [0] to /usr/lib/oracle/12.2/client64/lib/libclntsh.so.12.1 [0]: normal symbol `ora_ldap_simple_bind_s'
     32629:	binding file /usr/lib/oracle/12.2/client64/lib/libclntsh.so.12.1 [0] to /usr/lib/oracle/12.2/client64/lib/libclntsh.so.12.1 [0]: normal symbol `ora_ldap_simple_bind'
     32629:	binding file /usr/lib/oracle/12.2/client64/lib/libclntsh.so.12.1 [0] to /usr/lib/oracle/12.2/client64/lib/libclntsh.so.12.1 [0]: normal symbol `ora_ldap_result'
     32629:	binding file /usr/lib/oracle/12.2/client64/lib/libclntsh.so.12.1 [0] to /usr/lib/oracle/12.2/client64/lib/libclntsh.so.12.1 [0]: normal symbol `ora_ldap_result2error'
     32629:	binding file /usr/lib/oracle/12.2/client64/lib/libclntsh.so.12.1 [0] to /usr/lib/oracle/12.2/client64/lib/libclntsh.so.12.1 [0]: normal symbol `ldap_search_st'
     32629:	binding file /usr/lib/oracle/12.2/client64/lib/libclntsh.so.12.1 [0] to /usr/lib/oracle/12.2/client64/lib/libclntsh.so.12.1 [0]: normal symbol `ora_ldap_search_st'
     32629:	binding file /usr/lib/oracle/12.2/client64/lib/libclntsh.so.12.1 [0] to /usr/lib/oracle/12.2/client64/lib/libclntsh.so.12.1 [0]: normal symbol `ldap_unbind'
     32629:	binding file /usr/lib/oracle/12.2/client64/lib/libclntsh.so.12.1 [0] to /usr/lib/oracle/12.2/client64/lib/libclntsh.so.12.1 [0]: normal symbol `ora_ldap_unbind'

Thank you for trying out!

Comment by Rostislav Palivoda [ 2017 Jul 14 ]

wiper please review and propose your way of solution.

Comment by Andris Zeila [ 2017 Jul 26 ]

Another way might be to load unixodbc dynamically with RTLD_DEEPBIND (supported since glibc 2.3.4) flag.

Place the lookup scope of the symbols in this library ahead of the global scope. This means that a self-contained library will use its own symbols in preference to global symbols with the same name contained in libraries that have already been loaded. This flag is not specified in POSIX.1-2001.

This seems to be working on my test examples, however it raises compatibility questions and also we would lose ability to specify unixodbc library location during compilation.

wiper I thought about patching unixodbc to use deep bind flag, but unixodbc uses libltdl rather than directly calling dlopen and libltdl does not support RTLD_DEEPBIND.

glebs.ivanovskis I'll add one more option to the party. Loading ODBC drivers ourselves. Datasource and driver configuration can be moved then from odbc.ini and odbcinst.ini files to Zabbix database.

Comment by Andris Mednis [ 2018 Feb 15 ]

As of rev.77676, specific DB dependencies are scattered over 22 files:

include/db.h				    HAVE_ORACLE				   HAVE_SQLITE3	HAVE_IBM_DB2
include/zbxdb.h				    HAVE_ORACLE				   HAVE_SQLITE3
src/libs/zbxdb/db.c			    HAVE_ORACLE	HAVE_MYSQL HAVE_POSTGRESQL HAVE_SQLITE3	HAVE_IBM_DB2
src/libs/zbxdbhigh/db.c			    HAVE_ORACLE	HAVE_MYSQL HAVE_POSTGRESQL HAVE_SQLITE3	HAVE_IBM_DB2
src/libs/zbxdbhigh/dbschema.c		    HAVE_ORACLE				   HAVE_SQLITE3	HAVE_IBM_DB2
src/libs/zbxdbhigh/proxy.c				HAVE_MYSQL					
src/libs/zbxdbupgrade/dbupgrade_2010.c	    HAVE_ORACLE	HAVE_MYSQL HAVE_POSTGRESQL HAVE_SQLITE3
src/libs/zbxdbupgrade/dbupgrade_2020.c						   HAVE_SQLITE3
src/libs/zbxdbupgrade/dbupgrade_2030.c	    HAVE_ORACLE				   HAVE_SQLITE3	HAVE_IBM_DB2
src/libs/zbxdbupgrade/dbupgrade_2040.c						   HAVE_SQLITE3
src/libs/zbxdbupgrade/dbupgrade_2050.c						   HAVE_SQLITE3
src/libs/zbxdbupgrade/dbupgrade_3000.c						   HAVE_SQLITE3
src/libs/zbxdbupgrade/dbupgrade_3010.c						   HAVE_SQLITE3
src/libs/zbxdbupgrade/dbupgrade_3020.c						   HAVE_SQLITE3
src/libs/zbxdbupgrade/dbupgrade_3030.c						   HAVE_SQLITE3	HAVE_IBM_DB2
src/libs/zbxdbupgrade/dbupgrade_3040.c						   HAVE_SQLITE3
src/libs/zbxdbupgrade/dbupgrade_3050.c						   HAVE_SQLITE3
src/libs/zbxdbupgrade/dbupgrade.c	    HAVE_ORACLE	HAVE_MYSQL HAVE_POSTGRESQL HAVE_SQLITE3	HAVE_IBM_DB2
src/libs/zbxdbupgrade/dbupgrade.h						   HAVE_SQLITE3
src/zabbix_server/alerter/alert_manager.c   HAVE_ORACLE	HAVE_MYSQL HAVE_POSTGRESQL 		HAVE_IBM_DB2
src/zabbix_server/housekeeper/housekeeper.c HAVE_ORACLE	HAVE_MYSQL HAVE_POSTGRESQL HAVE_SQLITE3	HAVE_IBM_DB2
src/zabbix_server/server.c							   HAVE_SQLITE3

The most "normal" is PostgreSQL: - only in 6 files.
The most scattered is SQLite3 - in 20 files.

Comment by Andris Mednis [ 2018 Feb 15 ]

Files dependening on crypto libraries:

include/comms.h				    HAVE_POLARSSL HAVE_GNUTLS HAVE_OPENSSL
include/dbcache.h			    HAVE_POLARSSL HAVE_GNUTLS HAVE_OPENSSL
include/zbxnix.h			    HAVE_POLARSSL HAVE_GNUTLS HAVE_OPENSSL
src/libs/zbxcomms/comms.c		    HAVE_POLARSSL HAVE_GNUTLS HAVE_OPENSSL
src/libs/zbxcrypto/tls.c		    HAVE_POLARSSL HAVE_GNUTLS HAVE_OPENSSL
src/libs/zbxcrypto/tls.h		    HAVE_POLARSSL HAVE_GNUTLS HAVE_OPENSSL
src/libs/zbxcrypto/tls_tcp_active.h	    HAVE_POLARSSL HAVE_GNUTLS HAVE_OPENSSL
src/libs/zbxcrypto/tls_tcp.h		    HAVE_POLARSSL HAVE_GNUTLS HAVE_OPENSSL
src/libs/zbxdbcache/dbconfig.c		    HAVE_POLARSSL HAVE_GNUTLS HAVE_OPENSSL
src/libs/zbxdbcache/dbconfig_dump.c	    HAVE_POLARSSL HAVE_GNUTLS HAVE_OPENSSL
src/libs/zbxdbcache/dbconfig.h		    HAVE_POLARSSL HAVE_GNUTLS HAVE_OPENSSL
src/libs/zbxdbcache/dbsync.c		    HAVE_POLARSSL HAVE_GNUTLS HAVE_OPENSSL
src/libs/zbxdbhigh/proxy.c		    HAVE_POLARSSL HAVE_GNUTLS HAVE_OPENSSL
src/libs/zbxnix/coredump.c		    HAVE_POLARSSL HAVE_GNUTLS HAVE_OPENSSL
src/libs/zbxnix/sighandler.c		    HAVE_POLARSSL HAVE_GNUTLS HAVE_OPENSSL
src/zabbix_agent/active.c		    HAVE_POLARSSL HAVE_GNUTLS HAVE_OPENSSL
src/zabbix_agent/listener.c		    HAVE_POLARSSL HAVE_GNUTLS HAVE_OPENSSL
src/zabbix_agent/zabbix_agentd.c	    HAVE_POLARSSL HAVE_GNUTLS HAVE_OPENSSL
src/zabbix_get/zabbix_get.c		    HAVE_POLARSSL HAVE_GNUTLS HAVE_OPENSSL
src/zabbix_proxy/datasender/datasender.c    HAVE_POLARSSL HAVE_GNUTLS HAVE_OPENSSL
src/zabbix_proxy/heart/heart.c		    HAVE_POLARSSL HAVE_GNUTLS HAVE_OPENSSL
src/zabbix_proxy/proxy.c		    HAVE_POLARSSL HAVE_GNUTLS HAVE_OPENSSL
src/zabbix_proxy/proxyconfig/proxyconfig.c  HAVE_POLARSSL HAVE_GNUTLS HAVE_OPENSSL
src/zabbix_proxy/servercomms.c		    HAVE_POLARSSL HAVE_GNUTLS HAVE_OPENSSL
src/zabbix_proxy/taskmanager/taskmanager.c  HAVE_POLARSSL HAVE_GNUTLS HAVE_OPENSSL
src/zabbix_sender/zabbix_sender.c	    HAVE_POLARSSL HAVE_GNUTLS HAVE_OPENSSL
src/zabbix_server/discoverer/discoverer.c   HAVE_POLARSSL HAVE_GNUTLS HAVE_OPENSSL
src/zabbix_server/escalator/escalator.c	    HAVE_POLARSSL HAVE_GNUTLS HAVE_OPENSSL
src/zabbix_server/poller/checks_agent.c	    HAVE_POLARSSL HAVE_GNUTLS HAVE_OPENSSL
src/zabbix_server/poller/poller.c	    HAVE_POLARSSL HAVE_GNUTLS HAVE_OPENSSL
src/zabbix_server/proxypoller/proxypoller.c HAVE_POLARSSL HAVE_GNUTLS HAVE_OPENSSL
src/zabbix_server/server.c		    HAVE_POLARSSL HAVE_GNUTLS HAVE_OPENSSL
src/zabbix_server/trapper/active.c	    HAVE_POLARSSL HAVE_GNUTLS HAVE_OPENSSL
src/zabbix_server/trapper/trapper.c	    HAVE_POLARSSL HAVE_GNUTLS HAVE_OPENSSL
tests/zbxmocktest.c			    HAVE_POLARSSL HAVE_GNUTLS HAVE_OPENSSL

In most of files the dependency is not specific to particular crypto library:

#if defined(HAVE_POLARSSL) || defined(HAVE_GNUTLS) || defined(HAVE_OPENSSL)

i.e. it is "with encryption" vs. "no-encryption", it could be replaced with more general #if defined(HAVE_ENCRYPTION), without name of crypto library used.

Only 4 files have really specific dependencies on crypto libraries::

src/libs/zbxcomms/comms.c	  specific on HAVE_OPENSSL
src/libs/zbxcrypto/tls.c	  specifc on all 3 crypto libraries
src/libs/zbxcrypto/tls.h	  specifc on all 3 crypto libraries
src/libs/zbxcrypto/tls_tcp.h	  specific on HAVE_OPENSSL
Comment by Andris Mednis [ 2018 Feb 15 ]

Files depending on HAVE_LIBCURL or HAVE_LIBXML2 (often used together):

include/sysinc.h				HAVE_LIBCURL
src/libs/zbxhistory/history_elastic.c		HAVE_LIBCURL
src/libs/zbxmedia/eztexting.c			HAVE_LIBCURL
src/libs/zbxsysinfo/simple/simple.c		HAVE_LIBCURL
src/zabbix_proxy/proxy.c			HAVE_LIBCURL HAVE_LIBXML2
src/zabbix_server/httppoller/httptest.c		HAVE_LIBCURL
src/zabbix_server/poller/checks_simple.c	HAVE_LIBCURL HAVE_LIBXML2
src/zabbix_server/poller/checks_simple_vmware.c	HAVE_LIBCURL HAVE_LIBXML2
src/zabbix_server/poller/checks_simple_vmware.h	HAVE_LIBCURL HAVE_LIBXML2
src/zabbix_server/preprocessor/item_preproc.c	   	     HAVE_LIBXML2
src/zabbix_server/server.c			HAVE_LIBCURL HAVE_LIBXML2
src/zabbix_server/vmware/vmware.c		HAVE_LIBCURL HAVE_LIBXML2
src/zabbix_server/vmware/vmware.h		HAVE_LIBCURL HAVE_LIBXML2
Comment by Andris Mednis [ 2018 Feb 15 ]

The simplest, most beautiful dependency is on HAVE_LDAP:

include/config.h   <---- defines existence of HAVE_LDAP
src/libs/zbxsysinfo/simple/simple.c

Only 1 file depends on HAVE_LDAP

Comment by Andris Mednis [ 2018 Feb 15 ]

Files depending on HAVE_SSH2:

src/zabbix_proxy/proxy.c
src/zabbix_server/poller/checks_ssh.c
src/zabbix_server/poller/checks_ssh.h
src/zabbix_server/poller/poller.c
src/zabbix_server/scripts/scripts.c
src/zabbix_server/server.c
Comment by Andris Mednis [ 2018 Feb 15 ]

Files depending on HAVE_OPENIPMI:

src/zabbix_proxy/proxy.c
src/zabbix_server/escalator/escalator.c
src/zabbix_server/ipmi/checks_ipmi.c
src/zabbix_server/ipmi/checks_ipmi.h
src/zabbix_server/ipmi/ipmi.c
src/zabbix_server/ipmi/ipmi_manager.c
src/zabbix_server/ipmi/ipmi_manager.h
src/zabbix_server/ipmi/ipmi_poller.c
src/zabbix_server/ipmi/ipmi_poller.h
src/zabbix_server/ipmi/ipmi_protocol.c
src/zabbix_server/ipmi/ipmi_protocol.h
src/zabbix_server/scripts/scripts.c
src/zabbix_server/server.c
Comment by Andris Mednis [ 2018 Feb 16 ]

Files depending on HAVE_JABBER:

include/zbxmedia.h
src/libs/zbxmedia/jabber.c
src/zabbix_server/alerter/alerter.c
src/zabbix_server/server.c
Comment by Andris Mednis [ 2018 Mar 02 ]

Experiment 1. Compile Zabbix server form trunk, r78280 with gcc 7.3 for Oracle DB and '--with-ldap':

$ export LD_LIBRARY_PATH='/usr/lib/oracle/12.2/client64/lib'
$ ./bootstrap.sh
$ ./configure --enable-server --enable-ipv6 \
     --with-oracle \
     --with-oracle-include=/usr/include/oracle/12.2/client64 \
     --with-oracle-lib=/usr/lib/oracle/12.2/client64/lib \
     --with-ldap \
    --prefix=`pwd`
...
Configuration:

  Detected OS:           linux-gnu
  Install path:          /home/zabbix40/ZBX-12159/trunk
  Compilation arch:      linux

  Compiler:              cc
  Compiler flags:         -g -O2 

  Library-specific flags:
    database:                -I/usr/include/oracle/12.2/client64  
    LDAP:                  -I/usr/include

  Enable server:         yes
  Server details:
    With database:         Oracle
    WEB Monitoring:        no
    Native Jabber:         no
    SNMP:                  no
    IPMI:                  no
    SSH:                   no
    TLS:                   no
    ODBC:                  no
    Linker flags:             -L/usr/lib/oracle/12.2/client64/lib           -L/usr/lib  -rdynamic   
    Libraries:                -lclntsh -lnnz12         -levent  -lldap -llber   -lm -ldl  -lresolv -lpcreposix -lpcre 
....
$ make dbschema
$ make install

Start server with LD_BIND_NOW and LD_DEBUG_OUTPUT options to see all activity and trouble from run-time linker:

$ LD_BIND_NOW=1 LD_DEBUG_OUTPUT=/tmp/ld_debug_out_default LD_DEBUG=all sbin/zabbix_server -c ~/ZBX-12159/zabbix_server.conf

Observe in debug output:

$ grep -E '(dynamically loaded|needed by|relocation|ldap_init)' ld_debug_out_default.24727 \
     | grep -v _ldap_ | grep -v ldap_init_SSL
     24727:     file=libclntsh.so.12.1 [0];  needed by sbin/zabbix_server [0]
     24727:     file=libnnz12.so [0];  needed by sbin/zabbix_server [0]
     24727:     file=libevent-2.1.so.6 [0];  needed by sbin/zabbix_server [0]
     24727:     file=libldap_r-2.4.so.2 [0];  needed by sbin/zabbix_server [0]
...
     24727:     relocation processing: /usr/lib/x86_64-linux-gnu/libldap_r-2.4.so.2
     24727:     symbol=ldap_init;  lookup in file=sbin/zabbix_server [0]
     24727:     symbol=ldap_init;  lookup in file=/usr/lib/oracle/12.2/client64/lib/libclntsh.so.12.1 [0]
     24727:     binding file /usr/lib/x86_64-linux-gnu/libldap_r-2.4.so.2 [0] to /usr/lib/oracle/12.2/client64/lib/libclntsh.so.12.1 [0]: normal symbol `ldap_init' [OPENLDAP_2.4_2]
     		<---- OpenLDAP library got a wrong ldap_init() from Oracle client.
...
     24727:     relocation processing: /usr/lib/oracle/12.2/client64/lib/libclntsh.so.12.1
     24727:     symbol=ldap_init;  lookup in file=sbin/zabbix_server [0]
     24727:     symbol=ldap_init;  lookup in file=/usr/lib/oracle/12.2/client64/lib/libclntsh.so.12.1 [0]
     24727:     binding file /usr/lib/oracle/12.2/client64/lib/libclntsh.so.12.1 [0] to /usr/lib/oracle/12.2/client64/lib/libclntsh.so.12.1 [0]: normal symbol `ldap_init'
     		<---- Oracle client got the right ldap_init() - its own built-in one.
...
     24727:     relocation processing: sbin/zabbix_server
     24727:     symbol=ldap_init;  lookup in file=sbin/zabbix_server [0]
     24727:     symbol=ldap_init;  lookup in file=/usr/lib/oracle/12.2/client64/lib/libclntsh.so.12.1 [0]
     24727:     binding file sbin/zabbix_server [0] to /usr/lib/oracle/12.2/client64/lib/libclntsh.so.12.1 [0]: normal symbol `ldap_init'
     		<---- Zabbix server got a wrong ldap_init() from Oracle client.

Zabbix server and OpenLDAP library got bindings to the wrong ldap_init().

Comment by Andris Mednis [ 2018 Mar 05 ]

Experiment 2: In Experiment 1 Oracle client was linked first, then OpenLDAP was linked:

...
    Libraries:     -lclntsh -lnnz12   -levent  -lldap -llber   -lm -ldl  -lresolv -lpcreposix -lpcre
...    

Let's try to change order of linking with Oracle client and OpenLDAP by putting OpenLDAP first.
For that we change configure.ac, put LDAP check before DB checks:

Index: configure.ac
===================================================================
--- configure.ac	(revision 78280)
+++ configure.ac	(working copy)
@@ -1174,6 +1174,25 @@
 	have_ipv6="yes"
 fi
 
+found_ldap="no"
+dnl Check for libLDAP [by default - skip]
+LIBLDAP_CHECK_CONFIG([no])
+if test "x$want_ldap" = "xyes"; then
+	if test "x$found_ldap" != "xyes"; then
+		AC_MSG_ERROR([Invalid LDAP directory - unable to find ldap.h])
+	fi
+fi
+SERVER_LDFLAGS="$SERVER_LDFLAGS $LDAP_LDFLAGS"
+SERVER_LIBS="$SERVER_LIBS $LDAP_LIBS"
+
+PROXY_LDFLAGS="$PROXY_LDFLAGS $LDAP_LDFLAGS"
+PROXY_LIBS="$PROXY_LIBS $LDAP_LIBS"
+
+AGENT_LDFLAGS="$AGENT_LDFLAGS $LDAP_LDFLAGS"
+AGENT_LIBS="$AGENT_LIBS $LDAP_LIBS"
+
+AC_SUBST(LDAP_CPPFLAGS)
+
 if test "x$server" = "xyes" || test "x$proxy" = "xyes"; then
 
 	have_multirow_insert="no"
@@ -1548,25 +1567,6 @@
 	fi
 fi
 
-found_ldap="no"
-dnl Check for libLDAP [by default - skip]
-LIBLDAP_CHECK_CONFIG([no])
-if test "x$want_ldap" = "xyes"; then
-	if test "x$found_ldap" != "xyes"; then
-		AC_MSG_ERROR([Invalid LDAP directory - unable to find ldap.h])
-	fi
-fi
-SERVER_LDFLAGS="$SERVER_LDFLAGS $LDAP_LDFLAGS"
-SERVER_LIBS="$SERVER_LIBS $LDAP_LIBS"
-
-PROXY_LDFLAGS="$PROXY_LDFLAGS $LDAP_LDFLAGS"
-PROXY_LIBS="$PROXY_LIBS $LDAP_LIBS"
-
-AGENT_LDFLAGS="$AGENT_LDFLAGS $LDAP_LDFLAGS"
-AGENT_LIBS="$AGENT_LIBS $LDAP_LIBS"
-
-AC_SUBST(LDAP_CPPFLAGS)
-
 found_curl="no"
 dnl Checking for libCurl [by default - skip]
 LIBCURL_CHECK_CONFIG(, [7.13.1], [],[])

Recompile:

$ export LD_LIBRARY_PATH='/usr/lib/oracle/12.2/client64/lib'
$ ./bootstrap.sh
$ ./configure --enable-server --enable-ipv6 \
        --with-oracle \
        --with-oracle-include=/usr/include/oracle/12.2/client64 \
        --with-oracle-lib=/usr/lib/oracle/12.2/client64/lib \
        --with-ldap --prefix=`pwd`
...
Configuration:

  Detected OS:           linux-gnu
  Install path:          /home/zabbix40/ZBX-12159/trunk
  Compilation arch:      linux

  Compiler:              cc
  Compiler flags:         -g -O2 

  Library-specific flags:
    database:                -I/usr/include/oracle/12.2/client64  
    LDAP:                  -I/usr/include

  Enable server:         yes
  Server details:
    With database:         Oracle
    WEB Monitoring:        no
    Native Jabber:         no
    SNMP:                  no
    IPMI:                  no
    SSH:                   no
    TLS:                   no
    ODBC:                  no
    Linker flags:           -L/usr/lib   -L/usr/lib/oracle/12.2/client64/lib            -rdynamic   
    Libraries:              -lldap -llber    -lclntsh -lnnz12         -levent   -lm -ldl  -lresolv -lpcreposix -lpcre 
--------------------------------------------------

Now '-lldap' is before '-lclntsh'.

$ make clean
$ make dbschema
$ make install

Start as

$ LD_BIND_NOW=1 LD_DEBUG_OUTPUT=/tmp/ld_debug_out_ldap_first LD_DEBUG=all sbin/zabbix_server -c ~/ZBX-12159/zabbix_server.conf

Observe in debug output:

$ grep -E '(dynamically loaded|needed by|relocation|ldap_init)' ld_debug_out_ldap_first.6112 \
     | grep -v _ldap_ | grep -v ldap_init_SSL
      6112:     file=libldap_r-2.4.so.2 [0];  needed by sbin/zabbix_server [0]
      6112:     file=liblber-2.4.so.2 [0];  needed by sbin/zabbix_server [0]
      6112:     file=libclntsh.so.12.1 [0];  needed by sbin/zabbix_server [0]
...
      6112:     relocation processing: /usr/lib/oracle/12.2/client64/lib/libclntsh.so.12.1
      6112:     symbol=ldap_init;  lookup in file=sbin/zabbix_server [0]
      6112:     symbol=ldap_init;  lookup in file=/usr/lib/x86_64-linux-gnu/libldap_r-2.4.so.2 [0]
      6112:     binding file /usr/lib/oracle/12.2/client64/lib/libclntsh.so.12.1 [0] to /usr/lib/x86_64-linux-gnu/libldap_r-2.4.so.2 [0]: normal symbol `ldap_init'
      		<---- Oracle client got a wrong ldap_init() from OpenLDAP.
...
      6112:     relocation processing: /usr/lib/x86_64-linux-gnu/libldap_r-2.4.so.2
      6112:     symbol=ldap_init;  lookup in file=sbin/zabbix_server [0]
      6112:     symbol=ldap_init;  lookup in file=/usr/lib/x86_64-linux-gnu/libldap_r-2.4.so.2 [0]
      6112:     binding file /usr/lib/x86_64-linux-gnu/libldap_r-2.4.so.2 [0] to /usr/lib/x86_64-linux-gnu/libldap_r-2.4.so.2 [0]: normal symbol `ldap_init' [OPENLDAP_2.4_2]
      		<---- OpenLDAP got the right ldap_init() from itself.
      6112:     relocation processing: sbin/zabbix_server
      6112:     symbol=ldap_init;  lookup in file=sbin/zabbix_server [0]
      6112:     symbol=ldap_init;  lookup in file=/usr/lib/x86_64-linux-gnu/libldap_r-2.4.so.2 [0]
      6112:     binding file sbin/zabbix_server [0] to /usr/lib/x86_64-linux-gnu/libldap_r-2.4.so.2 [0]: normal symbol `ldap_init' [OPENLDAP_2.4_2]
      		<---- Zabbix server got the right ldap_init() from OpenLDAP.
...

So, reversing of library order does not help.

Comment by Andris Mednis [ 2018 Mar 05 ]

Experiment 3. Let's try development branch svn://svn.zabbix.com/branches/dev/ZBX-12159 (thanks, Gleb!), where Oracle client is linked as before, but OpenLDAP is loaded dynamically via dlopen().

$ export LD_LIBRARY_PATH='/usr/lib/oracle/12.2/client64/lib'
$ ./bootstrap.sh
$ ./configure --enable-server --enable-ipv6 --with-oracle \
       --with-oracle-include=/usr/include/oracle/12.2/client64 \
       --with-oracle-lib=/usr/lib/oracle/12.2/client64/lib --with-ldap \
       --prefix=`pwd`
  Configuration:

  Detected OS:           linux-gnu
  Install path:          /home/zabbix40/ZBX-12159/ZBX-12159
  Compilation arch:      linux

  Compiler:              gcc
  Compiler flags:         -g -O2 

  Library-specific flags:
    database:                -I/usr/include/oracle/12.2/client64  
    LDAP:                  -I/usr/include

  Enable server:         yes
  Server details:
    With database:         Oracle
    WEB Monitoring:        no
    Native Jabber:         no
    SNMP:                  no
    IPMI:                  no
    SSH:                   no
    TLS:                   no
    ODBC:                  no
    Linker flags:             -L/usr/lib/oracle/12.2/client64/lib          -L/usr/lib  -rdynamic   
    Libraries:                -lclntsh -lnnz12          -lldap -llber   -lm -ldl  -lresolv -lpcreposix -lpcre 
...
$ make clean
$ make dbschema
$ make install

Start as

$ LD_BIND_NOW=1 LD_DEBUG_OUTPUT=/tmp/ld_debug_out_ZBX-12159 LD_DEBUG=all sbin/zabbix_server -c ~/ZBX-12159/zabbix_server.conf

Observe in debug output:

$ grep -E '(dynamically loaded|needed by|relocation|ldap_init)' ld_debug_out_ZBX-12159.18104 \
     | grep -v _ldap_ | grep -v ldap_init_SSL
     18104:     file=libclntsh.so.12.1 [0];  needed by sbin/zabbix_server [0]
     18104:     file=libnnz12.so [0];  needed by sbin/zabbix_server [0]
     18104:     file=libldap_r-2.4.so.2 [0];  needed by sbin/zabbix_server [0]
...
     18104:     relocation processing: /usr/lib/x86_64-linux-gnu/libldap_r-2.4.so.2
     18104:     symbol=ldap_init;  lookup in file=sbin/zabbix_server [0]
     18104:     symbol=ldap_init;  lookup in file=/usr/lib/oracle/12.2/client64/lib/libclntsh.so.12.1 [0]
     18104:     binding file /usr/lib/x86_64-linux-gnu/libldap_r-2.4.so.2 [0] to /usr/lib/oracle/12.2/client64/lib/libclntsh.so.12.1 [0]: normal symbol `ldap_init' [OPENLDAP_2.4_2]
             <---- OpenLDAP library got a wrong ldap_init() from Oracle client.
...
     18104:     relocation processing: /usr/lib/oracle/12.2/client64/lib/libclntsh.so.12.1
     18104:     symbol=ldap_init;  lookup in file=sbin/zabbix_server [0]
     18104:     symbol=ldap_init;  lookup in file=/usr/lib/oracle/12.2/client64/lib/libclntsh.so.12.1 [0]
     18104:     binding file /usr/lib/oracle/12.2/client64/lib/libclntsh.so.12.1 [0] to /usr/lib/oracle/12.2/client64/lib/libclntsh.so.12.1 [0]: normal symbol `ldap_init'
             <---- Oracle client got the right ldap_init() from itself.
     18104:     relocation processing: sbin/zabbix_server
....          
	     <---- Zabbix server fork()-ed
     18106:     file=libldap.so [0];  dynamically loaded by sbin/zabbix_server [0]
     	     <---- Zabbix server dlopen()-ed OpenLDAP library
     18106:     symbol=ldap_init;  lookup in file=/usr/lib/x86_64-linux-gnu/libldap_r-2.4.so.2 [0]
     18106:     binding file /usr/lib/x86_64-linux-gnu/libldap_r-2.4.so.2 [0] to /usr/lib/x86_64-linux-gnu/libldap_r-2.4.so.2 [0]: normal symbol `ldap_init'
     	     <---- OpenLDAP library got the right ldap_init().

Although OpenLDAP was loaded with dlopen(), it was also loaded as a dependency because of -lldap flag.

Comment by Andris Mednis [ 2018 Mar 05 ]

Experiment 4. Let's repeat Experiment 3 but remove linking to OpenLDAP library.
For that we edit configure.ac to skip LDAP flags, but keep the LDAP check to enable HAVE_LDAP in source code:

Index: configure.ac
===================================================================
--- configure.ac	(revision 78022)
+++ configure.ac	(working copy)
@@ -1480,14 +1480,14 @@
 		AC_MSG_ERROR([Invalid LDAP directory - unable to find ldap.h])
 	fi
 fi
-SERVER_LDFLAGS="$SERVER_LDFLAGS $LDAP_LDFLAGS"
-SERVER_LIBS="$SERVER_LIBS $LDAP_LIBS"
+SERVER_LDFLAGS="$SERVER_LDFLAGS"
+SERVER_LIBS="$SERVER_LIBS"
 
-PROXY_LDFLAGS="$PROXY_LDFLAGS $LDAP_LDFLAGS"
-PROXY_LIBS="$PROXY_LIBS $LDAP_LIBS"
+PROXY_LDFLAGS="$PROXY_LDFLAGS"
+PROXY_LIBS="$PROXY_LIBS"
 
-AGENT_LDFLAGS="$AGENT_LDFLAGS $LDAP_LDFLAGS"
-AGENT_LIBS="$AGENT_LIBS $LDAP_LIBS"
+AGENT_LDFLAGS="$AGENT_LDFLAGS"
+AGENT_LIBS="$AGENT_LIBS"
 
 AC_SUBST(LDAP_CPPFLAGS)

Recompile:

$ export LD_LIBRARY_PATH='/usr/lib/oracle/12.2/client64/lib'
$ ./bootstrap.sh
$ ./configure --enable-server --enable-ipv6 --with-oracle \
    --with-oracle-include=/usr/include/oracle/12.2/client64 \
    --with-oracle-lib=/usr/lib/oracle/12.2/client64/lib --with-ldap \
    --prefix=`pwd`
Configuration:
...
  Library-specific flags:
    database:                -I/usr/include/oracle/12.2/client64  
    LDAP:                  -I/usr/include

  Enable server:         yes
  Server details:
    With database:         Oracle
...
    Linker flags:             -L/usr/lib/oracle/12.2/client64/lib           -rdynamic   
    Libraries:                -lclntsh -lnnz12           -lm -ldl  -lresolv -lpcreposix -lpcre 
...

No more -lldap flag - no more dependency on OpenLDAP.

$ make clean
$ make dbschema
$ make install 2>&1 | tee my_make_install.out

Start as

$ LD_BIND_NOW=1 LD_DEBUG_OUTPUT=/tmp/ld_debug_out_ZBX-12159_noldap_linking LD_DEBUG=all sbin/zabbix_server -c ~/ZBX-12159/zabbix_server.conf

Observe in debug output:

$ grep -E '(dynamically loaded|needed by|relocation|ldap_init)' ld_debug_out_ZBX-12159_noldap_linking.664 \
     | grep -v _ldap_ | grep -v ldap_init_SSL
  664:     file=libclntsh.so.12.1 [0];  needed by sbin/zabbix_server [0]
  664:     file=libnnz12.so [0];  needed by sbin/zabbix_server [0]
  664:     file=libm.so.6 [0];  needed by sbin/zabbix_server [0]
  664:     file=libdl.so.2 [0];  needed by sbin/zabbix_server [0]
  664:     file=libresolv.so.2 [0];  needed by sbin/zabbix_server [0]
  664:     file=libpcreposix.so.3 [0];  needed by sbin/zabbix_server [0]
  664:     file=libpcre.so.3 [0];  needed by sbin/zabbix_server [0]
  664:     file=libc.so.6 [0];  needed by sbin/zabbix_server [0]
  664:     file=libmql1.so [0];  needed by /usr/lib/oracle/12.2/client64/lib/libclntsh.so.12.1 [0]
  664:     file=libipc1.so [0];  needed by /usr/lib/oracle/12.2/client64/lib/libclntsh.so.12.1 [0]
  664:     file=libons.so [0];  needed by /usr/lib/oracle/12.2/client64/lib/libclntsh.so.12.1 [0]
  664:     file=libpthread.so.0 [0];  needed by /usr/lib/oracle/12.2/client64/lib/libclntsh.so.12.1 [0]
  664:     file=libnsl.so.1 [0];  needed by /usr/lib/oracle/12.2/client64/lib/libclntsh.so.12.1 [0]
  664:     file=librt.so.1 [0];  needed by /usr/lib/oracle/12.2/client64/lib/libclntsh.so.12.1 [0]
  664:     file=libaio.so.1 [0];  needed by /usr/lib/oracle/12.2/client64/lib/libclntsh.so.12.1 [0]
  664:     file=libclntshcore.so.12.1 [0];  needed by /usr/lib/oracle/12.2/client64/lib/libclntsh.so.12.1 [0]
  664:     file=libgcc_s.so.1 [0];  needed by /usr/lib/oracle/12.2/client64/lib/libclntsh.so.12.1 [0]
        <---- OpenLDAP not yet loaded (good).
...
  664:     relocation processing: /usr/lib/oracle/12.2/client64/lib/libclntsh.so.12.1
  664:     symbol=ldap_init;  lookup in file=sbin/zabbix_server [0]
  664:     symbol=ldap_init;  lookup in file=/usr/lib/oracle/12.2/client64/lib/libclntsh.so.12.1 [0]
  664:     binding file /usr/lib/oracle/12.2/client64/lib/libclntsh.so.12.1 [0] to /usr/lib/oracle/12.2/client64/lib/libclntsh.so.12.1 [0]: normal symbol `ldap_init'
        <---- Oracle client got the right ldap_init() from itself.
  664:     relocation processing: sbin/zabbix_server
  664:     relocation processing: /lib64/ld-linux-x86-64.so.2
        <---- Zabbix server fork()-ed
  666:     file=libldap.so [0];  dynamically loaded by sbin/zabbix_server [0]
        <---- Zabbix server dlopen()-ed OpenLDAP library
  666:     file=liblber-2.4.so.2 [0];  needed by /usr/lib/x86_64-linux-gnu/libldap.so [0]
  666:     file=libsasl2.so.2 [0];  needed by /usr/lib/x86_64-linux-gnu/libldap.so [0]
  666:     file=libgnutls.so.30 [0];  needed by /usr/lib/x86_64-linux-gnu/libldap.so [0]
  666:     file=libz.so.1 [0];  needed by /usr/lib/x86_64-linux-gnu/libgnutls.so.30 [0]
  666:     file=libp11-kit.so.0 [0];  needed by /usr/lib/x86_64-linux-gnu/libgnutls.so.30 [0]
  666:     file=libidn2.so.0 [0];  needed by /usr/lib/x86_64-linux-gnu/libgnutls.so.30 [0]
  666:     file=libunistring.so.2 [0];  needed by /usr/lib/x86_64-linux-gnu/libgnutls.so.30 [0]
  666:     file=libtasn1.so.6 [0];  needed by /usr/lib/x86_64-linux-gnu/libgnutls.so.30 [0]
  666:     file=libnettle.so.6 [0];  needed by /usr/lib/x86_64-linux-gnu/libgnutls.so.30 [0]
  666:     file=libhogweed.so.4 [0];  needed by /usr/lib/x86_64-linux-gnu/libgnutls.so.30 [0]
  666:     file=libgmp.so.10 [0];  needed by /usr/lib/x86_64-linux-gnu/libgnutls.so.30 [0]
  666:     file=libffi.so.6 [0];  needed by /usr/lib/x86_64-linux-gnu/libp11-kit.so.0 [0]
        <---- OpenLDAP dependencies automatically loaded.
....
  666:     relocation processing: /usr/lib/x86_64-linux-gnu/libldap.so
  666:     symbol=ldap_init;  lookup in file=sbin/zabbix_server [0]
  666:     symbol=ldap_init;  lookup in file=/usr/lib/oracle/12.2/client64/lib/libclntsh.so.12.1 [0]
  666:     binding file /usr/lib/x86_64-linux-gnu/libldap.so [0] to /usr/lib/oracle/12.2/client64/lib/libclntsh.so.12.1 [0]: normal symbol `ldap_init' [OPENLDAP_2.4_2]
             <---- OpenLDAP library got a wrong ldap_init() from Oracle client.
  666:     symbol=ldap_init;  lookup in file=/usr/lib/x86_64-linux-gnu/libldap.so [0]
  666:     binding file /usr/lib/x86_64-linux-gnu/libldap.so [0] to /usr/lib/x86_64-linux-gnu/libldap.so [0]: normal symbol `ldap_init'
        <---- Yet another binding, but this one was correct.

Even dlopen() together with no linking of OpenLDAP did not solve the problem.

Comment by Andris Mednis [ 2018 Mar 05 ]

Experiment 5. Let's repeat Experiment 4, but add RTLD_DEEPBIND to dlopen():
So, the change from original is:

Index: configure.ac
===================================================================
--- configure.ac	(revision 78022)
+++ configure.ac	(working copy)
@@ -1480,14 +1480,14 @@
 		AC_MSG_ERROR([Invalid LDAP directory - unable to find ldap.h])
 	fi
 fi
-SERVER_LDFLAGS="$SERVER_LDFLAGS $LDAP_LDFLAGS"
-SERVER_LIBS="$SERVER_LIBS $LDAP_LIBS"
+SERVER_LDFLAGS="$SERVER_LDFLAGS"
+SERVER_LIBS="$SERVER_LIBS"
 
-PROXY_LDFLAGS="$PROXY_LDFLAGS $LDAP_LDFLAGS"
-PROXY_LIBS="$PROXY_LIBS $LDAP_LIBS"
+PROXY_LDFLAGS="$PROXY_LDFLAGS"
+PROXY_LIBS="$PROXY_LIBS"
 
-AGENT_LDFLAGS="$AGENT_LDFLAGS $LDAP_LDFLAGS"
-AGENT_LIBS="$AGENT_LIBS $LDAP_LIBS"
+AGENT_LDFLAGS="$AGENT_LDFLAGS"
+AGENT_LIBS="$AGENT_LIBS"
 
 AC_SUBST(LDAP_CPPFLAGS)
 
Index: src/libs/zbxldap/ldap.c
===================================================================
--- src/libs/zbxldap/ldap.c	(revision 78022)
+++ src/libs/zbxldap/ldap.c	(working copy)
@@ -141,7 +141,7 @@
 {
 	const zbx_lib_sym_t	*symbol;
 
-	if (NULL == (*library = dlopen(path, RTLD_NOW)))
+	if (NULL == (*library = dlopen(path, RTLD_NOW | RTLD_DEEPBIND)))
 	{
 		*error = zbx_dsprintf(*error, "cannot open \"%s\" library: %s", path, dlerror());
 		return FAIL;

Recompile and run:

$ export LD_LIBRARY_PATH='/usr/lib/oracle/12.2/client64/lib'
$ ./bootstrap.sh
$ ./configure --enable-server --enable-ipv6 --with-oracle --with-oracle-include=/usr/include/oracle/12.2/client64 --with-oracle-lib=/usr/lib/oracle/12.2/client64/lib --with-ldap \
    --prefix=`pwd` 2>&1 | tee my_configure.out
...
  Library-specific flags:
    database:                -I/usr/include/oracle/12.2/client64  
    LDAP:                  -I/usr/include

  Enable server:         yes
  Server details:
    With database:         Oracle
...    
    Linker flags:             -L/usr/lib/oracle/12.2/client64/lib           -rdynamic   
    Libraries:                -lclntsh -lnnz12           -lm -ldl  -lresolv -lpcreposix -lpcre 
...
$ make clean
$ make dbschema
$ make install 2>&1 | tee my_make_install.out
$ LD_BIND_NOW=1 LD_DEBUG_OUTPUT=/tmp/ld_debug_out_ZBX-12159_noldap_linking_with_RTLD_DEEPBIND LD_DEBUG=all sbin/zabbix_server -c ~/ZBX-12159/zabbix_server.conf

Observe in debug output:

$ grep -E '(dynamically loaded|needed by|relocation|ldap_init)' ld_debug_out_ZBX-12159_noldap_linking_with_RTLD_DEEPBIND.12426 \
     | grep -v _ldap_ | grep -v ldap_init_SSL
...
12426:     relocation processing: /usr/lib/oracle/12.2/client64/lib/libclntsh.so.12.1
12426:     symbol=ldap_init;  lookup in file=sbin/zabbix_server [0]
12426:     symbol=ldap_init;  lookup in file=/usr/lib/oracle/12.2/client64/lib/libclntsh.so.12.1 [0]
12426:     binding file /usr/lib/oracle/12.2/client64/lib/libclntsh.so.12.1 [0] to /usr/lib/oracle/12.2/client64/lib/libclntsh.so.12.1 [0]: normal symbol `ldap_init'
        <---- Oracle client got the right ldap_init() from itself.
12426:     relocation processing: sbin/zabbix_server
12426:     relocation processing: /lib64/ld-linux-x86-64.so.2
        <---- Zabbix server fork()-ed
12428:     file=libldap.so [0];  dynamically loaded by sbin/zabbix_server [0]
        <---- Zabbix server dlopen()-ed OpenLDAP library
12428:     file=liblber-2.4.so.2 [0];  needed by /usr/lib/x86_64-linux-gnu/libldap.so [0]
12428:     file=libsasl2.so.2 [0];  needed by /usr/lib/x86_64-linux-gnu/libldap.so [0]
12428:     file=libgnutls.so.30 [0];  needed by /usr/lib/x86_64-linux-gnu/libldap.so [0]
....
12428:     relocation processing: /usr/lib/x86_64-linux-gnu/libldap.so
12428:     symbol=ldap_init;  lookup in file=/usr/lib/x86_64-linux-gnu/libldap.so [0]
12428:     binding file /usr/lib/x86_64-linux-gnu/libldap.so [0] to /usr/lib/x86_64-linux-gnu/libldap.so [0]: normal symbol `ldap_init' [OPENLDAP_2.4_2]
        <---- OpenLDAP library got the right ldap_init() from itself.
12428:     symbol=ldap_init;  lookup in file=/usr/lib/x86_64-linux-gnu/libldap.so [0]
12428:     binding file /usr/lib/x86_64-linux-gnu/libldap.so [0] to /usr/lib/x86_64-linux-gnu/libldap.so [0]: normal symbol `ldap_init'
        <---- OpenLDAP library got the right ldap_init() from itself.
...

RTLD_DEEPBIND worked as expected and helped, but it has drawbacks (e.g. not portable - glibc specific).
With 2 conflicting libraries and only one of them opened with dlopen() we still need RTLD_DEEPBIND to solve the problem.
dlopen()/dlsym() helps Zabbix server to get the required function from the right library. Another task is to ensure that each conflicting library built-in symbols have priority over same-name symbols from its "competitors". Probably it can be done only with dlopen()-ing both conflicting libraries (as RTLD_DEEPBIND is not welcome).

Comment by Andris Mednis [ 2018 Mar 05 ]

So far we have explored the conflict between Oracle client and OpenLDAP libraries.
This issue is about problems with the third one - Oracle ODBC library.
Looking at its dependencies shows:

$ export LD_LIBRARY_PATH='/usr/lib/oracle/12.2/client64/lib'
$ ldd /usr/lib/oracle/12.2/client64/lib/libsqora.so.12.1
	linux-vdso.so.1 (0x00007ffc9f9cd000)
	libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007ff6f94dc000)
	libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007ff6f9149000)
	libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007ff6f8f2b000)
	libnsl.so.1 => /lib/x86_64-linux-gnu/libnsl.so.1 (0x00007ff6f8d13000)
	librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007ff6f8b0b000)
	libaio.so.1 => /lib/x86_64-linux-gnu/libaio.so.1 (0x00007ff6f8909000)
	libresolv.so.2 => /lib/x86_64-linux-gnu/libresolv.so.2 (0x00007ff6f86f2000)
	libclntsh.so.12.1 => /usr/lib/oracle/12.2/client64/lib/libclntsh.so.12.1 (0x00007ff6f4c4d000)
	libclntshcore.so.12.1 => /usr/lib/oracle/12.2/client64/lib/libclntshcore.so.12.1 (0x00007ff6f467f000)
	libodbcinst.so.2 => /usr/lib/x86_64-linux-gnu/libodbcinst.so.2 (0x00007ff6f446a000)
	libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007ff6f40b0000)
	libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007ff6f3e98000)
	/lib64/ld-linux-x86-64.so.2 (0x00007ff6f99a5000)
	libmql1.so => /usr/lib/oracle/12.2/client64/lib/libmql1.so (0x00007ff6f3c21000)
	libipc1.so => /usr/lib/oracle/12.2/client64/lib/libipc1.so (0x00007ff6f37ee000)
	libnnz12.so => /usr/lib/oracle/12.2/client64/lib/libnnz12.so (0x00007ff6f30a5000)
	libons.so => /usr/lib/oracle/12.2/client64/lib/libons.so (0x00007ff6f2e57000)
	libltdl.so.7 => /usr/lib/x86_64-linux-gnu/libltdl.so.7 (0x00007ff6f2c4d000)

So, Oracle ODBC library depends on libclntsh.so.12.1, which we already explored in details.
This matches with earlier comments - Oracle ODBC library libsqora.so.12.1 causes loading of Oracle libclntsh.so.12.1 which clashes with OpenLDAP library libldap.
Seems like all 3 libraries have to be wrapped into dlopen(): OpenLDAP's libldap, Oracle's libclntsh.so.12.1 and libsqora.so.12.1 to cover 8 use cases (Zabbix with Oracle or non-Oracle DB, Zabbix with or without LDAP, Zabbix with and without Oracle ODBC).

Comment by Andris Mednis [ 2018 Mar 07 ]

If we try to configure server as

$ export LD_LIBRARY_PATH='/usr/lib/oracle/12.2/client64/lib'
$ ./configure --enable-server --enable-ipv6 --with-oracle \
   --with-oracle-include=/usr/include/oracle/12.2/client64 \
   --with-oracle-lib=/usr/lib/oracle/12.2/client64/lib \
   --with-jabber --with-libxml2 --with-unixodbc --with-net-snmp --with-ssh2 \
   --with-openipmi --with-libevent --with-libpcre --with-openssl --with-ldap \
   --with-libcurl --with-iconv

to get as many libraries as possible (we got 68) , then it an be seen that Oracle client library

libclntsh.so.12.1 => /usr/lib/oracle/12.2/client64/lib/libclntsh.so.12.1

exports symbols which also can be found in (with some examples)

libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 <--- adler32(), adler32_combine(), compress() crc32()
libldap_r-2.4.so.2 => /usr/lib/x86_64-linux-gnu/libldap_r-2.4.so.2 <--- ldap_init()
libgssapi_krb5.so.2 => /usr/lib/x86_64-linux-gnu/libgssapi_krb5.so.2 <--- gss_canonicalize_name()
libkrb5.so.3 => /usr/lib/x86_64-linux-gnu/libkrb5.so.3 <--- decode_krb5_as_req()
libcom_err.so.2 => /lib/x86_64-linux-gnu/libcom_err.so.2 <--- add_error_table()
liblber-2.4.so.2 => /usr/lib/x86_64-linux-gnu/liblber-2.4.so.2 <--- ber_alloc_t(), ber_scanf()

So, it is not only a conflict between Oracle client and OpenLDAP library.

Compiling without --with-ldap does not break dependency on OpenLDAP library.
'-with-ldap' and '-with-libcurl' should be dropped to avoid dependency on OpenLDAP library.

Comment by Andris Mednis [ 2018 Mar 08 ]

Oracle full client also does not help - it exports ldap_init() and other functions like Oracle Instant client.

Comment by Andris Mednis [ 2018 Mar 08 ]

As an experiment Zabbix server can be compiled to use ldap_*() functions from Oracle Instant client:

Index: m4/ldap.m4
===================================================================
--- m4/ldap.m4	(revision 78478)
+++ m4/ldap.m4	(working copy)
@@ -70,9 +70,12 @@
                        LDAP_LIBS=" -lgnutls -lpthread -lsasl2 $LDAP_LIBS"
                fi
 
+		LDAP_INCDIR=/usr/include/oracle/12.2/client64
+		LDAP_LIBDIR=/usr/lib/oracle/12.2/client64/lib
+
                LDAP_CPPFLAGS="-I$LDAP_INCDIR"
                LDAP_LDFLAGS="-L$LDAP_LIBDIR"
-               LDAP_LIBS="-lldap -llber $LDAP_LIBS"
+               LDAP_LIBS="$LDAP_LIBS"
 
                found_ldap="yes"
                AC_DEFINE(HAVE_LDAP,1,[Define to 1 if LDAP should be enabled.])
Index: src/libs/zbxsysinfo/simple/simple.c
===================================================================
--- src/libs/zbxsysinfo/simple/simple.c	(revision 78478)
+++ src/libs/zbxsysinfo/simple/simple.c	(working copy)
@@ -31,10 +31,6 @@
 #	include <ldap.h>
 #endif
 
-#ifdef HAVE_LBER_H
-#	include <lber.h>
-#endif
-
 ZBX_METRIC	parameters_simple[] =
 /*      KEY                     FLAG		FUNCTION        	TEST PARAMETERS */
 {

and

$ ./bootstrap.sh
$ ./configure --enable-server --enable-ipv6 --with-oracle --with-oracle-include=/usr/include/oracle/12.2/client64 \
    --with-oracle-lib=/usr/lib/oracle/12.2/client64/lib --with-jabber --with-libxml2 --with-unixodbc --with-net-snmp \
    --with-ssh2 --with-openipmi --with-libevent --with-libpcre --with-openssl --with-ldap --with-libcurl \
    --with-iconv --prefix=`pwd`

but libcurl still brings in dependency on OpenLDAP.

Comment by Andris Mednis [ 2018 Mar 08 ]

Compiling Zabbix server with C++ compiler also does not help:

$ CC=g++ CFLAGS='-fpermissive -g -O2' ./configure --enable-server --enable-ipv6 --with-oracle \
    --with-oracle-include=/usr/include/oracle/12.2/client64 -\
    -with-oracle-lib=/usr/lib/oracle/12.2/client64/lib --with-jabber --with-libxml2 --with-unixodbc \
    --with-net-snmp --with-ssh2 --with-openipmi --with-libevent --with-libpcre --with-openssl \
    --with-ldap --with-libcurl \
    --with-iconv --prefix=`pwd`
Comment by Vladislavs Sokurenko [ 2021 May 25 ]

Note that after ZBXNEXT-782 pollers no longer use database connection, could it help if pollers are forked before database connection is made in main process ?

Comment by Glebs Ivanovskis [ 2021 May 25 ]

I would say no, because this change does not change dependencies of the main Zabbix server/proxy binary and these are what are searched first:

       Symbol references in the shared object are resolved using (in
       order): symbols in the link map of objects loaded for the main
       program and its dependencies; symbols in shared objects (and
       their dependencies) that were previously opened with dlopen()
       using the RTLD_GLOBAL flag; and definitions in the shared object
       itself (and any dependencies that were loaded for that object).
Comment by Vladislavs Sokurenko [ 2021 May 25 ]

Agree with cyclone unfortunately it does not help alone but maybe something could be built on it, for example allowing to specify C API library file for Oracle and loading it with dlopen or some other solution. As solution no longer should take into account that symbol conflict happens within same process

Generated at Fri Mar 29 01:50:27 EET 2024 using Jira 9.12.4#9120004-sha1:625303b708afdb767e17cb2838290c41888e9ff0.