[ZBX-16223] Database monitoring with odbc and Oracle driver crashes Created: 2019 Jun 06  Updated: 2019 Jun 12  Resolved: 2019 Jun 12

Status: Closed
Project: ZABBIX BUGS AND ISSUES
Component/s: Server (S)
Affects Version/s: 4.0.7
Fix Version/s: None

Type: Problem report Priority: Critical
Reporter: Roger Assignee: Vladislavs Boborikins (Inactive)
Resolution: Duplicate Votes: 0
Labels: Oracle, crash, items, odbc
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

Ubuntu 18.04
Zabbix Server 4.0.7
UnixODBC 2.3.4-1.1ubuntu3
oracle-instantclient-odbc-10.2.0.5-1.x86_64


Attachments: File LD_log_isql.tar.gz     File LD_log_proxy.tar.gz     PNG File image-2019-06-06-15-45-16-992.png     PNG File image-2019-06-06-15-52-52-308.png     PNG File image-2019-06-06-18-12-49-953.png     PNG File image-2019-06-06-18-34-24-765.png     File libodbc.so.2.0.0.gz     File libpthread-2.27.so.gz     File libsqora.so.10.1.gz     File odbc.ini     File odbcinst.ini     File zabbix-server.service     File zabbix_proxy.gz     File zabbix_server.gz     Text File zabbix_server.log    
Issue Links:
Duplicate
duplicates ZBX-10897 crash using unixODBC with an Oracle d... Closed

 Description   

I have installed Zabbix server with "--unix-odbc" option.

I have installed the client for Oracle 10.2 databases with odbc, basic and sqlplus from https://www.oracle.com/technetwork/database/enterprise-edition/downloads/linuxx86-64soft-092277.html

I have configured the enviroment variables on /etc/environment and systemd zabbix service (see zabbix-server.service).

The file /etc/odbcinst.ini has the path to Oracle's driver (see odbcinst.ini).

With Zabbix user, if i execute "isql" it connects to database:

But on Zabbix, item of type "Database monitoring" with odbc does not work.

Everytime it's checked Zabbix shows a backtrace on the log (see zabbix_server.log). Zabbix server process does not stops after crash, it keeps running normally.

 



 Comments   
Comment by Andris Mednis [ 2019 Jun 06 ]

Backtrace from log:

  4934:20190606:152111.875 Got signal [signal:11(SIGSEGV),reason:1,refaddr:0x38]. Crashing ...
...
  4934:20190606:152111.876 === Backtrace: ===
  4934:20190606:152111.877 18: /usr/local/sbin/zabbix_server: poller #8 [got 0 values in 0.000045 sec, getting values](zbx_backtrace+0x4d) [0x55e45895310d]
  4934:20190606:152111.877 17: /usr/local/sbin/zabbix_server: poller #8 [got 0 values in 0.000045 sec, getting values](zbx_log_fatal_info+0x435) [0x55e458953665]
  4934:20190606:152111.877 16: /usr/local/sbin/zabbix_server: poller #8 [got 0 values in 0.000045 sec, getting values](+0xd2a86) [0x55e458953a86]
  4934:20190606:152111.877 15: /lib/x86_64-linux-gnu/libpthread.so.0(+0x12890) [0x7fb17d37f890]
  4934:20190606:152111.877 14: /usr/lib/oracle/10.2.0.5/client/lib/libsqora.so(bcoSQLSetStmtOption+0x13) [0x7fb1098016b3]
  4934:20190606:152111.877 13: /usr/lib/oracle/10.2.0.5/client/lib/libsqora.so(bccSQLSetStmtOption+0xdf) [0x7fb10981f16d]
  4934:20190606:152111.877 12: /usr/lib/oracle/10.2.0.5/client/lib/libsqora.so(SQLSetConnectAttrW+0x35d) [0x7fb10981cc05]
  4934:20190606:152111.877 11: /usr/lib/x86_64-linux-gnu/libodbc.so.2(+0xd249) [0x7fb17c37f249]
  4934:20190606:152111.877 10: /usr/lib/x86_64-linux-gnu/libodbc.so.2(SQLConnect+0x20a) [0x7fb17c38168a]
  4934:20190606:152111.877 9: /usr/local/sbin/zabbix_server: poller #8 [got 0 values in 0.000045 sec, getting values](zbx_odbc_connect+0x201) [0x55e4588ffb81]
  4934:20190606:152111.877 8: /usr/local/sbin/zabbix_server: poller #8 [got 0 values in 0.000045 sec, getting values](get_value_db+0xed) [0x55e4588dbead]
  4934:20190606:152111.878 7: /usr/local/sbin/zabbix_server: poller #8 [got 0 values in 0.000045 sec, getting values](+0x50939) [0x55e4588d1939]
  4934:20190606:152111.878 6: /usr/local/sbin/zabbix_server: poller #8 [got 0 values in 0.000045 sec, getting values](poller_thread+0xfb) [0x55e4588d1e4b]
  4934:20190606:152111.878 5: /usr/local/sbin/zabbix_server: poller #8 [got 0 values in 0.000045 sec, getting values](zbx_thread_start+0x27) [0x55e45895cce7]
  4934:20190606:152111.878 4: /usr/local/sbin/zabbix_server: poller #8 [got 0 values in 0.000045 sec, getting values](MAIN_ZABBIX_ENTRY+0x8f2) [0x55e4588c2f52]
  4934:20190606:152111.878 3: /usr/local/sbin/zabbix_server: poller #8 [got 0 values in 0.000045 sec, getting values](daemon_start+0x16d) [0x55e458952a5d]
  4934:20190606:152111.878 2: /usr/local/sbin/zabbix_server: poller #8 [got 0 values in 0.000045 sec, getting values](main+0x418) [0x55e4588c1e48]
  4934:20190606:152111.878 1: /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xe7) [0x7fb17a0f4b97]
  4934:20190606:152111.878 0: /usr/local/sbin/zabbix_server: poller #8 [got 0 values in 0.000045 sec, getting values](_start+0x2a) [0x55e4588c218a]
Comment by Andris Mednis [ 2019 Jun 06 ]

Could be a duplicate of ZBX-12159.

Take a look at ZBX-12159 - there is a workaround: compile proxy without OpenLDAP and curl, then do unixODBC checks from this proxy.

Comment by Roger [ 2019 Jun 06 ]

Must it be a proxy?

I have compiled server without ldap and curl to test but it keeps crashing with the same backtrace:

./configure --enable-java --enable-server --enable-agent --with-mysql --enable-ipv6 --with-net-snmp --with-openipmi --with-unixodbc --with-libxml2 --with-ssh2 --with-jabber --sysconfdir=/etc/zabbix/ --datadir=/usr/local/share/ --prefix=/usr/local/ --with-openssl

 

 23164:20190606:165957.897 === Backtrace: ===
 23164:20190606:165957.898 18: /usr/local/sbin/zabbix_server: poller #1 [got 0 values in 0.000037 sec, getting values](zbx_backtrace+0x4d) [0x557f4547816d]
 23164:20190606:165957.898 17: /usr/local/sbin/zabbix_server: poller #1 [got 0 values in 0.000037 sec, getting values](zbx_log_fatal_info+0x435) [0x557f454786c5]
 23164:20190606:165957.898 16: /usr/local/sbin/zabbix_server: poller #1 [got 0 values in 0.000037 sec, getting values](+0xbaae6) [0x557f45478ae6]
 23164:20190606:165957.898 15: /lib/x86_64-linux-gnu/libpthread.so.0(+0x12890) [0x7f5c7fe23890]
 23164:20190606:165957.899 14: /usr/lib/oracle/10.2.0.5/client/lib/libsqora.so(bcoSQLSetStmtOption+0x13) [0x7f5c0f8dc6b3]
 23164:20190606:165957.899 13: /usr/lib/oracle/10.2.0.5/client/lib/libsqora.so(bccSQLSetStmtOption+0xdf) [0x7f5c0f8fa16d]
 23164:20190606:165957.899 12: /usr/lib/oracle/10.2.0.5/client/lib/libsqora.so(SQLSetConnectAttrW+0x35d) [0x7f5c0f8f7c05]
 23164:20190606:165957.899 11: /usr/lib/x86_64-linux-gnu/libodbc.so.2(+0xd249) [0x7f5c7ee23249]
 23164:20190606:165957.899 10: /usr/lib/x86_64-linux-gnu/libodbc.so.2(SQLConnect+0x20a) [0x7f5c7ee2568a]
 23164:20190606:165957.899 9: /usr/local/sbin/zabbix_server: poller #1 [got 0 values in 0.000037 sec, getting values](zbx_odbc_connect+0x201) [0x557f45426d91]
 23164:20190606:165957.899 8: /usr/local/sbin/zabbix_server: poller #1 [got 0 values in 0.000037 sec, getting values](get_value_db+0xed) [0x557f4540eb1d]
 23164:20190606:165957.899 7: /usr/local/sbin/zabbix_server: poller #1 [got 0 values in 0.000037 sec, getting values](+0x4d857) [0x557f4540b857]
 23164:20190606:165957.899 6: /usr/local/sbin/zabbix_server: poller #1 [got 0 values in 0.000037 sec, getting values](poller_thread+0xfb) [0x557f4540bd5b]
 23164:20190606:165957.900 5: /usr/local/sbin/zabbix_server: poller #1 [got 0 values in 0.000037 sec, getting values](zbx_thread_start+0x27) [0x557f45481d47]
 23164:20190606:165957.900 4: /usr/local/sbin/zabbix_server: poller #1 [got 0 values in 0.000037 sec, getting values](MAIN_ZABBIX_ENTRY+0x8f2) [0x557f453fce52]
 23164:20190606:165957.900 3: /usr/local/sbin/zabbix_server: poller #1 [got 0 values in 0.000037 sec, getting values](daemon_start+0x16d) [0x557f45477abd]
 23164:20190606:165957.900 2: /usr/local/sbin/zabbix_server: poller #1 [got 0 values in 0.000037 sec, getting values](main+0x4f3) [0x557f453fbdb3]
 23164:20190606:165957.900 1: /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xe7) [0x7f5c7d277b97]
 23164:20190606:165957.900 0: /usr/local/sbin/zabbix_server: poller #1 [got 0 values in 0.000037 sec, getting values](_start+0x2a) [0x557f453fc08a]

Comment by Andris Mednis [ 2019 Jun 06 ]

Compiling without LDAP and cURL is good, but let's check did you get free of libldap library:

$ ldd zabbix_server | grep -i ldap
Comment by Roger [ 2019 Jun 06 ]

The result of this command is blank.

Comment by Roger [ 2019 Jun 06 ]

I have compiled and configured zabbix_proxy but it has the same error:

./configure --enable-proxy --with-unixodbc --with-mysql --sysconfdir=/etc/zabbix/ --datadir=/usr/local/share/ --prefix=/usr/local/

 

 28763:20190606:183159.846 === Backtrace: ===
 28763:20190606:183159.846 18: /usr/local/sbin/zabbix_proxy: poller #2 [got 0 values in 0.000030 sec, getting values](zbx_backtrace+0x4d) [0x5565ac413b5d]
 28763:20190606:183159.846 17: /usr/local/sbin/zabbix_proxy: poller #2 [got 0 values in 0.000030 sec, getting values](zbx_log_fatal_info+0x435) [0x5565ac4140b5]
 28763:20190606:183159.846 16: /usr/local/sbin/zabbix_proxy: poller #2 [got 0 values in 0.000030 sec, getting values](+0x9a4e6) [0x5565ac4144e6]
 28763:20190606:183159.846 15: /lib/x86_64-linux-gnu/libpthread.so.0(+0x12890) [0x7fb08da24890]
 28763:20190606:183159.846 14: /usr/lib/oracle/10.2.0.5/client/lib/libsqora.so(bcoSQLSetStmtOption+0x13) [0x7fb08e2d46b3]
 28763:20190606:183159.847 13: /usr/lib/oracle/10.2.0.5/client/lib/libsqora.so(bccSQLSetStmtOption+0xdf) [0x7fb08e2f216d]
 28763:20190606:183159.847 12: /usr/lib/oracle/10.2.0.5/client/lib/libsqora.so(SQLSetConnectAttrW+0x35d) [0x7fb08e2efc05]
 28763:20190606:183159.847 11: /usr/lib/x86_64-linux-gnu/libodbc.so.2(+0xd249) [0x7fb08cff3249]
 28763:20190606:183159.847 10: /usr/lib/x86_64-linux-gnu/libodbc.so.2(SQLConnect+0x20a) [0x7fb08cff568a]
 28763:20190606:183159.847 9: /usr/local/sbin/zabbix_proxy: poller #2 [got 0 values in 0.000030 sec, getting values](zbx_odbc_connect+0x201) [0x5565ac3c67c1]
 28763:20190606:183159.847 8: /usr/local/sbin/zabbix_proxy: poller #2 [got 0 values in 0.000030 sec, getting values](get_value_db+0xed) [0x5565ac3bc6ed]
 28763:20190606:183159.847 7: /usr/local/sbin/zabbix_proxy: poller #2 [got 0 values in 0.000030 sec, getting values](+0x3f448) [0x5565ac3b9448]
 28763:20190606:183159.847 6: /usr/local/sbin/zabbix_proxy: poller #2 [got 0 values in 0.000030 sec, getting values](poller_thread+0xeb) [0x5565ac3b993b]
 28763:20190606:183159.847 5: /usr/local/sbin/zabbix_proxy: poller #2 [got 0 values in 0.000030 sec, getting values](zbx_thread_start+0x27) [0x5565ac414cc7]
 28763:20190606:183159.847 4: /usr/local/sbin/zabbix_proxy: poller #2 [got 0 values in 0.000030 sec, getting values](MAIN_ZABBIX_ENTRY+0x58d) [0x5565ac3b16dd]
 28763:20190606:183159.847 3: /usr/local/sbin/zabbix_proxy: poller #2 [got 0 values in 0.000030 sec, getting values](daemon_start+0x16d) [0x5565ac4134ad]
 28763:20190606:183159.847 2: /usr/local/sbin/zabbix_proxy: poller #2 [got 0 values in 0.000030 sec, getting values](main+0x62d) [0x5565ac3b07dd]
 28763:20190606:183159.847 1: /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xe7) [0x7fb08c789b97]
 28763:20190606:183159.847 0: /usr/local/sbin/zabbix_proxy: poller #2 [got 0 values in 0.000030 sec, getting values](_start+0x2a) [0x5565ac3b0d2a]
Comment by Andris Mednis [ 2019 Jun 06 ]

Can you upload files

  • /usr/local/sbin/zabbix_server
  • /usr/local/sbin/zabbix_proxy
  • /usr/lib/x86_64-linux-gnu/libodbc.so.2
  • /usr/lib/oracle/10.2.0.5/client/lib/libsqora.so
  • /lib/x86_64-linux-gnu/libpthread.so.0

as gzipped attachments ?

Comment by Glebs Ivanovskis [ 2019 Jun 06 ]

rblancop, make sure you have all Oracle environment variables set for Zabbix processes.

From your zabbix-server.service:

Environment=ORACLE_HOME=/usr/lib/oracle/10.2.0.5/client64/lib

Shouldn't ORACLE_HOME end with "client64" (without "lib")?

Also see ZBX-10897, backtrace looks similar to yours.

Comment by Roger [ 2019 Jun 07 ]

I have created an environment file to both services and tested with and without "lib" in ORACLE_HOME variable, but there is no difference, isql works with both and zabbix keeps crashing.

root@mczabbix:~# sstrings -a /proc/12745/environ
LANG=es_ES.UTF-8
PATH=/usr/lib/oracle/10.2.0.5/client64/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games
INVOCATION_ID=46458fa908f64e819a16df8a8c59ed64
JOURNAL_STREAM=9:585708
LD_LIBRARY_PATH=/usr/lib/oracle/10.2.0.5/client64/lib
ORACLE_HOME=/usr/lib/oracle/10.2.0.5/client64

I have attached all the compressed files. The files libodbc.so.2, libsqora.so and libpthread.so.0 are symlinks to libodbc.so.2.0.0, libsqora.so.10.1 and libpthread-2.27.so

I will try to downgrade unixODBC to 2.2.14.

Comment by Roger [ 2019 Jun 07 ]

How can i compile zabbix_server with unixodbc in another folder?

I installed unixODBC 2.2.14 in "/opt/unixodbc2.2.14" and tried to compile zabbix with "-with-unixodbc=/opt/unixodbc2.2.14/lib/libodbc.so" and "-with-unixodbc=/opt/unixodbc2.2.14/lib", but it shows this error at configuration:

checking for odbc_config... /opt/unixodbc2.2.14/lib/libodbc.so
checking for SQLAllocHandle in -lodbc... no
configure: error: unixODBC library not found
Comment by Glebs Ivanovskis [ 2019 Jun 07 ]

I think you don't need to recompile, just set LD_LIBRARY_PATH=/opt/unixodbc2.2.14/lib for Zabbix server.

Comment by Roger [ 2019 Jun 07 ]

If i set it, zabbix and isql don't work:

Cannot connect to ODBC DSN: [SQL_ERROR]:[01000][0][[unixODBC][Driver Manager]Can't open lib '/usr/lib/oracle/10.2.0.5/client/lib/libsqora.so' : file not found]

 

Comment by Glebs Ivanovskis [ 2019 Jun 07 ]
LD_LIBRARY_PATH=/usr/lib/oracle/10.2.0.5/client64/lib:/opt/unixodbc2.2.14/lib
Comment by Roger [ 2019 Jun 07 ]

I tried it too, isql works but zabbix keeps crashing (it's like if it were using only the unixodbc 2.3.4).

Comment by Glebs Ivanovskis [ 2019 Jun 07 ]

Could you please run isql and Zabbix with LD_DEBUG=bindings (and probably also LD_DEBUG_OUTPUT=...)?

Comment by Roger [ 2019 Jun 07 ]

I configured both LD_DEBUG and LD_DEBUG_OUTPUT in environment and Zabbix proxy. Then connected as zabbix user and executed isql (it worked and connected to a database) (see LD_log_isql.tar.gz).

Finally configured Zabbix proxy environment variables, started the service and forced the check of the database item (it's the only item enabled) (see LD_log_proxy.tar.gz).

LD_LIBRARY_PATH is as you said: LD_LIBRARY_PATH=/usr/lib/oracle/10.2.0.5/client64/lib:/opt/unixodbc2.2.14/lib

 

Comment by Glebs Ivanovskis [ 2019 Jun 11 ]

Proxy still seems to use libodbc.so from /usr/lib/x86_64-linux-gnu/ instead of /opt/unixodbc2.2.14/lib/

Have you tried experimenting with different Oracle driver versions?

Comment by Roger [ 2019 Jun 12 ]

I finally made it work. It seems that the problem is with unixodbc 2.3.4, donwgrading to 2.2.14 resolves the problem.

 

From a fresh installed Debian 9.9 first installed all the zabbix dependences, the Oracle 10 drivers and compiled and installed  unixodbc2.2.14 with all default values:

/usr/local/src/unixODBC-2.2.14/configure --enable-gui=no

 

Added this links:

ln -s /usr/local/lib/libodbc.so.1.0.0 /usr/lib/x86_64-linux-gnu/libodbc.so.1
ln -s /usr/lib/oracle/10.2.0.5/client/lib/libsqora.so.10.1 /usr/lib/oracle/10.2.0.5/client/lib/libsqora.so

 

Compiled Zabbix server with all options (including ldap and libcurl)

./configure --enable-server --enable-agent --with-mysql --enable-ipv6 --with-net-snmp --with-libcurl --with-libxml2 --with-jabber --with-openipmi --with-unixodbc --with-ssh2 --with-openssl --with-ldap  --sysconfdir=/etc/zabbix/ --datadir=/usr/local/share/ --prefix=/usr/local/

 

And finally set the environment variables at the Zabbix Server systemd service (EnvironmentFile=-/etc/systemd/system/zabbix.environment):

PATH=/usr/lib/oracle/10.2.0.5/client64/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games
LD_LIBRARY_PATH=:/usr/lib/oracle/10.2.0.5/client64/lib
ORACLE_HOME=/usr/lib/oracle/10.2.0.5/client64/lib

 

On another server, after following this steps, i had to add this links:

ln -s /usr/local/lib/libodbcinst.so.1.0.0 /lib/x86_64-linux-gnu/libodbcinst.so.1
ln -s /usr/lib/oracle/10.2.0.5/client64/lib/libclntsh.so.10.1 /lib/x86_64-linux-gnu/libclntsh.so.10.1
ln -s /usr/lib/oracle/10.2.0.5/client64/lib/libnnz10.so /lib/x86_64-linux-gnu/libnnz10.so

Comment by Roger [ 2019 Jun 12 ]

I can't try with different Oracle driver.

With this configuration, in Ubuntu should it work too.

We can close the issue. Thank you!

Comment by Glebs Ivanovskis [ 2019 Jun 12 ]

I'm glad you've managed to find a solution to your problem!

I've added a comment with a link in ZBX-10897. Can someone please link these two issues and close this one as Duplicate?

Comment by Vladislavs Boborikins (Inactive) [ 2019 Jun 12 ]

The issue linked to ZBX-10897 and closed as duplicate.

Comment by Glebs Ivanovskis [ 2019 Jun 12 ]

Thank you!

Generated at Fri Mar 29 07:49:49 EET 2024 using Jira 9.12.4#9120004-sha1:625303b708afdb767e17cb2838290c41888e9ff0.