[ZBX-16223] Database monitoring with odbc and Oracle driver crashes Created: 2019 Jun 06 Updated: 2019 Jun 12 Resolved: 2019 Jun 12 |
|
Status: | Closed |
Project: | ZABBIX BUGS AND ISSUES |
Component/s: | Server (S) |
Affects Version/s: | 4.0.7 |
Fix Version/s: | None |
Type: | Problem report | Priority: | Critical |
Reporter: | Roger | Assignee: | Vladislavs Boborikins (Inactive) |
Resolution: | Duplicate | Votes: | 0 |
Labels: | Oracle, crash, items, odbc | ||
Remaining Estimate: | Not Specified | ||
Time Spent: | Not Specified | ||
Original Estimate: | Not Specified | ||
Environment: |
Ubuntu 18.04 |
Attachments: | LD_log_isql.tar.gz LD_log_proxy.tar.gz image-2019-06-06-15-45-16-992.png image-2019-06-06-15-52-52-308.png image-2019-06-06-18-12-49-953.png image-2019-06-06-18-34-24-765.png libodbc.so.2.0.0.gz libpthread-2.27.so.gz libsqora.so.10.1.gz odbc.ini odbcinst.ini zabbix-server.service zabbix_proxy.gz zabbix_server.gz zabbix_server.log | ||||||||
Issue Links: |
|
Description |
I have installed Zabbix server with "--unix-odbc" option. I have installed the client for Oracle 10.2 databases with odbc, basic and sqlplus from https://www.oracle.com/technetwork/database/enterprise-edition/downloads/linuxx86-64soft-092277.html
I have configured the enviroment variables on /etc/environment and systemd zabbix service (see zabbix-server.service). The file /etc/odbcinst.ini has the path to Oracle's driver (see odbcinst.ini). With Zabbix user, if i execute "isql" it connects to database:
But on Zabbix, item of type "Database monitoring" with odbc does not work. Everytime it's checked Zabbix shows a backtrace on the log (see zabbix_server.log). Zabbix server process does not stops after crash, it keeps running normally.
|
Comments |
Comment by Andris Mednis [ 2019 Jun 06 ] |
Backtrace from log: 4934:20190606:152111.875 Got signal [signal:11(SIGSEGV),reason:1,refaddr:0x38]. Crashing ... ... 4934:20190606:152111.876 === Backtrace: === 4934:20190606:152111.877 18: /usr/local/sbin/zabbix_server: poller #8 [got 0 values in 0.000045 sec, getting values](zbx_backtrace+0x4d) [0x55e45895310d] 4934:20190606:152111.877 17: /usr/local/sbin/zabbix_server: poller #8 [got 0 values in 0.000045 sec, getting values](zbx_log_fatal_info+0x435) [0x55e458953665] 4934:20190606:152111.877 16: /usr/local/sbin/zabbix_server: poller #8 [got 0 values in 0.000045 sec, getting values](+0xd2a86) [0x55e458953a86] 4934:20190606:152111.877 15: /lib/x86_64-linux-gnu/libpthread.so.0(+0x12890) [0x7fb17d37f890] 4934:20190606:152111.877 14: /usr/lib/oracle/10.2.0.5/client/lib/libsqora.so(bcoSQLSetStmtOption+0x13) [0x7fb1098016b3] 4934:20190606:152111.877 13: /usr/lib/oracle/10.2.0.5/client/lib/libsqora.so(bccSQLSetStmtOption+0xdf) [0x7fb10981f16d] 4934:20190606:152111.877 12: /usr/lib/oracle/10.2.0.5/client/lib/libsqora.so(SQLSetConnectAttrW+0x35d) [0x7fb10981cc05] 4934:20190606:152111.877 11: /usr/lib/x86_64-linux-gnu/libodbc.so.2(+0xd249) [0x7fb17c37f249] 4934:20190606:152111.877 10: /usr/lib/x86_64-linux-gnu/libodbc.so.2(SQLConnect+0x20a) [0x7fb17c38168a] 4934:20190606:152111.877 9: /usr/local/sbin/zabbix_server: poller #8 [got 0 values in 0.000045 sec, getting values](zbx_odbc_connect+0x201) [0x55e4588ffb81] 4934:20190606:152111.877 8: /usr/local/sbin/zabbix_server: poller #8 [got 0 values in 0.000045 sec, getting values](get_value_db+0xed) [0x55e4588dbead] 4934:20190606:152111.878 7: /usr/local/sbin/zabbix_server: poller #8 [got 0 values in 0.000045 sec, getting values](+0x50939) [0x55e4588d1939] 4934:20190606:152111.878 6: /usr/local/sbin/zabbix_server: poller #8 [got 0 values in 0.000045 sec, getting values](poller_thread+0xfb) [0x55e4588d1e4b] 4934:20190606:152111.878 5: /usr/local/sbin/zabbix_server: poller #8 [got 0 values in 0.000045 sec, getting values](zbx_thread_start+0x27) [0x55e45895cce7] 4934:20190606:152111.878 4: /usr/local/sbin/zabbix_server: poller #8 [got 0 values in 0.000045 sec, getting values](MAIN_ZABBIX_ENTRY+0x8f2) [0x55e4588c2f52] 4934:20190606:152111.878 3: /usr/local/sbin/zabbix_server: poller #8 [got 0 values in 0.000045 sec, getting values](daemon_start+0x16d) [0x55e458952a5d] 4934:20190606:152111.878 2: /usr/local/sbin/zabbix_server: poller #8 [got 0 values in 0.000045 sec, getting values](main+0x418) [0x55e4588c1e48] 4934:20190606:152111.878 1: /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xe7) [0x7fb17a0f4b97] 4934:20190606:152111.878 0: /usr/local/sbin/zabbix_server: poller #8 [got 0 values in 0.000045 sec, getting values](_start+0x2a) [0x55e4588c218a] |
Comment by Andris Mednis [ 2019 Jun 06 ] |
Could be a duplicate of ZBX-12159. Take a look at ZBX-12159 - there is a workaround: compile proxy without OpenLDAP and curl, then do unixODBC checks from this proxy. |
Comment by Roger [ 2019 Jun 06 ] |
Must it be a proxy? I have compiled server without ldap and curl to test but it keeps crashing with the same backtrace: ./configure --enable-java --enable-server --enable-agent --with-mysql --enable-ipv6 --with-net-snmp --with-openipmi --with-unixodbc --with-libxml2 --with-ssh2 --with-jabber --sysconfdir=/etc/zabbix/ --datadir=/usr/local/share/ --prefix=/usr/local/ --with-openssl
23164:20190606:165957.897 === Backtrace: === 23164:20190606:165957.898 18: /usr/local/sbin/zabbix_server: poller #1 [got 0 values in 0.000037 sec, getting values](zbx_backtrace+0x4d) [0x557f4547816d] 23164:20190606:165957.898 17: /usr/local/sbin/zabbix_server: poller #1 [got 0 values in 0.000037 sec, getting values](zbx_log_fatal_info+0x435) [0x557f454786c5] 23164:20190606:165957.898 16: /usr/local/sbin/zabbix_server: poller #1 [got 0 values in 0.000037 sec, getting values](+0xbaae6) [0x557f45478ae6] 23164:20190606:165957.898 15: /lib/x86_64-linux-gnu/libpthread.so.0(+0x12890) [0x7f5c7fe23890] 23164:20190606:165957.899 14: /usr/lib/oracle/10.2.0.5/client/lib/libsqora.so(bcoSQLSetStmtOption+0x13) [0x7f5c0f8dc6b3] 23164:20190606:165957.899 13: /usr/lib/oracle/10.2.0.5/client/lib/libsqora.so(bccSQLSetStmtOption+0xdf) [0x7f5c0f8fa16d] 23164:20190606:165957.899 12: /usr/lib/oracle/10.2.0.5/client/lib/libsqora.so(SQLSetConnectAttrW+0x35d) [0x7f5c0f8f7c05] 23164:20190606:165957.899 11: /usr/lib/x86_64-linux-gnu/libodbc.so.2(+0xd249) [0x7f5c7ee23249] 23164:20190606:165957.899 10: /usr/lib/x86_64-linux-gnu/libodbc.so.2(SQLConnect+0x20a) [0x7f5c7ee2568a] 23164:20190606:165957.899 9: /usr/local/sbin/zabbix_server: poller #1 [got 0 values in 0.000037 sec, getting values](zbx_odbc_connect+0x201) [0x557f45426d91] 23164:20190606:165957.899 8: /usr/local/sbin/zabbix_server: poller #1 [got 0 values in 0.000037 sec, getting values](get_value_db+0xed) [0x557f4540eb1d] 23164:20190606:165957.899 7: /usr/local/sbin/zabbix_server: poller #1 [got 0 values in 0.000037 sec, getting values](+0x4d857) [0x557f4540b857] 23164:20190606:165957.899 6: /usr/local/sbin/zabbix_server: poller #1 [got 0 values in 0.000037 sec, getting values](poller_thread+0xfb) [0x557f4540bd5b] 23164:20190606:165957.900 5: /usr/local/sbin/zabbix_server: poller #1 [got 0 values in 0.000037 sec, getting values](zbx_thread_start+0x27) [0x557f45481d47] 23164:20190606:165957.900 4: /usr/local/sbin/zabbix_server: poller #1 [got 0 values in 0.000037 sec, getting values](MAIN_ZABBIX_ENTRY+0x8f2) [0x557f453fce52] 23164:20190606:165957.900 3: /usr/local/sbin/zabbix_server: poller #1 [got 0 values in 0.000037 sec, getting values](daemon_start+0x16d) [0x557f45477abd] 23164:20190606:165957.900 2: /usr/local/sbin/zabbix_server: poller #1 [got 0 values in 0.000037 sec, getting values](main+0x4f3) [0x557f453fbdb3] 23164:20190606:165957.900 1: /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xe7) [0x7f5c7d277b97] 23164:20190606:165957.900 0: /usr/local/sbin/zabbix_server: poller #1 [got 0 values in 0.000037 sec, getting values](_start+0x2a) [0x557f453fc08a] |
Comment by Andris Mednis [ 2019 Jun 06 ] |
Compiling without LDAP and cURL is good, but let's check did you get free of libldap library: $ ldd zabbix_server | grep -i ldap |
Comment by Roger [ 2019 Jun 06 ] |
The result of this command is blank. |
Comment by Roger [ 2019 Jun 06 ] |
I have compiled and configured zabbix_proxy but it has the same error: ./configure --enable-proxy --with-unixodbc --with-mysql --sysconfdir=/etc/zabbix/ --datadir=/usr/local/share/ --prefix=/usr/local/
28763:20190606:183159.846 === Backtrace: === 28763:20190606:183159.846 18: /usr/local/sbin/zabbix_proxy: poller #2 [got 0 values in 0.000030 sec, getting values](zbx_backtrace+0x4d) [0x5565ac413b5d] 28763:20190606:183159.846 17: /usr/local/sbin/zabbix_proxy: poller #2 [got 0 values in 0.000030 sec, getting values](zbx_log_fatal_info+0x435) [0x5565ac4140b5] 28763:20190606:183159.846 16: /usr/local/sbin/zabbix_proxy: poller #2 [got 0 values in 0.000030 sec, getting values](+0x9a4e6) [0x5565ac4144e6] 28763:20190606:183159.846 15: /lib/x86_64-linux-gnu/libpthread.so.0(+0x12890) [0x7fb08da24890] 28763:20190606:183159.846 14: /usr/lib/oracle/10.2.0.5/client/lib/libsqora.so(bcoSQLSetStmtOption+0x13) [0x7fb08e2d46b3] 28763:20190606:183159.847 13: /usr/lib/oracle/10.2.0.5/client/lib/libsqora.so(bccSQLSetStmtOption+0xdf) [0x7fb08e2f216d] 28763:20190606:183159.847 12: /usr/lib/oracle/10.2.0.5/client/lib/libsqora.so(SQLSetConnectAttrW+0x35d) [0x7fb08e2efc05] 28763:20190606:183159.847 11: /usr/lib/x86_64-linux-gnu/libodbc.so.2(+0xd249) [0x7fb08cff3249] 28763:20190606:183159.847 10: /usr/lib/x86_64-linux-gnu/libodbc.so.2(SQLConnect+0x20a) [0x7fb08cff568a] 28763:20190606:183159.847 9: /usr/local/sbin/zabbix_proxy: poller #2 [got 0 values in 0.000030 sec, getting values](zbx_odbc_connect+0x201) [0x5565ac3c67c1] 28763:20190606:183159.847 8: /usr/local/sbin/zabbix_proxy: poller #2 [got 0 values in 0.000030 sec, getting values](get_value_db+0xed) [0x5565ac3bc6ed] 28763:20190606:183159.847 7: /usr/local/sbin/zabbix_proxy: poller #2 [got 0 values in 0.000030 sec, getting values](+0x3f448) [0x5565ac3b9448] 28763:20190606:183159.847 6: /usr/local/sbin/zabbix_proxy: poller #2 [got 0 values in 0.000030 sec, getting values](poller_thread+0xeb) [0x5565ac3b993b] 28763:20190606:183159.847 5: /usr/local/sbin/zabbix_proxy: poller #2 [got 0 values in 0.000030 sec, getting values](zbx_thread_start+0x27) [0x5565ac414cc7] 28763:20190606:183159.847 4: /usr/local/sbin/zabbix_proxy: poller #2 [got 0 values in 0.000030 sec, getting values](MAIN_ZABBIX_ENTRY+0x58d) [0x5565ac3b16dd] 28763:20190606:183159.847 3: /usr/local/sbin/zabbix_proxy: poller #2 [got 0 values in 0.000030 sec, getting values](daemon_start+0x16d) [0x5565ac4134ad] 28763:20190606:183159.847 2: /usr/local/sbin/zabbix_proxy: poller #2 [got 0 values in 0.000030 sec, getting values](main+0x62d) [0x5565ac3b07dd] 28763:20190606:183159.847 1: /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xe7) [0x7fb08c789b97] 28763:20190606:183159.847 0: /usr/local/sbin/zabbix_proxy: poller #2 [got 0 values in 0.000030 sec, getting values](_start+0x2a) [0x5565ac3b0d2a] |
Comment by Andris Mednis [ 2019 Jun 06 ] |
Can you upload files
as gzipped attachments ? |
Comment by Glebs Ivanovskis [ 2019 Jun 06 ] |
rblancop, make sure you have all Oracle environment variables set for Zabbix processes. From your zabbix-server.service: Environment=ORACLE_HOME=/usr/lib/oracle/10.2.0.5/client64/lib Shouldn't ORACLE_HOME end with "client64" (without "lib")? Also see |
Comment by Roger [ 2019 Jun 07 ] |
I have created an environment file to both services and tested with and without "lib" in ORACLE_HOME variable, but there is no difference, isql works with both and zabbix keeps crashing. root@mczabbix:~# sstrings -a /proc/12745/environ LANG=es_ES.UTF-8 PATH=/usr/lib/oracle/10.2.0.5/client64/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games INVOCATION_ID=46458fa908f64e819a16df8a8c59ed64 JOURNAL_STREAM=9:585708 LD_LIBRARY_PATH=/usr/lib/oracle/10.2.0.5/client64/lib ORACLE_HOME=/usr/lib/oracle/10.2.0.5/client64 I have attached all the compressed files. The files libodbc.so.2, libsqora.so and libpthread.so.0 are symlinks to libodbc.so.2.0.0, libsqora.so.10.1 and libpthread-2.27.so I will try to downgrade unixODBC to 2.2.14. |
Comment by Roger [ 2019 Jun 07 ] |
How can i compile zabbix_server with unixodbc in another folder? I installed unixODBC 2.2.14 in "/opt/unixodbc2.2.14" and tried to compile zabbix with "-with-unixodbc=/opt/unixodbc2.2.14/lib/libodbc.so" and "-with-unixodbc=/opt/unixodbc2.2.14/lib", but it shows this error at configuration: checking for odbc_config... /opt/unixodbc2.2.14/lib/libodbc.so checking for SQLAllocHandle in -lodbc... no configure: error: unixODBC library not found |
Comment by Glebs Ivanovskis [ 2019 Jun 07 ] |
I think you don't need to recompile, just set LD_LIBRARY_PATH=/opt/unixodbc2.2.14/lib for Zabbix server. |
Comment by Roger [ 2019 Jun 07 ] |
If i set it, zabbix and isql don't work: Cannot connect to ODBC DSN: [SQL_ERROR]:[01000][0][[unixODBC][Driver Manager]Can't open lib '/usr/lib/oracle/10.2.0.5/client/lib/libsqora.so' : file not found]
|
Comment by Glebs Ivanovskis [ 2019 Jun 07 ] |
LD_LIBRARY_PATH=/usr/lib/oracle/10.2.0.5/client64/lib:/opt/unixodbc2.2.14/lib |
Comment by Roger [ 2019 Jun 07 ] |
I tried it too, isql works but zabbix keeps crashing (it's like if it were using only the unixodbc 2.3.4). |
Comment by Glebs Ivanovskis [ 2019 Jun 07 ] |
Could you please run isql and Zabbix with LD_DEBUG=bindings (and probably also LD_DEBUG_OUTPUT=...)? |
Comment by Roger [ 2019 Jun 07 ] |
I configured both LD_DEBUG and LD_DEBUG_OUTPUT in environment and Zabbix proxy. Then connected as zabbix user and executed isql (it worked and connected to a database) (see LD_log_isql.tar.gz). Finally configured Zabbix proxy environment variables, started the service and forced the check of the database item (it's the only item enabled) (see LD_log_proxy.tar.gz). LD_LIBRARY_PATH is as you said: LD_LIBRARY_PATH=/usr/lib/oracle/10.2.0.5/client64/lib:/opt/unixodbc2.2.14/lib
|
Comment by Glebs Ivanovskis [ 2019 Jun 11 ] |
Proxy still seems to use libodbc.so from /usr/lib/x86_64-linux-gnu/ instead of /opt/unixodbc2.2.14/lib/ Have you tried experimenting with different Oracle driver versions? |
Comment by Roger [ 2019 Jun 12 ] |
I finally made it work. It seems that the problem is with unixodbc 2.3.4, donwgrading to 2.2.14 resolves the problem.
From a fresh installed Debian 9.9 first installed all the zabbix dependences, the Oracle 10 drivers and compiled and installed unixodbc2.2.14 with all default values: /usr/local/src/unixODBC-2.2.14/configure --enable-gui=no
Added this links: ln -s /usr/local/lib/libodbc.so.1.0.0 /usr/lib/x86_64-linux-gnu/libodbc.so.1
Compiled Zabbix server with all options (including ldap and libcurl) ./configure --enable-server --enable-agent --with-mysql --enable-ipv6 --with-net-snmp --with-libcurl --with-libxml2 --with-jabber --with-openipmi --with-unixodbc --with-ssh2 --with-openssl --with-ldap --sysconfdir=/etc/zabbix/ --datadir=/usr/local/share/ --prefix=/usr/local/
And finally set the environment variables at the Zabbix Server systemd service (EnvironmentFile=-/etc/systemd/system/zabbix.environment): PATH=/usr/lib/oracle/10.2.0.5/client64/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games
On another server, after following this steps, i had to add this links: ln -s /usr/local/lib/libodbcinst.so.1.0.0 /lib/x86_64-linux-gnu/libodbcinst.so.1 |
Comment by Roger [ 2019 Jun 12 ] |
I can't try with different Oracle driver. With this configuration, in Ubuntu should it work too. We can close the issue. Thank you! |
Comment by Glebs Ivanovskis [ 2019 Jun 12 ] |
I'm glad you've managed to find a solution to your problem! I've added a comment with a link in |
Comment by Vladislavs Boborikins (Inactive) [ 2019 Jun 12 ] |
The issue linked to |
Comment by Glebs Ivanovskis [ 2019 Jun 12 ] |
Thank you! |