[ZBX-10897] crash using unixODBC with an Oracle database Created: 2016 Jun 12  Updated: 2019 Nov 29  Resolved: 2018 Mar 06

Status: Closed
Project: ZABBIX BUGS AND ISSUES
Component/s: Server (S)
Affects Version/s: 3.0.1
Fix Version/s: None

Type: Documentation task Priority: Major
Reporter: Aitor Assignee: Unassigned
Resolution: Workaround proposed Votes: 0
Labels: crash, oracle, unixodbc
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

Linux estutahr 2.6.32-573.18.1.el6.x86_64 #1 SMP Wed Jan 6 11:20:49 EST 2016 x86_64 x86_64 x86_64 GNU/Linux
unixODBC 2.3.4
zabbix_server (Zabbix) 3.0.1
Test DB: Oracle 11.2.0.1.0
Attached files: odbcinst.ini and odbc.ini


Attachments: File dump.zabbix_server.gz     File log.zabbix_server.gz     File odbc.ini     File odbcinst.ini    
Issue Links:
Duplicate
is duplicated by ZBX-7665 Zabbix 2.2.1 ODBC monitor crash - in ... Closed
is duplicated by ZBX-9009 Oracle ODBC monitoring pollers crash Closed
is duplicated by ZBX-10928 Zabbix Proxy started crashing after a... Closed
is duplicated by ZBX-16223 Database monitoring with odbc and Ora... Closed
Sub-task
depends on ZBX-12159 Resolving TNS names via LDAP crash on... Confirmed

 Description   

Crash with UNIXODBC from zabbix server.
Query: select * from dual
Key: db.odbc.select[sessions_used2,test]
Type: Text
with isql from unixodbc works fine.

+---------------------------------------+
| Connected!                            |
|                                       |
| sql-statement                         |
| help [tablename]                      |
| quit                                  |
|                                       |
+---------------------------------------+
SQL> select * from dual
+------+
| DUMMY|
+------+
| X    |
+------+
SQLRowCount returns 4294967295
1 rows fetched
SQL> quit

Attached dump and server crash log



 Comments   
Comment by Aleksandrs Saveljevs [ 2016 Jun 13 ]

Backtrace for easier searching:

 19620:20160612:045543.162 === Backtrace: ===
 19620:20160612:045543.163 17: /home/swcomun/herrtux/aaresmi/mon-zabbix/zabbix_301/sbin/zabbix_server: poller #24 [got 43 values in 4.985385 sec, getting values](print_fatal_info+0xa6) [0x479446]
 19620:20160612:045543.163 16: /home/swcomun/herrtux/aaresmi/mon-zabbix/zabbix_301/sbin/zabbix_server: poller #24 [got 43 values in 4.985385 sec, getting values]() [0x479b66]
 19620:20160612:045543.163 15: /lib64/libc.so.6(+0x326a0) [0x7f10f1a046a0]
 19620:20160612:045543.163 14: /app/oracle/product/11.2.0/client_1/lib/libsqora.so.11.1(bcoSQLSetStmtOption+0x11) [0x7f0fa7e988a1]
 19620:20160612:045543.163 13: /app/oracle/product/11.2.0/client_1/lib/libsqora.so.11.1(bccSQLSetStmtOption+0xd7) [0x7f0fa7eb1b2b]
 19620:20160612:045543.163 12: /app/oracle/product/11.2.0/client_1/lib/libsqora.so.11.1(SQLSetConnectAttrW+0x334) [0x7f0fa7eafb80]
 19620:20160612:045543.163 11: /home/swcomun/herrtux/aaresmi/mon-zabbix/tools/unixODBC/lib/libodbc.so.2(+0xef41) [0x7f10f2d75f41]
 19620:20160612:045543.163 10: /home/swcomun/herrtux/aaresmi/mon-zabbix/tools/unixODBC/lib/libodbc.so.2(SQLConnect+0x1a3) [0x7f10f2d76c73]
 19620:20160612:045543.164 9: /home/swcomun/herrtux/aaresmi/mon-zabbix/zabbix_301/sbin/zabbix_server: poller #24 [got 43 values in 4.985385 sec, getting values](odbc_DBconnect+0x141) [0x4a21a1]
 19620:20160612:045543.164 8: /home/swcomun/herrtux/aaresmi/mon-zabbix/zabbix_301/sbin/zabbix_server: poller #24 [got 43 values in 4.985385 sec, getting values](get_value_db+0x116) [0x42b1a6]
 19620:20160612:045543.164 7: /home/swcomun/herrtux/aaresmi/mon-zabbix/zabbix_301/sbin/zabbix_server: poller #24 [got 43 values in 4.985385 sec, getting values]() [0x422a76]
 19620:20160612:045543.164 6: /home/swcomun/herrtux/aaresmi/mon-zabbix/zabbix_301/sbin/zabbix_server: poller #24 [got 43 values in 4.985385 sec, getting values](poller_thread+0xe6) [0x422c36]
 19620:20160612:045543.164 5: /home/swcomun/herrtux/aaresmi/mon-zabbix/zabbix_301/sbin/zabbix_server: poller #24 [got 43 values in 4.985385 sec, getting values](zbx_thread_start+0x62) [0x47a102]
 19620:20160612:045543.164 4: /home/swcomun/herrtux/aaresmi/mon-zabbix/zabbix_301/sbin/zabbix_server: poller #24 [got 43 values in 4.985385 sec, getting values](MAIN_ZABBIX_ENTRY+0x38b) [0x417eab]
 19620:20160612:045543.164 3: /home/swcomun/herrtux/aaresmi/mon-zabbix/zabbix_301/sbin/zabbix_server: poller #24 [got 43 values in 4.985385 sec, getting values](daemon_start+0x1ae) [0x4787ae]
 19620:20160612:045543.164 2: /home/swcomun/herrtux/aaresmi/mon-zabbix/zabbix_301/sbin/zabbix_server: poller #24 [got 43 values in 4.985385 sec, getting values](main+0x3df) [0x4187ff]
 19620:20160612:045543.164 1: /lib64/libc.so.6(__libc_start_main+0xfd) [0x7f10f19f0d5d]
 19620:20160612:045543.164 0: /home/swcomun/herrtux/aaresmi/mon-zabbix/zabbix_301/sbin/zabbix_server: poller #24 [got 43 values in 4.985385 sec, getting values]() [0x413059]
Comment by Aleksandrs Saveljevs [ 2016 Jun 13 ]

It seems that you also use Oracle as the Zabbix backend database. Does it crash on every ODBC check or from time to time? Did it ever work for you before with a previous version of Zabbix? What Oracle libraries do you have installed on the system?

Comment by Glebs Ivanovskis (Inactive) [ 2016 Jun 13 ]

Seems like unixODBC was linked with /home/swcomun/herrtux/aaresmi/mon-zabbix/tools/unixODBC/lib/libodbcinst.so.2.0.0 but calls functions from /app/oracle/product/11.2.0/client_1/lib/libsqora.so.11.1 and crashes because of incompatible binary interfaces.

Comment by Aitor [ 2016 Jun 13 ]

@Aleksandrs. This is the first time that I try ODBC check. And fails every time. I worked with 2.4 version of zabbix but i didn't use unixODBC. The Oracle libraries used are 11.2.0

@Glebs, I compiled unixODBC 2.3.4 from sources and the instaled client in my server is 11.2.0. Do you mean that unixODBC is not compatible with 11.2.0 Oracle libraries? I'm see that the libodbcinst.so.1 libraries at Oracle path has this ldd:
linux-vdso.so.1 => (0x00007fff84246000)
libdl.so.2 => /lib64/libdl.so.2 (0x00007f6339a03000)
libm.so.6 => /lib64/libm.so.6 (0x00007f633977e000)
libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f6339561000)
libnsl.so.1 => /lib64/libnsl.so.1 (0x00007f6339348000)
libclntsh.so.11.1 => /app/oracle/product/11.2.0/client_1/lib/libclntsh.so.11.1 (0x00007f6336d1c000)
libodbcinst.so.1 => not found
libc.so.6 => /lib64/libc.so.6 (0x00007f6336988000)
/lib64/ld-linux-x86-64.so.2 (0x00007f6339dd8000)
libnnz11.so => /app/oracle/product/11.2.0/client_1/lib/libnnz11.so (0x00007f63365bf000)
libaio.so.1 => /lib64/libaio.so.1 (0x00007f63363be000)
when unixODBC 2.3.4 generate the libodbcinst.so.2.0.0

Comment by Glebs Ivanovskis (Inactive) [ 2016 Jun 14 ]

I haven't investigated your issue properly, but from the first glance it seems like the root cause is that both Zabbix and ODBC connector library depend on Oracle DB client library. Since Zabbix and ODBC connector (probably as a dependency for unixODBC) are built separately, generally speaking, same symbols (i.e. function calls) may resolve (at linking time) to different potentially incompatible implementations. System loader will resolve these symbols (at runtime) using either of libraries used at linking time (and consider its work done). But these leaves half of function calls with potentially incompatible implementation which leads to crashes.

From the backtrace I see that ODBC check is being processed, but functions from libsqora.so are being called. Instead of calling functions from libsqora.so for direct Zabbix connections to Oracle DB and calling functions from libodbcinst.so when connection is via unixODBC layer. Unfortunately, it is not a straightforward fix.

Comment by Glebs Ivanovskis (Inactive) [ 2016 Jul 01 ]

Dear aaresmi, there was a change of SQLLEN size from 4 to 8 bytes for 64 bit platforms between unixODBC versions 2.2.14 and 2.3.x, hence different major versions of the library (libodbc.so.1 and libodbc.so.2). As they say, this can affect drivers compiled with old definition of SQLLEN type. And you can use slencheck utility to find out what size of SQLLEN your ODBC driver adopts. Can you try this?

Comment by kanibal [ 2016 Jul 23 ]

I had the same issue and as you pointed, as you suggest I remove installed unixODBC 2.3 of Base repo of CentOS 7 and manually install 2.2.14 from base repo of CentOS 6. Also I use the Oracle client 11.2.0.4 that is the recommended for unixODBC 2.2.14.
Additional, I added the enviroment variables to the systemd script of zabbix-server, because, systemd units don't take enviroments from the system. This was checking the EnviromentFile variable of the script

[root@zabbix ~]# cat /usr/lib/systemd/system/zabbix-server.service
[Unit]
Description=Zabbix Server
After=syslog.target
After=network.target

[Service]
Environment="CONFFILE=/etc/zabbix/zabbix_server.conf"
EnvironmentFile=-/etc/sysconfig/zabbix-server
Type=forking
Restart=on-failure
PIDFile=/run/zabbix/zabbix_server.pid
KillMode=control-group
ExecStart=/usr/sbin/zabbix_server -c $CONFFILE
ExecStop=/bin/kill -SIGTERM $MAINPID
RestartSec=10s

[Install]
WantedBy=multi-user.target

And later adding to the file that is called:

[root@zabbix ~]# cat /etc/sysconfig/zabbix-server
ORACLE_HOME=/usr/lib/oracle/11.2/client64
LD_LIBRARY_PATH=/usr/lib/oracle/11.2/client64/lib:/usr/lib64:/usr/lib:$LD_LIBRARY_PATH
TNS_ADMIN=/usr/lib/oracle/11.2/client64/network/admin
PATH=$PATH:$ORACLE_HOME/bin

You can check this by findind the main pid of zabbix-server and runnning:

[root@zabbix ~]# strings -a /proc/24415/environ 
LANG=en_US.UTF-8
PATH=$PATH:$ORACLE_HOME/bin
CONFFILE=/etc/zabbix/zabbix_server.conf
ORACLE_HOME=/usr/lib/oracle/11.2/client64
LD_LIBRARY_PATH=/usr/lib/oracle/11.2/client64/lib:/usr/lib64:/usr/lib:$LD_LIBRARY_PATH
TNS_ADMIN=/usr/lib/oracle/11.2/client64/network/admin
Setup

Using zabbix 3, CentOS 7, Postgres and Nginx.

Comment by Glebs Ivanovskis (Inactive) [ 2016 Aug 09 ]

Dear aaresmi, do you have any updates?

Comment by Aitor [ 2017 Jul 16 ]

Sorry for the long delay. With the instructions added by kanibal (thx a lot) all worked with zabbix 3.2.5 and Oracle client 11.2.0.4.
I think that is needed to add to the documentation that the correct installation versions of the UnixOBDC and Oracle client are 2.2.14 and 11.2.0.4 respectively.
thx

Comment by Glebs Ivanovskis [ 2019 Jun 12 ]

A similar issue was reported in ZBX-16223 and a solution was eventually found.

Generated at Tue Apr 23 22:11:38 EEST 2024 using Jira 9.12.4#9120004-sha1:625303b708afdb767e17cb2838290c41888e9ff0.