[ZBX-10897] crash using unixODBC with an Oracle database Created: 2016 Jun 12 Updated: 2019 Nov 29 Resolved: 2018 Mar 06 |
|
Status: | Closed |
Project: | ZABBIX BUGS AND ISSUES |
Component/s: | Server (S) |
Affects Version/s: | 3.0.1 |
Fix Version/s: | None |
Type: | Documentation task | Priority: | Major |
Reporter: | Aitor | Assignee: | Unassigned |
Resolution: | Workaround proposed | Votes: | 0 |
Labels: | crash, oracle, unixodbc | ||
Remaining Estimate: | Not Specified | ||
Time Spent: | Not Specified | ||
Original Estimate: | Not Specified | ||
Environment: |
Linux estutahr 2.6.32-573.18.1.el6.x86_64 #1 SMP Wed Jan 6 11:20:49 EST 2016 x86_64 x86_64 x86_64 GNU/Linux |
Attachments: |
![]() ![]() ![]() ![]() |
||||||||||||||||||||||||||||
Issue Links: |
|
Description |
Crash with UNIXODBC from zabbix server. +---------------------------------------+ | Connected! | | | | sql-statement | | help [tablename] | | quit | | | +---------------------------------------+ SQL> select * from dual +------+ | DUMMY| +------+ | X | +------+ SQLRowCount returns 4294967295 1 rows fetched SQL> quit Attached dump and server crash log |
Comments |
Comment by Aleksandrs Saveljevs [ 2016 Jun 13 ] |
Backtrace for easier searching: 19620:20160612:045543.162 === Backtrace: === 19620:20160612:045543.163 17: /home/swcomun/herrtux/aaresmi/mon-zabbix/zabbix_301/sbin/zabbix_server: poller #24 [got 43 values in 4.985385 sec, getting values](print_fatal_info+0xa6) [0x479446] 19620:20160612:045543.163 16: /home/swcomun/herrtux/aaresmi/mon-zabbix/zabbix_301/sbin/zabbix_server: poller #24 [got 43 values in 4.985385 sec, getting values]() [0x479b66] 19620:20160612:045543.163 15: /lib64/libc.so.6(+0x326a0) [0x7f10f1a046a0] 19620:20160612:045543.163 14: /app/oracle/product/11.2.0/client_1/lib/libsqora.so.11.1(bcoSQLSetStmtOption+0x11) [0x7f0fa7e988a1] 19620:20160612:045543.163 13: /app/oracle/product/11.2.0/client_1/lib/libsqora.so.11.1(bccSQLSetStmtOption+0xd7) [0x7f0fa7eb1b2b] 19620:20160612:045543.163 12: /app/oracle/product/11.2.0/client_1/lib/libsqora.so.11.1(SQLSetConnectAttrW+0x334) [0x7f0fa7eafb80] 19620:20160612:045543.163 11: /home/swcomun/herrtux/aaresmi/mon-zabbix/tools/unixODBC/lib/libodbc.so.2(+0xef41) [0x7f10f2d75f41] 19620:20160612:045543.163 10: /home/swcomun/herrtux/aaresmi/mon-zabbix/tools/unixODBC/lib/libodbc.so.2(SQLConnect+0x1a3) [0x7f10f2d76c73] 19620:20160612:045543.164 9: /home/swcomun/herrtux/aaresmi/mon-zabbix/zabbix_301/sbin/zabbix_server: poller #24 [got 43 values in 4.985385 sec, getting values](odbc_DBconnect+0x141) [0x4a21a1] 19620:20160612:045543.164 8: /home/swcomun/herrtux/aaresmi/mon-zabbix/zabbix_301/sbin/zabbix_server: poller #24 [got 43 values in 4.985385 sec, getting values](get_value_db+0x116) [0x42b1a6] 19620:20160612:045543.164 7: /home/swcomun/herrtux/aaresmi/mon-zabbix/zabbix_301/sbin/zabbix_server: poller #24 [got 43 values in 4.985385 sec, getting values]() [0x422a76] 19620:20160612:045543.164 6: /home/swcomun/herrtux/aaresmi/mon-zabbix/zabbix_301/sbin/zabbix_server: poller #24 [got 43 values in 4.985385 sec, getting values](poller_thread+0xe6) [0x422c36] 19620:20160612:045543.164 5: /home/swcomun/herrtux/aaresmi/mon-zabbix/zabbix_301/sbin/zabbix_server: poller #24 [got 43 values in 4.985385 sec, getting values](zbx_thread_start+0x62) [0x47a102] 19620:20160612:045543.164 4: /home/swcomun/herrtux/aaresmi/mon-zabbix/zabbix_301/sbin/zabbix_server: poller #24 [got 43 values in 4.985385 sec, getting values](MAIN_ZABBIX_ENTRY+0x38b) [0x417eab] 19620:20160612:045543.164 3: /home/swcomun/herrtux/aaresmi/mon-zabbix/zabbix_301/sbin/zabbix_server: poller #24 [got 43 values in 4.985385 sec, getting values](daemon_start+0x1ae) [0x4787ae] 19620:20160612:045543.164 2: /home/swcomun/herrtux/aaresmi/mon-zabbix/zabbix_301/sbin/zabbix_server: poller #24 [got 43 values in 4.985385 sec, getting values](main+0x3df) [0x4187ff] 19620:20160612:045543.164 1: /lib64/libc.so.6(__libc_start_main+0xfd) [0x7f10f19f0d5d] 19620:20160612:045543.164 0: /home/swcomun/herrtux/aaresmi/mon-zabbix/zabbix_301/sbin/zabbix_server: poller #24 [got 43 values in 4.985385 sec, getting values]() [0x413059] |
Comment by Aleksandrs Saveljevs [ 2016 Jun 13 ] |
It seems that you also use Oracle as the Zabbix backend database. Does it crash on every ODBC check or from time to time? Did it ever work for you before with a previous version of Zabbix? What Oracle libraries do you have installed on the system? |
Comment by Glebs Ivanovskis (Inactive) [ 2016 Jun 13 ] |
Seems like unixODBC was linked with /home/swcomun/herrtux/aaresmi/mon-zabbix/tools/unixODBC/lib/libodbcinst.so.2.0.0 but calls functions from /app/oracle/product/11.2.0/client_1/lib/libsqora.so.11.1 and crashes because of incompatible binary interfaces. |
Comment by Aitor [ 2016 Jun 13 ] |
@Aleksandrs. This is the first time that I try ODBC check. And fails every time. I worked with 2.4 version of zabbix but i didn't use unixODBC. The Oracle libraries used are 11.2.0 @Glebs, I compiled unixODBC 2.3.4 from sources and the instaled client in my server is 11.2.0. Do you mean that unixODBC is not compatible with 11.2.0 Oracle libraries? I'm see that the libodbcinst.so.1 libraries at Oracle path has this ldd: |
Comment by Glebs Ivanovskis (Inactive) [ 2016 Jun 14 ] |
I haven't investigated your issue properly, but from the first glance it seems like the root cause is that both Zabbix and ODBC connector library depend on Oracle DB client library. Since Zabbix and ODBC connector (probably as a dependency for unixODBC) are built separately, generally speaking, same symbols (i.e. function calls) may resolve (at linking time) to different potentially incompatible implementations. System loader will resolve these symbols (at runtime) using either of libraries used at linking time (and consider its work done). But these leaves half of function calls with potentially incompatible implementation which leads to crashes. From the backtrace I see that ODBC check is being processed, but functions from libsqora.so are being called. Instead of calling functions from libsqora.so for direct Zabbix connections to Oracle DB and calling functions from libodbcinst.so when connection is via unixODBC layer. Unfortunately, it is not a straightforward fix. |
Comment by Glebs Ivanovskis (Inactive) [ 2016 Jul 01 ] |
Dear aaresmi, there was a change of SQLLEN size from 4 to 8 bytes for 64 bit platforms between unixODBC versions 2.2.14 and 2.3.x, hence different major versions of the library (libodbc.so.1 and libodbc.so.2). As they say, this can affect drivers compiled with old definition of SQLLEN type. And you can use slencheck utility to find out what size of SQLLEN your ODBC driver adopts. Can you try this? |
Comment by kanibal [ 2016 Jul 23 ] |
I had the same issue and as you pointed, as you suggest I remove installed unixODBC 2.3 of Base repo of CentOS 7 and manually install 2.2.14 from base repo of CentOS 6. Also I use the Oracle client 11.2.0.4 that is the recommended for unixODBC 2.2.14.
[root@zabbix ~]# cat /usr/lib/systemd/system/zabbix-server.service
[Unit]
Description=Zabbix Server
After=syslog.target
After=network.target
[Service]
Environment="CONFFILE=/etc/zabbix/zabbix_server.conf"
EnvironmentFile=-/etc/sysconfig/zabbix-server
Type=forking
Restart=on-failure
PIDFile=/run/zabbix/zabbix_server.pid
KillMode=control-group
ExecStart=/usr/sbin/zabbix_server -c $CONFFILE
ExecStop=/bin/kill -SIGTERM $MAINPID
RestartSec=10s
[Install]
WantedBy=multi-user.target
And later adding to the file that is called: [root@zabbix ~]# cat /etc/sysconfig/zabbix-server ORACLE_HOME=/usr/lib/oracle/11.2/client64 LD_LIBRARY_PATH=/usr/lib/oracle/11.2/client64/lib:/usr/lib64:/usr/lib:$LD_LIBRARY_PATH TNS_ADMIN=/usr/lib/oracle/11.2/client64/network/admin PATH=$PATH:$ORACLE_HOME/bin You can check this by findind the main pid of zabbix-server and runnning: [root@zabbix ~]# strings -a /proc/24415/environ LANG=en_US.UTF-8 PATH=$PATH:$ORACLE_HOME/bin CONFFILE=/etc/zabbix/zabbix_server.conf ORACLE_HOME=/usr/lib/oracle/11.2/client64 LD_LIBRARY_PATH=/usr/lib/oracle/11.2/client64/lib:/usr/lib64:/usr/lib:$LD_LIBRARY_PATH TNS_ADMIN=/usr/lib/oracle/11.2/client64/network/admin Setup Using zabbix 3, CentOS 7, Postgres and Nginx. |
Comment by Glebs Ivanovskis (Inactive) [ 2016 Aug 09 ] |
Dear aaresmi, do you have any updates? |
Comment by Aitor [ 2017 Jul 16 ] |
Sorry for the long delay. With the instructions added by kanibal (thx a lot) all worked with zabbix 3.2.5 and Oracle client 11.2.0.4. |
Comment by Glebs Ivanovskis [ 2019 Jun 12 ] |
A similar issue was reported in |