[ZBX-9960] ODBC error: file not found Created: 2015 Oct 15  Updated: 2017 May 30  Resolved: 2015 Oct 19

Status: Closed
Project: ZABBIX BUGS AND ISSUES
Component/s: Server (S)
Affects Version/s: 2.4.6
Fix Version/s: None

Type: Incident report Priority: Major
Reporter: Bruno Galindro da Costa Assignee: Unassigned
Resolution: Duplicate Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

root@zabbix:~# dpkg -l |grep zabbix
ii percona-zabbix-templates 1.1.5-1 all Percona Monitoring Plugins for Zabbix
ii zabbix-agent 1:2.4.6-1+trusty amd64 network monitoring solution - agent
ii zabbix-frontend-php 1:2.4.6-1+trusty all network monitoring solution - PHP front-end
ii zabbix-get 1:2.4.6-1+trusty amd64 network monitoring solution - get
ii zabbix-java-gateway 1:2.4.6-1+trusty amd64 network monitoring solution - java-gateway
ii zabbix-release 2.4-1+trusty all Zabbix official repository configuration
ii zabbix-sender 1:2.4.6-1+trusty amd64 network monitoring solution - sender
ii zabbix-server-mysql 1:2.4.6-1+trusty amd64 network monitoring solution - server (using MySQL)

root@zabbix:~# dpkg -l |grep odbc
ii libodbc1:amd64 2.2.14p2-5ubuntu5 amd64 ODBC library for Unix
ii odbcinst 2.2.14p2-5ubuntu5 amd64 Helper program for accessing odbc ini files
ii odbcinst1debian2:amd64 2.2.14p2-5ubuntu5 amd64 Support library for accessing odbc ini files
ii unixodbc 2.2.14p2-5ubuntu5 amd64 Basic ODBC tools
ii unixodbc-dev 2.2.14p2-5ubuntu5 amd64 ODBC libraries for UNIX (development files)

root@zabbix:~# lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 14.04.3 LTS
Release: 14.04
Codename: trusty

          1. Zabbix version #####
            7942:20151015:105321.145 Starting Zabbix Server. Zabbix 2.4.6 (revision 54796).
            7942:20151015:105321.145 ****** Enabled features ******
            7942:20151015:105321.145 SNMP monitoring: YES
            7942:20151015:105321.146 IPMI monitoring: YES
            7942:20151015:105321.146 WEB monitoring: YES
            7942:20151015:105321.146 VMware monitoring: YES
            7942:20151015:105321.146 Jabber notifications: YES
            7942:20151015:105321.146 Ez Texting notifications: YES
            7942:20151015:105321.146 ODBC: YES
            7942:20151015:105321.146 SSH2 support: YES
            7942:20151015:105321.146 IPv6 support: YES
            7942:20151015:105321.146 ******************************
            7942:20151015:105321.146 using configuration file: /etc/zabbix/zabbix_server.conf
            7942:20151015:105321.163 current database version (mandatory/optional): 02040000/02040000
            7942:20151015:105321.163 required mandatory version: 02040000

Issue Links:
Duplicate
duplicates ZBX-7665 Zabbix 2.2.1 ODBC monitor crash - in ... Closed

 Description   

I've setup a database monitor item to make a simple query in a MySQL database through ODBC. This is my Item configuration. I'll show only the parameters that I have changed:

Name: Ticket count
Type: Database monitor
Key: db.odbc.select[otrs.count.ticket,mysql56_otrs]
SQL query: select count(id) as total from ticket
Type of information: Numeric
Data type Decimal
History Storage Period: 7
Applications: OTRS

The above item was created in a template called "Template OTRS Stats" and it was associated to the mysql host.

My Zabbix server is compiled with ODBC support as showed in "Environment" section. The latest mysql odbc driver (mysql-connector-odbc-5.2.7-linux-glibc2.5-x86-64bit) downloaded from https://dev.mysql.com/downloads/connector/odbc/5.1.html was installed under /etc/zabbix/ and the owner of the files is the zabbix user:

root@zabbix:~# ls -lh /etc/zabbix/libmyodbc5*
-rwxr-xr-x 1 zabbix zabbix 8,9M Out 15 12:45 /etc/zabbix/libmyodbc5a.so
-rwxr-xr-x 1 zabbix zabbix 8,9M Out 15 12:45 /etc/zabbix/libmyodbc5w.so

There's no immutable flag set on those libraries:

root@zabbix:~# lsattr /etc/zabbix/libmyodbc5*
-------------e-- /etc/zabbix/libmyodbc5a.so
-------------e-- /etc/zabbix/libmyodbc5w.so

The unixODBC package is installed as showed in "Environment" section too. Here are the config files:

root@zabbix:~# cat /etc/odbcinst.ini 
[mysql_unicode]
Driver=/etc/zabbix/libmyodbc5w.so

[mysql_ansi]
Driver=/etc/zabbix/libmyodbc5a.so

root@zabbix:~# cat /etc/odbc.ini 
[mysql56_otrs]
Driver=mysql_unicode
Server=mysqlserver.domain
Database=otrs
Port=3306
Option=3
User=*****
Password=******

I can connect to mysql server from zabbix server using isql:

root@zabbix:~# isql mysql56_otrs
+---------------------------------------+
| Connected!                            |
|                                       |
| sql-statement                         |
| help [tablename]                      |
| quit                                  |
|                                       |
+---------------------------------------+
SQL> select count(id) as total from ticket
+---------------------+
| total               |
+---------------------+
| 70465               |
+---------------------+
SQLRowCount returns 1
1 rows fetched
SQL> quit

The item was not in unsupported state. But no data is registered under last data. So, I've turned zabbix into debug mode (debuglevel=4) and finally I've see an error message:

 13849:20151015:122800.165 In substitute_key_macros() data:'db.odbc.select[otrs.count.ticket,mysql56_otrs]'
 13849:20151015:122800.167 End of substitute_key_macros():SUCCEED data:'db.odbc.select[otrs.count.ticket,mysql56_otrs]'
 13849:20151015:122800.173 In get_value() key:'db.odbc.select[otrs.count.ticket,mysql56_otrs]'
 13849:20151015:122800.175 In get_value_db() key_orig:'db.odbc.select[otrs.count.ticket,mysql56_otrs]'
 13849:20151015:122800.177 In odbc_DBconnect() db_dsn:'mysql56_otrs' user:''
 13849:20151015:122800.182 odbc_Diag(): rc_msg:'SQL_ERROR' rec_nr:1 sql_state:'01000' native_err_code:0 err_msg:'[unixODBC][Driver Manager]Can't open lib '/etc/zabbix/libmyodbc5w.so' : file not found'
 13849:20151015:122800.184 End of odbc_DBconnect():FAIL

Why zabbix can't open lib '/etc/zabbix/libmyodbc5w.so' ???
Why zabbix doesn't put item into Unsupported state?



 Comments   
Comment by Aleksandrs Saveljevs [ 2015 Oct 16 ]

We do not load /etc/zabbix/libmyodbc5w.so ourselves - instead, we rely on unixODBC library to do that. It might be better to discuss that on https://www.zabbix.org/wiki/Getting_help.

Regarding the item not being in an unsupported state, this might indicate a bug. Could you please show a longer part of the log?

Comment by Oleksii Zagorskyi [ 2015 Oct 16 ]

Test by "isql" command should be done under "zabbix" account only, to make sure it will work for zabbix server too.

We know issues where users performed incorrect installation of a driver and as result wrong permissions for driver or its dependent files (as for "zabbix" user) caused not working related odbc checks.

Comment by Bruno Galindro da Costa [ 2015 Oct 18 ]

Aleksandrs,

Thanks for return. I've openned this issue because all tests covered by official zabbix documentation are done fine. Why should I discuss this on that lists? Sorry, but I'm little convinced that this is realy a bug on Zabbix Server, not with unixODBC, just because isql test works fine for root and for zabbix user (as you can see bellow). Is there any other tests that I could do to prove my suspicion?

Regarding the log files, I'll attach them on this case.

Oleksiy,

I've tested with zabbix account and it works fine:

zabbix@zabbix:~$ isql mysql56_otrs
---------------------------------------
Connected!
 
sql-statement
help
[tablename]
quit
 
---------------------------------------
SQL> select count(id) as total from ticket
---------------------
total
---------------------
70631
---------------------
SQLRowCount returns 1
1 rows fetched
SQL> quit
Comment by Bruno Galindro da Costa [ 2015 Oct 18 ]

This is a 4 minute debug log from our zabbix server. It represents a lot of information, because we have a lot of monitored hosts. But fortunatelly it has an odbc item execution log:

4567:20151017:185856.954 In substitute_key_macros() data:'db.odbc.select[otrs.count.ticket,mysql56_otrs]'
4567:20151017:185856.956 End of substitute_key_macros():SUCCEED data:'db.odbc.select[otrs.count.ticket,mysql56_otrs]'
4567:20151017:185856.961 In get_value() key:'db.odbc.select[otrs.count.ticket,mysql56_otrs]'
4567:20151017:185856.962 In get_value_db() key_orig:'db.odbc.select[otrs.count.ticket,mysql56_otrs]'
4567:20151017:185856.963 In odbc_DBconnect() db_dsn:'mysql56_otrs' user:''
4567:20151017:185857.028 End of odbc_DBconnect():SUCCEED
4567:20151017:185857.030 In odbc_DBselect() query:'select count(id) as total from ticket'

Comment by Aleksandrs Saveljevs [ 2015 Oct 19 ]

Bruno, thank you for the log! The lines you quoted are the only relevant lines, so we shall remove the attachment so that it does not take space and so that any private information is not on the Internet.

It can be seen from the log that it connects successfully (so the error mentioned in the issue description is not there anymore). However, it can be seen that Zabbix just hangs in odbc_DBselect(). This is exactly the behavior described in one of my comments in ZBX-7665 (quoting here for convenience):

This crash in odbc_DBconnect() happens with libmyodbc5a.so, the ANSI version of MySQL driver. With the Unicode version of the library, libmyodbc5w.so, it connects successfully, but hangs in odbc_DBselect().

This seems to be a bug in MySQL ODBC that we have already reported. Please see ZBX-7665 for more information and possible workarounds.

Comment by Bruno Galindro da Costa [ 2015 Oct 19 ]

Oh sorry. I haven't noticed that the log shows a successfull connection. =)

After read ZBX-7665, I've tried these two worarrounds:

Brian Chand added a comment - 2014 Jun 09 21:4
Aleksandrs Saveljevs added a comment - 2014 Aug 22 11:13

But my Zabbix Server crashed.

10905:20151019:072430.740 Got signal [signal:11(SIGSEGV),reason:1,refaddr:0x7ffee136ef93]. Crashing ...
10905:20151019:072430.740 ====== Fatal information: ======
10905:20151019:072430.740 Program counter: 0x7f210f9551ad
10905:20151019:072430.741 === Registers: ===
10905:20151019:072430.741 r8 = c = 12 = 12
10905:20151019:072430.741 r9 = 7f210fdcea60 = 139779976784480 = 139779976784480
10905:20151019:072430.741 r10 = 0 = 0 = 0
10905:20151019:072430.741 r11 = 7ffedf9b13dc = 140732649903068 = 140732649903068
10905:20151019:072430.741 r12 = c = 12 = 12
10905:20151019:072430.741 r13 = 7f210fddda20 = 139779976845856 = 139779976845856
10905:20151019:072430.741 r14 = 7f210fddec20 = 139779976850464 = 139779976850464
10905:20151019:072430.741 r15 = 19bed6d = 26996077 = 26996077
10905:20151019:072430.741 rdi = 7f210fddda20 = 139779976845856 = 139779976845856
10905:20151019:072430.742 rsi = 7ffedf9b0e34 = 140732649901620 = 140732649901620
10905:20151019:072430.742 rbp = 7ffedf9b0e34 = 140732649901620 = 140732649901620
10905:20151019:072430.742 rbx = 7ffee136ef94 = 140732676894612 = 140732676894612
10905:20151019:072430.742 rdx = 19be160 = 26992992 = 26992992
10905:20151019:072430.742 rax = 7f210f955170 = 139779972092272 = 139779972092272
10905:20151019:072430.742 rcx = 19bed6d = 26996077 = 26996077
10905:20151019:072430.742 rsp = 7ffedf9b0d00 = 140732649901312 = 140732649901312
10905:20151019:072430.742 rip = 7f210f9551ad = 139779972092333 = 139779972092333
10905:20151019:072430.742 efl = 10287 = 66183 = 66183
10905:20151019:072430.742 csgsfs = 33 = 51 = 51
10905:20151019:072430.743 err = 4 = 4 = 4
10905:20151019:072430.743 trapno = e = 14 = 14
10905:20151019:072430.743 oldmask = 0 = 0 = 0
10905:20151019:072430.743 cr2 = 7ffee136ef93 = 140732676894611 = 140732676894611

The versions that I've used was:

5.1.13
5.2.7 ansi
5.2.7 unicode

All of those above versions aren't available in official Ubuntu 14.04 repositories. The version that is available in the official repo is 5.1.10-3. For Ubuntu 12.04, is 5.1.10-1.

I've tried to use those versions above because I've noticed another bug with mysql driver version 5.1.10-1 and mysql server 5.6 as reported here: https://mariadb.atlassian.net/browse/MDEV-6086. I could reproduce this bug from the host that my MySQL Server 5.6 is installed, wich is running in an Ubuntu 12.04.

Now, I've installed the 5.1.10-3 on my Zabbix Server to make a test. My Zabbix Server, as mentioned before, is running under an Ubuntu 14.04. And the result is: it works very well!!! No core dumps in isql (as noticed in ubuntu 12.04 with 5.1.10-1) and no crash in Zabbix Server (as noticed with 5.1.13 and 5.2.7 ansi / unicode). Now I've the Database monitor item working very well, collecting data, generating thrends, graphs, ...

I think this could be the best workarround to do for people that uses Ubuntu as Zabbix Server O.S.

Could you please link this ticket with ZBX-7665?

Thank you very much for patience and attention Aleksandrs.

Comment by Aleksandrs Saveljevs [ 2015 Oct 19 ]

This ticket is already linked to ZBX-7665 as a duplicate (see "Issue Links" towards the top). However, I have added a mention of your research as a separate comment in ZBX-7665. Thank you!

Comment by Bruno Galindro da Costa [ 2015 Oct 19 ]

=)

Generated at Thu Apr 25 06:35:44 EEST 2024 using Jira 9.12.4#9120004-sha1:625303b708afdb767e17cb2838290c41888e9ff0.