[ZBX-8629] ODBC Monitor always cause crash of zabbix_server (Zabbix 2.2.5) Created: 2014 Aug 19  Updated: 2017 May 30  Resolved: 2014 Aug 21

Status: Closed
Project: ZABBIX BUGS AND ISSUES
Component/s: Proxy (P), Server (S)
Affects Version/s: 2.0.12, 2.2.5, 2.3.3
Fix Version/s: 2.0.13rc1, 2.2.6rc1, 2.3.4

Type: Incident report Priority: Critical
Reporter: Bean Young Assignee: Unassigned
Resolution: Fixed Votes: 0
Labels: crash, odbc
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

CentOS 6.4 x86_64
MySQL 5.6.20
mysql-connector-odbc-5.3.4-1.el6.x86_64
unixODBC 2.2.14
gcc 4.8.2
g++ 4.8.2
GNU make 3.8.1


Attachments: HTML File Install Zabbix 2.2.5 From Sources.html     Text File zabbix_server.log     File zabbix_server_dump.7z    
Issue Links:
Duplicate
is duplicated by ZBX-8322 zabbix 2.2.0 proxy crash on ODBC chec... Closed

 Description   

I installed Zabbix 2.2.5 from source file of zabbix-2.2.5.tar.gz, my complile options are:
./configure --enable-server --enable-agent --enable-java --enable-ipv6 --with-mysql --with-jabber --with-libxml2 --with-unixodbc --with-net-snmp --with-ssh2 --with-openipmi --with-ldap --with-libcurl --with-iconv --with-iconv-include --with-iconv-lib

After installment, I tried to create an "Database monitor" item on the monitored host, but when zabbix server checked this DB item, it crashed in 10 ~ 15 seconds. I've checked the log file of zabbix_server, and found that error occured when odbc_DBfetch() was executed, then a lot of error message were shown :
Got signal [signal:11(SIGSEGV),reason:1,refaddr:(nil)]. Crashing ...

The attachment files are described as following:
Install Zabbix 2.2.5 From Sources.html: a step by step installation procedure used by myself
zabbix_server.log: an error log of zabbix_server
zabbix_server_dump.7z: the objdump file of zabbix_server



 Comments   
Comment by Bean Young [ 2014 Aug 19 ]

I've tried pre-zabbix-2.2.6rc1-48164.tar.gz on the same environment just now, but this problem still exists, how can I solve this issue ??

Comment by Aleksandrs Saveljevs [ 2014 Aug 19 ]

Backtrace for easier searching:

  2705:20140819:140450.276 === Backtrace: ===
  2705:20140819:140450.287 15: /usr/local/sbin/zabbix_server: poller #3 [got 1 values in 0.000233 sec, getting values](print_fatal_info+0x287) [0x46cb87]
  2705:20140819:140450.287 14: /usr/local/sbin/zabbix_server: poller #3 [got 1 values in 0.000233 sec, getting values]() [0x46ce06]
  2705:20140819:140450.287 13: /lib64/libc.so.6() [0x35a2a329a0]
  2705:20140819:140450.287 12: /usr/local/sbin/zabbix_server: poller #3 [got 1 values in 0.000233 sec, getting values](get_string+0x2f) [0x4764cf]
  2705:20140819:140450.287 11: /usr/lib64/libmyodbc5.so(sql_get_data+0x328) [0x7f6bb21368e8]
  2705:20140819:140450.287 10: /usr/lib64/libmyodbc5.so(+0x6138f) [0x7f6bb213738f]
  2705:20140819:140450.287 9: /usr/lib64/libmyodbc5.so(my_SQLExtendedFetch+0x4e6) [0x7f6bb21383e6]
  2705:20140819:140450.287 8: /usr/lib64/libodbc.so.2(SQLFetch+0x103) [0x7f6bbe5d8713]
  2705:20140819:140450.287 7: /usr/local/sbin/zabbix_server: poller #3 [got 1 values in 0.000233 sec, getting values](odbc_DBfetch+0x44) [0x49fb44]
  2705:20140819:140450.287 6: /usr/local/sbin/zabbix_server: poller #3 [got 1 values in 0.000233 sec, getting values](get_value_db+0xd8) [0x42ecc8]
  2705:20140819:140450.287 5: /usr/local/sbin/zabbix_server: poller #3 [got 1 values in 0.000233 sec, getting values]() [0x42753c]
  2705:20140819:140450.287 4: /usr/local/sbin/zabbix_server: poller #3 [got 1 values in 0.000233 sec, getting values](main_poller_loop+0x92) [0x427702]
  2705:20140819:140450.287 3: /usr/local/sbin/zabbix_server: poller #3 [got 1 values in 0.000233 sec, getting values](MAIN_ZABBIX_ENTRY+0x7b7) [0x41a527]
  2705:20140819:140450.287 2: /usr/local/sbin/zabbix_server: poller #3 [got 1 values in 0.000233 sec, getting values](daemon_start+0x1c4) [0x46c1a4]
  2705:20140819:140450.287 1: /lib64/libc.so.6(__libc_start_main+0xfd) [0x35a2a1ed1d]
  2705:20140819:140450.287 0: /usr/local/sbin/zabbix_server: poller #3 [got 1 values in 0.000233 sec, getting values]() [0x4150e5]
Comment by Aleksandrs Saveljevs [ 2014 Aug 21 ]

I could reproduce the issue on our Debian 6.0, which has unixODBC 2.2.14 and a manually downloaded MySQL ODBC connector 5.3.4 (from http://dev.mysql.com/downloads/connector/odbc/).

So what we have is zabbix_server, which links to libodbc.so. That library loads libmyodbc5w.so dynamically. That MySQL driver library has global function get_string(), but the problem is that zabbix_server also has global function get_string(). So when the linker loads libmyodbc5w.so and tries to resolve "get_string" symbol, it resolves it to the function in zabbix_server (see http://stackoverflow.com/questions/6538501/linking-two-shared-libraries-with-some-of-the-same-symbols). When called, our get_string() crashes, as we can see from the backtrace.

Note that get_string() is not used by zabbix_server. Rather, it is a function that is currently used by zabbix_sender only. Here is what dlopen() manual at http://man7.org/linux/man-pages/man3/dlsym.3.html has to say:

External references in the library are resolved using the libraries in that library's dependency list and any other libraries previously opened with the RTLD_GLOBAL flag. If the executable was linked with the flag "-rdynamic" (or, synonymously, "--export-dynamic"), then the global symbols in the executable will also be used to resolve references in a dynamically loaded library.

Currently, we compile zabbix_server with "-rdynamic", but if we recompile the server without that option, the problem does not manifest.

Anyhow, we need "-rdynamic" to get crash backtraces. So the proposed solution is to make get_string() function static in zabbix_sender.c and try to prefix all global functions with "zbx_" in the future.

Comment by Aleksandrs Saveljevs [ 2014 Aug 21 ]

Note that it seems to work fine with libmyodbc 5.1.x, which is currently available in Debian packages.

Comment by Aleksandrs Saveljevs [ 2014 Aug 21 ]

Fixed in development branch svn://svn.zabbix.com/branches/dev/ZBX-8629 .

Comment by Bean Young [ 2014 Aug 22 ]

Thanks a lot for locating this problem, these information is very helpful to me !

According to your advice, I've modified source files of common.h (line 847), str.c (line 1878, 1895) and zabbix_sender.c (line 445, 465, 474, 486), changed get_string to zbx_get_string, then I compiled and installed Zabbix 2.2.5 with the same options in the same environment (CentOS 6.4 x86_64) as described before.

Now, I can get monitoring data of MySQL by items of "Database monitor" type !

Comment by Aleksandrs Saveljevs [ 2014 Aug 22 ]

Fixed in pre-2.0.13 r48311, pre-2.2.6 r48313, and pre-2.3.4 (trunk) r48314.

Generated at Thu Apr 18 23:06:50 EEST 2024 using Jira 9.12.4#9120004-sha1:625303b708afdb767e17cb2838290c41888e9ff0.