[ZBX-8629] ODBC Monitor always cause crash of zabbix_server (Zabbix 2.2.5) Created: 2014 Aug 19 Updated: 2017 May 30 Resolved: 2014 Aug 21 |
|
Status: | Closed |
Project: | ZABBIX BUGS AND ISSUES |
Component/s: | Proxy (P), Server (S) |
Affects Version/s: | 2.0.12, 2.2.5, 2.3.3 |
Fix Version/s: | 2.0.13rc1, 2.2.6rc1, 2.3.4 |
Type: | Incident report | Priority: | Critical |
Reporter: | Bean Young | Assignee: | Unassigned |
Resolution: | Fixed | Votes: | 0 |
Labels: | crash, odbc | ||
Remaining Estimate: | Not Specified | ||
Time Spent: | Not Specified | ||
Original Estimate: | Not Specified | ||
Environment: |
CentOS 6.4 x86_64 |
Attachments: | Install Zabbix 2.2.5 From Sources.html zabbix_server.log zabbix_server_dump.7z | ||||||||
Issue Links: |
|
Description |
I installed Zabbix 2.2.5 from source file of zabbix-2.2.5.tar.gz, my complile options are: After installment, I tried to create an "Database monitor" item on the monitored host, but when zabbix server checked this DB item, it crashed in 10 ~ 15 seconds. I've checked the log file of zabbix_server, and found that error occured when odbc_DBfetch() was executed, then a lot of error message were shown : The attachment files are described as following: |
Comments |
Comment by Bean Young [ 2014 Aug 19 ] |
I've tried pre-zabbix-2.2.6rc1-48164.tar.gz on the same environment just now, but this problem still exists, how can I solve this issue ?? |
Comment by Aleksandrs Saveljevs [ 2014 Aug 19 ] |
Backtrace for easier searching: 2705:20140819:140450.276 === Backtrace: === 2705:20140819:140450.287 15: /usr/local/sbin/zabbix_server: poller #3 [got 1 values in 0.000233 sec, getting values](print_fatal_info+0x287) [0x46cb87] 2705:20140819:140450.287 14: /usr/local/sbin/zabbix_server: poller #3 [got 1 values in 0.000233 sec, getting values]() [0x46ce06] 2705:20140819:140450.287 13: /lib64/libc.so.6() [0x35a2a329a0] 2705:20140819:140450.287 12: /usr/local/sbin/zabbix_server: poller #3 [got 1 values in 0.000233 sec, getting values](get_string+0x2f) [0x4764cf] 2705:20140819:140450.287 11: /usr/lib64/libmyodbc5.so(sql_get_data+0x328) [0x7f6bb21368e8] 2705:20140819:140450.287 10: /usr/lib64/libmyodbc5.so(+0x6138f) [0x7f6bb213738f] 2705:20140819:140450.287 9: /usr/lib64/libmyodbc5.so(my_SQLExtendedFetch+0x4e6) [0x7f6bb21383e6] 2705:20140819:140450.287 8: /usr/lib64/libodbc.so.2(SQLFetch+0x103) [0x7f6bbe5d8713] 2705:20140819:140450.287 7: /usr/local/sbin/zabbix_server: poller #3 [got 1 values in 0.000233 sec, getting values](odbc_DBfetch+0x44) [0x49fb44] 2705:20140819:140450.287 6: /usr/local/sbin/zabbix_server: poller #3 [got 1 values in 0.000233 sec, getting values](get_value_db+0xd8) [0x42ecc8] 2705:20140819:140450.287 5: /usr/local/sbin/zabbix_server: poller #3 [got 1 values in 0.000233 sec, getting values]() [0x42753c] 2705:20140819:140450.287 4: /usr/local/sbin/zabbix_server: poller #3 [got 1 values in 0.000233 sec, getting values](main_poller_loop+0x92) [0x427702] 2705:20140819:140450.287 3: /usr/local/sbin/zabbix_server: poller #3 [got 1 values in 0.000233 sec, getting values](MAIN_ZABBIX_ENTRY+0x7b7) [0x41a527] 2705:20140819:140450.287 2: /usr/local/sbin/zabbix_server: poller #3 [got 1 values in 0.000233 sec, getting values](daemon_start+0x1c4) [0x46c1a4] 2705:20140819:140450.287 1: /lib64/libc.so.6(__libc_start_main+0xfd) [0x35a2a1ed1d] 2705:20140819:140450.287 0: /usr/local/sbin/zabbix_server: poller #3 [got 1 values in 0.000233 sec, getting values]() [0x4150e5] |
Comment by Aleksandrs Saveljevs [ 2014 Aug 21 ] |
I could reproduce the issue on our Debian 6.0, which has unixODBC 2.2.14 and a manually downloaded MySQL ODBC connector 5.3.4 (from http://dev.mysql.com/downloads/connector/odbc/). So what we have is zabbix_server, which links to libodbc.so. That library loads libmyodbc5w.so dynamically. That MySQL driver library has global function get_string(), but the problem is that zabbix_server also has global function get_string(). So when the linker loads libmyodbc5w.so and tries to resolve "get_string" symbol, it resolves it to the function in zabbix_server (see http://stackoverflow.com/questions/6538501/linking-two-shared-libraries-with-some-of-the-same-symbols). When called, our get_string() crashes, as we can see from the backtrace. Note that get_string() is not used by zabbix_server. Rather, it is a function that is currently used by zabbix_sender only. Here is what dlopen() manual at http://man7.org/linux/man-pages/man3/dlsym.3.html has to say:
Currently, we compile zabbix_server with "-rdynamic", but if we recompile the server without that option, the problem does not manifest. Anyhow, we need "-rdynamic" to get crash backtraces. So the proposed solution is to make get_string() function static in zabbix_sender.c and try to prefix all global functions with "zbx_" in the future. |
Comment by Aleksandrs Saveljevs [ 2014 Aug 21 ] |
Note that it seems to work fine with libmyodbc 5.1.x, which is currently available in Debian packages. |
Comment by Aleksandrs Saveljevs [ 2014 Aug 21 ] |
Fixed in development branch svn://svn.zabbix.com/branches/dev/ZBX-8629 . |
Comment by Bean Young [ 2014 Aug 22 ] |
Thanks a lot for locating this problem, these information is very helpful to me ! According to your advice, I've modified source files of common.h (line 847), str.c (line 1878, 1895) and zabbix_sender.c (line 445, 465, 474, 486), changed get_string to zbx_get_string, then I compiled and installed Zabbix 2.2.5 with the same options in the same environment (CentOS 6.4 x86_64) as described before. Now, I can get monitoring data of MySQL by items of "Database monitor" type ! |
Comment by Aleksandrs Saveljevs [ 2014 Aug 22 ] |
Fixed in pre-2.0.13 r48311, pre-2.2.6 r48313, and pre-2.3.4 (trunk) r48314. |