[ZBX-13239] ODBC crashes zabbix_server Created: 2017 Dec 21  Updated: 2018 Jan 15  Resolved: 2018 Jan 15

Status: Closed
Project: ZABBIX BUGS AND ISSUES
Component/s: Server (S)
Affects Version/s: 3.4.4
Fix Version/s: None

Type: Incident report Priority: Blocker
Reporter: Alain Ganuchaud Assignee: Unassigned
Resolution: Cannot Reproduce Votes: 0
Labels: crash, odbc, server
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

Debian 9
Mariadb 10.1.26
zabbix_server 3.4.4

libcrypto.so.1.0.0
libssl.so.1.0.0
libodbcinst.so.2.0.0

Attachments: Text File env_details.txt     File isql.DEBUG     Text File zabbix_server.log     Zip Archive zabbix_server_objdump.zip    
Issue Links:
Sub-task
depends on ZBXNEXT-3388 Load libraries in runtime with RTLD_L... Open

 Description   

No way to start zabbix_server with odbc items.
Isql is working fine.

I attach details from server log, libraries & objects dump.
Thanks for help, I have a look around, didn't find something similar ...



 Comments   
Comment by Glebs Ivanovskis (Inactive) [ 2017 Dec 21 ]

Please provide log file.

Comment by Alain Ganuchaud [ 2017 Dec 21 ]

zabbix_server.log

Comment by Glebs Ivanovskis (Inactive) [ 2017 Dec 21 ]

Thank you!

So I suppose you are doing Database monitor check using MariaDB ODBC driver and the backend for Zabbix Server is MariaDB too.

It is strange that libmariadbclient.so does not show up in ldd /usr/lib/x86_64-linux-gnu/odbc/libmaodbc.so...

Does isql work with this driver? (You need to try using same user as Zabbix.)

Could you launch isql and Zabbix server with environment variable LD_DEBUG=bindings and attach the output?

Comment by Alain Ganuchaud [ 2017 Dec 21 ]

Thanks,

yes same mariadb version for zabbix Mariadb backend & odbc.
isql is running well with exactly same selects & same zabbix user.

isql DEBUG produces errors, please find attached file although it works.
Do you think this is the root cause ? In this case, I guess I have to report this problem to Mariadb/Debian support.

Comment by Glebs Ivanovskis (Inactive) [ 2017 Dec 21 ]

Thank you! I don't think these binding errors are a serious thing, most likely UnixODBC can use several functions provided by ODBC drivers interchangeably.

However, I'm almost sure that Zabbix isn't guilty in this crash, crash is happening inside MariaDB ODBC connector. To gather more evidence, could you try launching Zabbix with LD_DEBUG=bindings as well? Also please tell me exact version of MariaDB ODBC driver installed on your system. I will have a look at the code.

Comment by Alain Ganuchaud [ 2017 Dec 21 ]

Thanks Glebs,

I replaced maria odbc driver by Mysql driver, same problem. 

ii  mysql-client                         5.7.20-1debian9                amd64        MySQL Client meta package depending on latest version
ii  mysql-community-client               5.7.20-1debian9                amd64        MySQL Client
ii  mysql-community-server               5.7.20-1debian9                amd64        MySQL Server
ii  mysql-server                         5.7.20-1debian9                amd64        MySQL Server meta package depending on latest version

ldd /usr/lib/x86_64-linux-gnu/odbc/libmyodbc.so
        linux-vdso.so.1 (0x00007ffcfad7e000)
        libodbcinst.so.2 => /usr/lib/x86_64-linux-gnu/libodbcinst.so.2 (0x00007efffe75a000)
        libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007efffe53d000)
        librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007efffe335000)
        libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007efffe131000)
        libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007efffdf17000)
        libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007efffdc13000)
        libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007efffd891000)
        libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007efffd67a000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007efffd2db000)
        libltdl.so.7 => /usr/lib/x86_64-linux-gnu/libltdl.so.7 (0x00007efffd0d1000)
        /lib64/ld-linux-x86-64.so.2 (0x00007effff014000)

So if zabbix is not guilty, I guess a system library is. I'll do further investigations in the next days.

Comment by Glebs Ivanovskis (Inactive) [ 2017 Dec 21 ]

For the record, here is the backtrace from the log file (for easier searching):

 13545:20171221:004431.576 === Backtrace: ===
 13545:20171221:004431.576 19: /usr/sbin/zabbix_server: poller #1 [got 0 values in 0.000007 sec, getting values](zbx_log_fatal_info+0x13c) [0x559c982d0a84]
 13545:20171221:004431.576 18: /usr/sbin/zabbix_server: poller #1 [got 0 values in 0.000007 sec, getting values](+0xe9ea4) [0x559c982d0ea4]
 13545:20171221:004431.576 17: /lib/x86_64-linux-gnu/libc.so.6(+0x33030) [0x7f9c3447e030]
 13545:20171221:004431.576 16: /usr/lib/x86_64-linux-gnu/odbc/libmaodbc.so(mariadb_get_infov+0x390) [0x7f9c1dabbc70]
 13545:20171221:004431.576 15: /usr/lib/x86_64-linux-gnu/odbc/libmaodbc.so(MADB_SetCapabilities+0x42) [0x7f9c1dab6092]
 13545:20171221:004431.577 14: /usr/lib/x86_64-linux-gnu/odbc/libmaodbc.so(MADB_DbcConnectDB+0x503) [0x7f9c1daa2b53]
 13545:20171221:004431.577 13: /usr/lib/x86_64-linux-gnu/odbc/libmaodbc.so(SQLConnectCommon+0x1c6) [0x7f9c1da99d26]
 13545:20171221:004431.577 12: /usr/lib/x86_64-linux-gnu/libodbc.so.2(SQLConnect+0x262) [0x7f9c36dd7e12]
 13545:20171221:004431.577 11: /usr/sbin/zabbix_server: poller #1 [got 0 values in 0.000007 sec, getting values](odbc_DBconnect+0x249) [0x559c98328370]
 13545:20171221:004431.577 10: /usr/sbin/zabbix_server: poller #1 [got 0 values in 0.000007 sec, getting values](+0x5efd4) [0x559c98245fd4]
 13545:20171221:004431.577 9: /usr/sbin/zabbix_server: poller #1 [got 0 values in 0.000007 sec, getting values](get_value_db+0xa9) [0x559c98246274]
 13545:20171221:004431.577 8: /usr/sbin/zabbix_server: poller #1 [got 0 values in 0.000007 sec, getting values](+0x50c93) [0x559c98237c93]
 13545:20171221:004431.577 7: /usr/sbin/zabbix_server: poller #1 [got 0 values in 0.000007 sec, getting values](+0x51f84) [0x559c98238f84]
 13545:20171221:004431.577 6: /usr/sbin/zabbix_server: poller #1 [got 0 values in 0.000007 sec, getting values](poller_thread+0x185) [0x559c98239be7]
 13545:20171221:004431.577 5: /usr/sbin/zabbix_server: poller #1 [got 0 values in 0.000007 sec, getting values](zbx_thread_start+0x37) [0x559c982ddf83]
 13545:20171221:004431.577 4: /usr/sbin/zabbix_server: poller #1 [got 0 values in 0.000007 sec, getting values](MAIN_ZABBIX_ENTRY+0x829) [0x559c98226530]
 13545:20171221:004431.577 3: /usr/sbin/zabbix_server: poller #1 [got 0 values in 0.000007 sec, getting values](daemon_start+0x31f) [0x559c982d019a]
 13545:20171221:004431.577 2: /usr/sbin/zabbix_server: poller #1 [got 0 values in 0.000007 sec, getting values](main+0x2fb) [0x559c98225d05]
 13545:20171221:004431.577 1: /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf1) [0x7f9c3446b2b1]
 13545:20171221:004431.577 0: /usr/sbin/zabbix_server: poller #1 [got 0 values in 0.000007 sec, getting values](_start+0x2a) [0x559c98219fea]

You know, this seems to be a bug in MariaDB client library. The following piece of code looks suspicious:

  case MARIADB_CONNECTION_EXTENDED_SERVER_CAPABILITIES:
    if (mysql)
      *((unsigned long *)arg)= mysql->extension->mariadb_server_capabilities;
    else
      goto error;
    break;

mysql is checked for not being NULL, but mysql->extension is dereferenced without a check. If mysql->extension is NULL and the offset of mariadb_server_capabilities field happens to be 0x70 bytes, then such line:

 13545:20171221:004431.576 Got signal [signal:11(SIGSEGV),reason:1,refaddr:0x70]. Crashing ...

...is exactly what you expect to see.

And yes, MariaDB ODBC driver calls mariadb_get_infov() from MADB_SetCapabilities() with MARIADB_CONNECTION_EXTENDED_SERVER_CAPABILITIES option.

Picture looks clear.

Comment by Glebs Ivanovskis (Inactive) [ 2017 Dec 22 ]

mysql->extension is defined as struct st_mariadb_extension * and here is the definition of struct st_mariadb_extension:

struct st_mariadb_extension {
  MA_CONNECTION_HANDLER *conn_hdlr;
  struct st_mariadb_session_state session_state[SESSION_TRACK_TYPES];
  unsigned long mariadb_client_flag; /* MariaDB specific client flags */
  unsigned long mariadb_server_capabilities; /* MariaDB specific server capabilities */
};

Assuming 64 bit architecture:

  1. conn_hdlr is a pointer, so 8 bytes;
  2. session_state is 6 by 16 byte array;
    • from here it is clear that SESSION_TRACK_TYPES is 6: 
      enum enum_session_state_type
{
  SESSION_TRACK_SYSTEM_VARIABLES= 0,
  SESSION_TRACK_SCHEMA,
  SESSION_TRACK_STATE_CHANGE,
  /* currently not supported by MariaDB Server */
  SESSION_TRACK_GTIDS,
  SESSION_TRACK_TRANSACTION_CHARACTERISTICS,
  SESSION_TRACK_TRANSACTION_TYPE /* make sure that SESSION_TRACK_END always points
                                    to last element of enum !! */
};

#define SESSION_TRACK_BEGIN 0
#define SESSION_TRACK_END SESSION_TRACK_TRANSACTION_TYPE
#define SESSION_TRACK_TYPES SESSION_TRACK_END + 1
    • and each struct st_mariadb_session_state is just two pointers, 16 bytes total
  3. unsigned long mariadb_client_flag is 8 bytes;
  4. mariadb_server_capabilities starts at 8 + 6*16 + 8 = 112‍10 = 70‍16

Everything aligns nicely

Comment by Glebs Ivanovskis (Inactive) [ 2017 Dec 24 ]

I tried to crash MariaDB client library with this short test program, but it works fine...

#include "mysql.h"
#include <stdio.h>

int	main(void)
{
	MYSQL		*mysql;
	int		ret;
	unsigned long	arg;

	if (NULL == (mysql = mysql_init(NULL)))
	{
		printf("Cannot initialize.\n");
		return -1;
	}

	printf("mysql->extension:%p\n", mysql->extension);

	ret = mariadb_get_infov(mysql, MARIADB_CONNECTION_EXTENDED_SERVER_CAPABILITIES, &arg);

	printf("ret:%d\n", ret);
	printf("arg:%lu\n", arg);

	mysql_close(mysql);

	return 0;
}

My current guess is that when ordinary client libraries and ODBC drivers are mixed in the same application, wrong mysql_init() may be called and MariaDB ODBC driver may get an incorrectly initialized MYSQL * handle.

I would still want to see the result of Zabbix server launch with LD_DEBUG=bindings.

Comment by Glebs Ivanovskis (Inactive) [ 2018 Jan 15 ]

No activity for more than two weeks after information was requested. Closing as Cannot Reproduce. Please take no offense and feel free to reopen and provide new information.

Generated at Fri Apr 26 23:22:14 EEST 2024 using Jira 9.12.4#9120004-sha1:625303b708afdb767e17cb2838290c41888e9ff0.