[ZBX-10654] Zabbix Server 3.0.1 crash shortly after starting Created: 2016 Apr 12  Updated: 2018 Apr 11  Resolved: 2016 Apr 12

Status: Closed
Project: ZABBIX BUGS AND ISSUES
Component/s: Server (S)
Affects Version/s: 3.0.1
Fix Version/s: None

Type: Incident report Priority: Critical
Reporter: Michael Mol Assignee: Unassigned
Resolution: Duplicate Votes: 0
Labels: crash
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

CentOS 7 x86_64

Attachments: File mysql-community.repo     Text File yum.log     Text File zabbix_server.log     File zabbix_server.objdump.xz    
Issue Links:
Duplicate
duplicates ZBX-7665 Zabbix 2.2.1 ODBC monitor crash - in ... Closed
is duplicated by ZBX-13554 Zabbix server crash with new item db.... Closed
is duplicated by ZBX-11341 Zabbix crash with ODBC Closed
is duplicated by ZBX-12245 ot signal [signal:11(SIGSEGV),reason:... Closed

 Description   

Zabbix server crashes a short time after starting.

This server has been in prod for a while. This is a new issue.

When the issue began, I was experimenting with the API, trying to pull recent history items. The MySQL server began consuming a ton of CPU, which didn't cease after I terminated my API calls or restarted the zabbix server, so I restarted the MySQL server.

I also had run a yum update earlier, which appears to have resulted in an update to mysql-connector-odbc. (Which may be of particular interest, as a mysql odbc poller thread appears to be in the backtrace, if I read the output correctly.)

When the zabbix server crashes, systemd restarts it. It then crashes again after a little while.

 Comments   
Comment by Michael Mol [ 2016 Apr 12 ]

There doesn't appear to be a way to disable ODBC pollers, or I'd try that.

Comment by Glebs Ivanovskis (Inactive) [ 2016 Apr 12 ]

Are different versions of mysql-connector-odbc compatible with the same version of unixODBC?

You are right, one of the pollers that attempts to poll ODBC item is involved. But from backtrace seems like the problem is between unixODBC library and mysql ODBC driver.

Can you provide log with DebugLevel=4 setting?

Please attach gzipped result of

objdump -DSswx zabbix_server
Comment by Michael Mol [ 2016 Apr 12 ]

Looks like it. Though it does turn out my odbc.ini had an incorrect entry I hadn't cleaned up; a database had moved from one host to another, and the ODBC items that depended on it hadn't been fixed. Have that dealt with; I can use isql on the zabbix server's host to connect to all of the odbc entries. Zabbix server still crashes.

Since I don't have any working ODBC-based things on this Zabbix server anyway, I'm taking out all the ODBC checks, hoping it stops prodding the ODBC poller.

Comment by Michael Mol [ 2016 Apr 12 ]

objdump -DSswx `which zabbix_server`

(Use "xz -d zabbix_server.objdump.xz" to decompress)

Comment by Michael Mol [ 2016 Apr 12 ]

Also, for reference, the source for the version of the mysql connector I use. Currently have MySQL 5.6 enabled.

Comment by Aleksandrs Saveljevs [ 2016 Apr 12 ]

Backtrace for easier searching:

 11984:20160412:085813.056 === Backtrace: ===
 11984:20160412:085813.058 18: /usr/sbin/zabbix_server: poller #3 [got 2 values in 0.230231 sec, getting values](print_fatal_info+0x287) [0x487d77]
 11984:20160412:085813.059 17: /usr/sbin/zabbix_server: poller #3 [got 2 values in 0.230231 sec, getting values]() [0x488076]
 11984:20160412:085813.060 16: /lib64/libc.so.6(+0x35670) [0x7f9081189670]
 11984:20160412:085813.062 15: /lib64/libc.so.6(+0x86341) [0x7f90811da341]
 11984:20160412:085813.064 14: /usr/lib64/mysql/libmysqlclient.so.18(my_strdup+0x18) [0x7f9083be3b18]
 11984:20160412:085813.066 13: /usr/lib64/libmyodbc5w.so(myodbc_init+0xaa) [0x7f9066656efa]
 11984:20160412:085813.067 12: /usr/lib64/libmyodbc5w.so(my_SQLAllocEnv+0x24) [0x7f906665ab34]
 11984:20160412:085813.069 11: /lib64/libodbc.so.2(+0xd8dc) [0x7f90833a08dc]
 11984:20160412:085813.071 10: /lib64/libodbc.so.2(SQLConnect+0x1a7) [0x7f90833a2af7]
 11984:20160412:085813.073 9: /usr/sbin/zabbix_server: poller #3 [got 2 values in 0.230231 sec, getting values](odbc_DBconnect+0x12a) [0x4bb8aa]
 11984:20160412:085813.074 8: /usr/sbin/zabbix_server: poller #3 [got 2 values in 0.230231 sec, getting values](get_value_db+0x165) [0x434a65]
 11984:20160412:085813.075 7: /usr/sbin/zabbix_server: poller #3 [got 2 values in 0.230231 sec, getting values]() [0x42baa3]
 11984:20160412:085813.076 6: /usr/sbin/zabbix_server: poller #3 [got 2 values in 0.230231 sec, getting values](poller_thread+0x116) [0x42bc76]
 11984:20160412:085813.079 5: /usr/sbin/zabbix_server: poller #3 [got 2 values in 0.230231 sec, getting values](zbx_thread_start+0x57) [0x488b27]
 11984:20160412:085813.080 4: /usr/sbin/zabbix_server: poller #3 [got 2 values in 0.230231 sec, getting values](MAIN_ZABBIX_ENTRY+0x41d) [0x41da8d]
 11984:20160412:085813.082 3: /usr/sbin/zabbix_server: poller #3 [got 2 values in 0.230231 sec, getting values](daemon_start+0x1da) [0x4872da]
 11984:20160412:085813.083 2: /usr/sbin/zabbix_server: poller #3 [got 2 values in 0.230231 sec, getting values](main+0x39f) [0x41812f]
 11984:20160412:085813.084 1: /lib64/libc.so.6(__libc_start_main+0xf5) [0x7f9081175b15]
 11984:20160412:085813.086 0: /usr/sbin/zabbix_server: poller #3 [got 2 values in 0.230231 sec, getting values]() [0x4183e5]
Comment by Glebs Ivanovskis (Inactive) [ 2016 Apr 12 ]

Thank you for provided information! Seems like it is an old and known problem. It is also documented here

Closing as duplicate of ZBX-7665.

Comment by Aleksandrs Saveljevs [ 2016 Apr 12 ]

Michael, you might wish to remove the debug log if it contains sensitive information.

Comment by Michael Mol [ 2016 Apr 12 ]

Similar issue, but certainly not the same; it was working fine until the latest update to the mysql ODBC connector, which means Oracle had their bug fixed, and then it came back.

Comment by Glebs Ivanovskis (Inactive) [ 2016 Apr 13 ]

An observation... MySQL ODBC Connector 5.3.6 change history says

Attempt to build Connector/ODBC from source on a Linux platform against the client library and header files shipped with MySQL server 5.7.6 or newer failed. It was due to the changes made to the client library and the header files, to which Connector/ODBC has now been adapted. (Bug #20685833)


And my_strdup() (where Zabbix crashes) declaration has changed somewhere between 5.6 

char *my_strdup(const char *from, myf my_flags);

to 5.7 

char *my_strdup(PSI_memory_key key, const char *from, myf my_flags);

Old-style function is now known as 

char *my_once_strdup(const char *src,myf myflags);

Maybe two variants of my_strdup() are confused in runtime?

Generated at Fri Apr 26 13:17:43 EEST 2024 using Jira 9.12.4#9120004-sha1:625303b708afdb767e17cb2838290c41888e9ff0.