|
list of package installed on FreeBSD server (where zabbix server run)
|
|
Another crash...
in the moment of crashing the remote mysql server (where zabbix database reside) has been rebooted, but may be a coincidence.
|
|
another one crash:
|
|
I was be able to reproduce crash:
HOST A : 192.168.77.222 server with installed and running zabbix24-server-2.4.5
HOST B : 192.168.77.111 server with installed and running mysql server (with zabbix database)
… start zabbix server on HOST (A)
.. everything is ok…
.. manual stopping mysql server on remote host (B) where zabbix database is stored [[systemctl stop mysql.service]]
..after one minute….manual start mysql service on remote host (B) where zabbix database is stored
(systemctl start mysql.service)
..after a while log of zabbix server (on host A) is populate with :
..and ten minutes later without other logs zabbix server (on host A) crash:
|
|
One hour later after restart of mysql (see attachment) zabbix server crash (see log with debug level=4)
|
|
This looks like a duplicate of ZBX-7765, but this task has a nice description and a way to reproduce the problem.
|
|
I've been trying to reproduce it without any luck. My setup was two virtual machines running 10.1-RELEASE FreeBSD 10.1-RELEASE #0 r274401. I tried running zabbix server from the default package and also one compiled from sources. It had default configuration monitoring one host. On the other system I was running mysql server from the default mysql56-server package.
So the setup was as close as I could get to the described one, but no stopping/starting mysql server caused any zabbix server crashes.
What I noticed you had mysql56-client-5.6.24 package listed, while on my installations I had mysql56-client-5.6.24_1. Could not find anything about fixed crashes in changelog - so I doubt it was the reason. Still maybe you could upgrade your mysql package and try to reproduce the crash again?
|
|
Reproduced.
[root@k.......]# date
Tue Jun 30 15:34:27 UTC 2015
[root@k.......]]# systemctl stop mysql.service
[root@k.......]]# date
Tue Jun 30 15:37:25 UTC 2015
[root@.......]# systemctl start mysql.service
root@zabbix# cat /var/log/zabbix_server.log | grep stopped
59902:20150630:153935.199 Zabbix Server stopped. Zabbix 2.4.5 (revision 53282).
|
|
I have upgraded mysql56-client-5.6.24_1 of zabbix server via source ports
but crash persist:
stopped mysqlserver (on Oracle Linux ) for 3 min
then zabbix (on freebsd 10.1) crashed few minutes later (view log with debug level=4)
|
|
Thanks, but still can't reproduce it. Nothing really suspicious in DB connection code.
Would it be possible to build Zabbix server from sources with the attached patch (freebsd_backtrace_x64.diff) and reproduce the crash? It should give proper stack trace. (Note that the patch will work only on 64 bit freebsd systems).
|
|
..the problem persist after each mysql down
Have you installed mysql-server on "Oracle Linux" distributiuon as my enviroment ?
I tried your patch but I am unable to complie:
|
|
The compilation error you see is not because of the patch (it is not related to cURL functionality is any way). Was the ./configure step successful? What was ./configure's output?
|
|
Sorry for late responde ... I was on holiday
OK I have done configure without curl:
/usr/local/etc/rc.d/zabbix_server stop
./configure --enable-server --with-mysql
then
make install
then
/usr/local/etc/rc.d/zabbix_server start
Starting zabbix_server.
After that to run zabbix server I have to copy my conf in another path:
cp /usr/local/etc/zabbix24/zabbix_server.conf /usr/local/etc/zabbix_server.conf
/usr/local/etc/rc.d/zabbix_server start
Do you need debug level 3 or 4 ?
|
Sorry for late responde ... I was on holiday
No problem, so was I 
4 would be better (though 3 should be enough if the crash originates from the place I thought).
|
|
So,
installing from source (with your patch) with this configure:
./configure --enable-server --enable-agent --with-mysql
I wasn't able to reproduce the problem.
But I need ODBC, SSH and other..
so I reinstalled from ports with this config:
CURL=on: Support for web monitoring
FPING=on: Build/install fping for ping checks
IPMI=off: Support for IPMI checks
IPV6=off: IPv6 protocol support
JABBER=off: Support for Jabber media type
JAVAGW=off: Support for Java gateway
LDAP=off: Support for LDAP server checks
LIBXML2=on: Support for libxml2 (required by monitoring VMware)
NMAP=off: Build/install nmap for o/s detection
SSH=on: Support for SSH-based checks
====> Options available for the single DB: you have to select exactly one of them
MYSQL=on: MySQL database support
PGSQL=off: PostgreSQL database support
SQLITE=off: SQLite database support
ORACLE=off: Oracle database support
====> Support for database checks via ODBC: you have to select exactly one of them
IODBC=off: ODBC backend via iODBC
UNIXODBC=on: ODBC backend via unixODBC
and I reproduced the problem simply turning off mysqld service and leave it down.
Could you suggest me the way to installing zabbix server from source with
CURL,FPING,LIBXML2,MYSQL,UNIXODBC
|
|
You should try configure with the following parameters:
./configure --enable-server --enable-agent --with-mysql --with-libcurl --with-libxml2 --with-unixodbc --with-ssh2
There are no specific fping build options in Zabbix, so I'm not sure what FPING=on means in ports, probably access rights confguration.
|
|
error:
checking for SSH2 support... yes
checking for gawk... (cached) nawk
checking for curl-config... /usr/local/bin/curl-config
checking for the version of libcurl... 7.43.0
checking for libcurl >= version 7.13.1... yes
checking for main in -lcurl... yes
checking whether libcurl is usable... yes
checking for curl_free... yes
checking for curl_easy_escape... yes
checking for ICONV support... configure: error: Unable to use iconv (libiconv check failed)
|
|
Strange, 10.1 should have native iconv support. Could you why libiconv check failed by running the following in zabbix build directory:
grep -A20 'checking for ICONV' config.log
|
|
/home/bid12/zabbix-2.4.5 # grep -A20 'checking for ICONV' config.log
|
|
Then you can try specifying the iconv installation directory:
If you want to specify iconv installation directories:
--with-iconv=[DIR] use iconv from given base install directory (DIR),
default is to search through a number of common
places for the iconv files.
--with-iconv-include=[DIR]
use iconv include headers from given path.
--with-iconv-lib=[DIR] use iconv libraries from given path.
You should be able to find the installation directories with pkg info -l libiconv-1.14_8 (or just grep filesystem for libiconv.so/iconv.h).
|
|
Always fail:
or
or
|
|
Oh, try then
LDFLAGS=-liconv ./configure --enable-server --enable-agent --with-mysql --with-libxml2 --with-unixodbc --with-ssh2
|
|
Ok
rm config.log
BUT AFTEER THIS:
I found error in confi log file:
Should I run 'make install' anyway?
|
|
Maybe it was the old config.log file? The final configure output looks ok and it would not finish if there were any errors. Also -liconv has been added to the server and agent libraries. Try making it.
|
|
ok I do make install,
but it was a new config.log (see the first line of my previous post: rm config.log.
So, it wasn't old config.log file.
make install
stoopped mysql on another server
150727 14:27:02 mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ended
... no crashing ....
|
|
restarted mysql 15:16:03
zabbix server stopped 15:28:00
see attach.
|
|
Thanks!
As I suspeceted it crashed in mysql_real_connect(), which almost certanly means that mysql_init() returned NULL. From MYSQL documentation it can return NULL only if there was insufficient memory to allocate a new object, which sounds unlikely.
Would you be up to trying few more patches while I'm trying to poke around this issue? I have few ideas which might or might not work, but nothing certain.
|
|
OK,
send me your patches
|
|
Thanks!
I attached 3 small patches:
- mysql_init_check.diff - adds a check if mysql_init() really returned NULL. In this case server would log a critical error and exit.
- mysql_library_init.diff - adds explicit mysql library initialization in main process. The mysql_library_init() is automatically called from mysql_init(), but in this case some thread related data would be initialized only on the next mysql_init() call. Zabbix does not use threads, so in therory it should not make any difference. But best is to check to be certain.
- mysql_static_conn.diff - instead of alocation connection object for each mysql_init() call it will define a static connection object and reuse it.
You can apply all 4 patches (the backtrace patch included) first to check if it fixes the crash. If everything is okay then you could remove either mysql_library_init.diff or mysql_static_conn.diff to find the fix.
The patches should be applied in the following order:
- mysql_static_conn.diff
- mysql_init_check.diff
- mysql_library_init.diff
- mysql_static_conn.diff
|
|
Crash again...
I have applied all patches in the following order:
patch < mysql_static_conn.diff
patch < mysql_static_conn.diff mysql_init_check.diff
patch < mysql_static_conn.diffmysql_library_init.diff
but, when I shutdown mysql on another server:
2015-07-29 07:38:17 25153 [Note] /usr/sbin/mysqld: Shutdown complete
150729 07:38:17 mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ended
..20 sec after, zabbix server died:
13668:20150729:073836.205 [Z3001] connection to database 'zabbix' failed: [2003] Can't connect to MySQL server on '192.168.77.111' (61)
13668:20150729:073836.205 database is down: reconnecting in 10 seconds
13651:20150729:073836.394 [Z3001] connection to database 'zabbix' failed: [2003] Can't connect to MySQL server on '192.168.77.111' (61)
13651:20150729:073836.394 database is down: reconnecting in 10 seconds
13665:20150729:073836.394 [Z3001] connection to database 'zabbix' failed: [2003] Can't connect to MySQL server on '192.168.77.111' (61)
13665:20150729:073836.394 database is down: reconnecting in 10 seconds
13663:20150729:073838.078 [Z3001] connection to database 'zabbix' failed: [2003] Can't connect to MySQL server on '192.168.77.111' (61)
13663:20150729:073838.078 database is down: reconnecting in 10 seconds
13658:20150729:073839.195 [Z3001] connection to database 'zabbix' failed: [2003] Can't connect to MySQL server on '192.168.77.111' (61)
13658:20150729:073839.195 database is down: reconnecting in 10 seconds
13655:20150729:073839.787 [Z3005] query failed: [2013] Lost connection to MySQL server during query [select hostid,key_,state,evaltype,formula,error,lifetime from items where itemid=23919]
13655:20150729:073839.787 cannot initialize MYSQL connection object
13650:20150729:073839.788 One child process died (PID:13655,exitcode/signal:1). Exiting ...
13650:20150729:073841.821 [Z3001] connection to database 'zabbix' failed: [2003] Can't connect to MySQL server on '192.168.77.111' (61)
13650:20150729:073841.821 Cannot connect to the database. Exiting...
|
|
See attachment"patch_command.txt" for Command History used for patching
|
|
again 15minutes after mysql down with debug level=4
2015-07-29 08:13:11 12724 [Note] /usr/sbin/mysqld: Shutdown complete
See attachment "zabbix_server_crash2.log "
|
|
Yes, Thanks
It's clear where the crash happens, but why mysql_init() might return NULL - currently have no idea. Will keep digging around.
|
|
At the end there is nothing much we can do if mysql connection initialization fails with what is supposed to be 'out of memory' error. It would be better to gracefully terminate the process, but the end result would be the same - the rest of processes would exit too.
If the default configuration is working fine it must be something with the shared libraries. You could try enabling zabbix features one by one to find out which library causes zabbix to crash. Maybe updating/rebuilding it would fix things.
|
|
Added check for mysql_init() returning NULL in development branch svn://svn.zabbix.com/branches/dev/ZBX-9655
|
|
Please review r55391
wiper reviewed, thanks
|
|
Released in:
- pre-2.2.11rc1 r55405
- pre-2.4.7rc1 r55406
- pre-3.0.0alpha2 r55407
|
|
I moved mysql server on the same server where zabbix server run. NO problem for many days.
But this morning, 58 min after adding a zabbix agent on the same server (so in one server there is mysql, zabbix server and zabbix agent) zabbix server suddenly crash.
Mysql is always up and running without reboot.
zabbix agent log (on the same server)
|
|
Mysql status
|
|
An observation. This:
looks like processing of LLD rule id lld_process_discovery_rule() and this:
looks like a query from db_host_update_availability() - the only two occasions when poller accesses our DB directly. Also, I see that unixODBC is involved (monitoring MySQL databases?) making it very similar to ZBX-9082.
Wild guess... Is it possible that during our ODBC check cycle (connect - select - fetch - close connection) under certain circumstances ODBC driver requests a too deep clean from DB client library and it cleans Zabbix DB handle too without poller noticing it?
|
|
So... When we perform ODBC check it ends with a cleanup. In odbc_DBclose() we free environment handle with unixODBC function SQLFreeHandle(), it hands the work over to MySQL ODBC Connector's my_SQLFreeEnv() (mysql-connector-odbc-5.3.6-src/driver/handle.c):
SQLRETURN SQL_API my_SQLFreeEnv(SQLHENV henv)
{
ENV *env= (ENV *) henv;
myodbc_mutex_destroy(&env->lock);
#ifndef _UNIX_
GlobalUnlock(GlobalHandle((HGLOBAL) henv));
GlobalFree(GlobalHandle((HGLOBAL) henv));
#else
x_free(henv);
myodbc_end();
#endif
return(SQL_SUCCESS);
}
myodbc_end() (mysql-connector-odbc-5.3.6-src/driver/dll.c) calls my_end() (mysql-5.6/mysys/my_init.c) which does my_thread_end(); and my_thread_global_end(); amongst everything else.
When poller loses connection to backend DB and attempts to reconnect it calls mysql_init() and as we see in attached logs for some reason it returns NULL. MySQL documentation says this could only be due to insufficient memory, but (mysql-5.6/sql-common/client.c):
MYSQL * STDCALL
mysql_init(MYSQL *mysql)
{
if (mysql_server_init(0, NULL, NULL))
return 0;
...
}
mysql_server_init() may return non-zero value is several cases (mysql-5.6/libmysql/libmysql.c):
int STDCALL mysql_server_init(int argc __attribute__((unused)),
char **argv __attribute__((unused)),
char **groups __attribute__((unused)))
{
int result= 0;
if (!mysql_client_init)
{
...
if (my_init())
return 1;
...
if (mysql_client_plugin_init())
return 1;
...
}
else
result= (int)my_thread_init();
return result;
}
I will not comment on first two, but the last one seems to be what doctor ordered (mysql-5.6/mysys/my_thr_init.c):
my_bool my_thread_init(void)
{
struct st_my_thread_var *tmp;
my_bool error=0;
if (!my_thread_global_init_done)
return 1;
...
Follow up. wiper asked a legitimate question whether mysql_client_init flag is reset in my_end(). Seems like it is not. It is reset only in mysql_server_end() (mysql-5.6/libmysql/libmysql.c) and mysql_server_end() is not called from my_end().
Actually, I'm not that sure about what I just said after I found this funny thing (mysql-5.6/sql/client_settings.h):
#define mysql_server_init(a,b,c) mysql_client_plugin_init()
#define mysql_server_end() mysql_client_plugin_deinit()
This makes life more complicated
|
|
Managed to reproduce! Created a separate ZBX-10962 with detailed instructions.
|
|
By the way, the ICONV detection bug on FreeBSD was observed also in ZBX-12466.
The bug occurs if libiconv package is installed AND --with-libcurl configure option is used.
Workaround - add --with-iconv=/usr/local to configure options.
|
|
Investigation continues for supported versions in ZBX-10962
|
Generated at Sat Apr 27 02:17:30 EEST 2024 using Jira 9.12.4#9120004-sha1:625303b708afdb767e17cb2838290c41888e9ff0.
Screen Shot 2015-09-29 at 15.08.31.png
freebsd_backtrace_x64.diff
list.txt