Centos 6.7 x86_64

Can you please attach the results of objdump -DSswx zabbix_server too?

HI,

I added an attachment with the result objdump -DSswx zabbix_server

Potentially related issue: ZBX-10940.

HI Aleksandrs,

In my enviroment i have packages versions
OpenIPMI-devel-2.0.16-14.el6.x86_64
OpenIPMI-libs-2.0.16-14.el6.x86_64
OpenIPMI-2.0.16-14.el6.x86_64
ipmitool-1.8.11-29.el6_7.x86_64

This is the latest version of these packages, available in the official Centos repository. Only ipmitool have new version, ipmitool-1.8.15-2. If is needed i can compile from source OpenIPMI in version OpenIPMI-2.0.22 (latest version). How is your recomendation?

The backtrace is too short and there is no information on what the server was doing prior to the crash. Theoretically, you could enable DebugLevel=4 for IPMI pollers, although it might generate quite a bit of logs if you have 17 IPMI pollers and it crashes once a month...

Compiled and installed to the latest version of OpenIPMI (2.0.22)

Any new crashes since July?

After OpenIPMI update to 2.0.22, problem doesn't occur.

Thank you! Closing as "Cannot reproduce" then. If the issue manifests again, please reopen.

This small change helped to prevent 'ipmi poller' crash:

Index: src/zabbix_server/poller/checks_ipmi.c
===================================================================
--- src/zabbix_server/poller/checks_ipmi.c      (revision 64040)
+++ src/zabbix_server/poller/checks_ipmi.c      (working copy)
@@ -1037,8 +1037,6 @@
 {
        int     i;
 
-       h->con->close_connection(h->con);
-
        for (i = 0; i < h->control_count; i++)
        {
                zbx_free(h->controls[i].c_name);

(Note: this change is not yet reviewed and approved).

There is one more place where modified zbx_free_ipmi_connection() is called - in zbx_free_ipmi_handler(). But zbx_free_ipmi_handler() is called only from zbx_on_exit() - we can leave connection open, it will be closed by OS.

Seems like ZBX-10940 is not related to this ZBX-10983 crash. ZBX-10940 is a crash in 'unreachable poller' process, zbx_free_ipmi_connection() do not run there as part of normal operation (except at termination in zbx_on_exit()).

Fixed in development branch svn://svn.zabbix.com/branches/dev/ZBX-10983 .

Took a look at OpenHPI source code (http://openhpi.org/), which also uses OpenIPMI library. It does not seem to call close_connection() either - only ipmi_domain_close().

The fix looks good. I could reproduce the crash and Valgrind complaints using Andris' scenario before the fix, but not with the fix applied.

The only suspicious place is zbx_close_inactive_host(), where we call zbx_domain_close_cb(), which calls ipmi_domain_close(). That last function can fail, in which case we do not free the host in zbx_close_inactive_host(). That leads to a question: if we fail to close a domain on first attempt, what are our chances of succeeding on following attempts? However, while I could reproduce ipmi_domain_close() failing in zbx_connection_change_cb(), which happens in zbx_init_ipmi_host(), I have not managed to reproduce it failing inside zbx_close_inactive_host(). Still, it may be a topic for further consideration.

I decided to keep a host in our list in case ipmi_domain_close() fails. This seems a safer choice as we do not know what data the OpenIPMI library might keep about that host, what our callbacks might be called later.

Fixed in versions:

• pre-3.0.7rc1 r64510,
• pre-3.2.3rc1 r64511,
• pre-3.3.0 (trunk) r64512.

Documented in:

• https://www.zabbix.com/documentation/3.0/manual/introduction/whatsnew307#daemon_improvements
• https://www.zabbix.com/documentation/3.2/manual/introduction/whatsnew323#daemon_improvements

martins-v Reviewed, with minor wording changes. CLOSED.