Instructions how to compile OpenIPMI library and Zabbix sserver with additional debugging ============================================================================ 0. Become 'root': -------------------------- appliance@zabbix:~$ sudo su - [sudo] password for appliance: <---- Enter password of 'appliance' user. 1. Install additional software packages into appliance:: ---------------------------------------- # apt-get install make gcc subversion autoconf automake libc6-dev libpopt-dev libncurses5-dev pkg-config \ libmysqlclient-dev libiksemel-dev libxml2-dev unixodbc-dev libsnmp-dev \ libssh2-1-dev libldap2-dev libcurl4-openssl-dev nano ... Do you want to continue? [Y/n] y 2. Compile OpenIPMI with custom added debugging: ------------------------------------------------ Login as user 'appliance' (if you are 'root', logout from 'root' by typing 'exit'). Go to home directory $ cd Download OpenIPMI source code with added (by me) debug logging: $ wget https://support.zabbix.com/secure/attachment/47725/OpenIPMI-2.0.22-Andris_debug.tgz Unpack the archive: $ tar zxf OpenIPMI-2.0.22-Andris_debug.tgz Go into folder: $ cd OpenIPMI-2.0.22-Andris_debug Compile OpenIPMI: $ CFLAGS="-g -O2 -D_FORTIFY_SOURCE=2" ./configure --prefix=/home/appliance/OpenIPMI-2.0.22-debug-install | tee my_configure.out $ make 2>&1 | tee my_make.out $ make install 2>&1 | tee my_make_install.out Now compile Zabbix. Go to home directory: $ cd $ svn co svn://svn.zabbix.com/tags/3.2.3 zabbix_323 ... Checked out revision .... $ cd zabbix_323 Make a backup of checks_ipmi.c. $ cp src/zabbix_server/poller/checks_ipmi.c src/zabbix_server/poller/checks_ipmi.c.ORG Replace 'checks_ipmi.c' file with a modified one from Jira attachment (I added more logging there): $ wget -O src/zabbix_server/poller/checks_ipmi.c https://support.zabbix.com/secure/attachment/47767/checks_ipmi.c --2017-02-06 16:14:01-- https://support.zabbix.com/secure/attachment/47767/checks_ipmi.c Resolving support.zabbix.com (support.zabbix.com)... 87.110.183.173 Connecting to support.zabbix.com (support.zabbix.com)|87.110.183.173|:443... connected. HTTP request sent, awaiting response... 200 OK Length: 47256 (46K) [text/x-csrc] Saving to: ‘src/zabbix_server/poller/checks_ipmi.c’ 100%[====================================================================================>] 47,256 --.-K/s in 0.02s 2017-02-06 16:14:02 (2.52 MB/s) - ‘src/zabbix_server/poller/checks_ipmi.c’ saved [47256/47256] Compile: $ export LD_RUN_PATH=/home/appliance/OpenIPMI-2.0.22-debug-install/lib $ ./bootstrap.sh $ CFLAGS="-g -O2 -D_FORTIFY_SOURCE=2" ./configure --enable-server --enable-proxy --enable-agent --enable-ipv6 \ --with-net-snmp --with-unixodbc --with-libxml2 --with-libcurl --with-jabber --with-ssh2 \ --with-openipmi=/home/appliance/OpenIPMI-2.0.22-debug-install --with-ldap --with-mysql --with-openssl \ --prefix=`pwd` 2>&1 | tee my_configure.out $ make dbschema $ make install 2>&1 | tee my_make_install.out Check which OpenIPMI libraries will be used: $ ldd sbin/zabbix_server | grep -i ipmi libOpenIPMI.so.0 => /home/appliance/OpenIPMI-2.0.22-debug-install/lib/libOpenIPMI.so.0 (0x00007f668132c000) libOpenIPMIposix.so.0 => /home/appliance/OpenIPMI-2.0.22-debug-install/lib/libOpenIPMIposix.so.0 (0x00007f6681126000) libOpenIPMIutils.so.0 => /home/appliance/OpenIPMI-2.0.22-debug-install/lib/libOpenIPMIutils.so.0 (0x00007f667e98a000) Note the '/home/appliance/OpenIPMI-....' - it should not be '/usr/lib/libOpenIPMI....' Become 'root': appliance@zabbix:~$ sudo su - Stop Zabbix server: # service zabbix-server stop * Stopping Zabbix server zabbix_server [ OK ] Start modified Zabbix server manually: root@zabbix:~# sudo -u zabbix /home/appliance/zabbix_323/sbin/zabbix_server -c /etc/zabbix/zabbix_server.conf Check Zabbix processes: # ps -fu zabbix UID PID PPID C STIME TTY TIME CMD zabbix 876 1 0 08:15 ? 00:00:00 /usr/sbin/zabbix_agentd zabbix 882 876 0 08:15 ? 00:00:04 /usr/sbin/zabbix_agentd: collector [idle 1 sec] zabbix 883 876 0 08:15 ? 00:00:00 /usr/sbin/zabbix_agentd: listener #1 [waiting for connection] zabbix 884 876 0 08:15 ? 00:00:00 /usr/sbin/zabbix_agentd: listener #2 [waiting for connection] zabbix 885 876 0 08:15 ? 00:00:00 /usr/sbin/zabbix_agentd: listener #3 [waiting for connection] zabbix 886 876 0 08:15 ? 00:00:01 /usr/sbin/zabbix_agentd: active checks #1 [idle 1 sec] zabbix 913 1 0 08:15 ? 00:00:24 java -server -Dlogback.configurationFile=/etc/zabbix/zabbix_java_gateway_logba zabbix 16716 16715 0 16:15 pts/0 00:00:00 -su zabbix 16721 1 0 16:15 ? 00:00:00 /home/appliance/zabbix_323/sbin/zabbix_server -c /etc//zabbix/zabbix_server.co zabbix 16723 16721 0 16:15 ? 00:00:00 /home/appliance/zabbix_323/sbin/zabbix_server: configuration syncer [waiting 6 ..... Increase logging level for 'unreachable poller' processes: $ /home/appliance/zabbix_323/sbin/zabbix_server -c /etc//zabbix/zabbix_server.conf -R log_level_increase="unreachable poller" zabbix_server [16780]: command sent successfully When communication problems start then IPMI devices are shfted to unreachable poller for monitoring. Then in /var/log/zabbix/zabbix_server.log additional OpenIPMI debug messages can be seen, like: 16021:20170206:153513.872 INFO: OpenIPMI: In ipmi_open_domain(): name:3 16021:20170206:153513.872 INFO: OpenIPMI: ipmi_open_domain(): name:3, adding known domain:0x13fc9a0 16021:20170206:153513.872 INFO: OpenIPMI: add_known_domain(): adding domain:0x13fc9a0 16021:20170206:153513.872 INFO: OpenIPMI: ipmi_open_domain(): name:3, domain:0x13fc9a0 id for new domain is 0x13fc9a0 16021:20170206:153513.872 INFO: OpenIPMI: ipmi_open_domain(): name:3, calling call_domain_change(0x13fc9a0, IPMI_ADDED) 16021:20170206:153513.872 INFO: OpenIPMI: End ipmi_open_domain(): name:3, calling _ipmi_domain_put(0x13fc9a0) and returning rv:0 16021:20170206:153513.872 In zbx_perform_openipmi_ops() host:'[192.168.3.40]:9007' phost:0x13fb580 from zbx_init_ipmi_host() 16021:20170206:153514.873 INFO: OpenIPMI: rsp_timeout_handler(): cb_data:0x148b750 16021:20170206:153515.875 INFO: OpenIPMI: rsp_timeout_handler(): cb_data:0x148b750 Now the modified Zabbix server is running. Note that if you have a crontab for autamatic restarting a crashed Zabbix server, your script will restart the original, unmodified server. Then get the logfile /var/log/zabbix/zabbix_server.log and write to me.... To stop Zabbix server: # service zabbix-server stop * Stopping Zabbix server zabbix_server [ OK ] This command starts unmodified Zabbix server: # service zabbix-server start * Starting Zabbix server zabbix_server [ OK ] ============================================================================