[ZBXNEXT-7746] Add more 'command line' options to IPMI polling Created: 2022 May 26  Updated: 2022 May 26

Status: Open
Project: ZABBIX FEATURE REQUESTS
Component/s: Server (S)
Affects Version/s: None
Fix Version/s: None

Type: Change Request Priority: Medium
Reporter: David Millians Assignee: Andris Zeila
Resolution: Unresolved Votes: 0
Labels: IPMI
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

Alma 8, MySql, Z6.0.x



 Description   

Hello.
(TLDR; my ipmi queries need a delay; has anybody had a workaround?)
I have a new install of Z6 running on AlmaLinux/RH8, that I started to replace our older Z4 box. (That box had been migrated from SuSe to Centos 7, and it just wasn't working right. Been a happy zabbix user for 7+ years)
As part of this, I'm moving things slowly over from one box to another, and making sure all the data is coming in fine, the alerts are relevant, creating the new dashboards to replace screens, etc.
(Aside: I LOVE the new dashboards and visualization. They are beautiful!)

It's mostly been fine- I did have a weird alert because a temperature was above 10, when all the triggers are at 65, and I'm still investigating that- but the only major thing I have a problem with so far is retrieval of data via IPMI from some supermicro/intel boards (s2600wft and s2600wt2r). It does not come in but every 10-15 minutes if I'm lucky.
To be honest, we had acquired these new servers and on z4, I hadn't really worried about them because I had other boxes at each of their locations that I was getting good data from via Dell idracs for environmentals, and I could get the VMs data using the agent, so the hardware stats weren't critical. Now it is, since the old hardware is gone.

I tried figuring a way to get the data via SNMP, but apparently, the boards don't provide SNMP that way- they can only send traps. Boo!

I've got plenty of HD space, speed, etc. There are no queues waiting or delayed. I am getting good consistent data everywhere else with the few Dells I've moved over, and with some Lenovo, and with all the agents on the box. There are no errors showing up. IPMItool queries work just fine and I can see all the data. I have NOT done a full debug run yet; that's next. But I poked a little bit more, and saw this in the docs for the recent board OOBM bios updates:
"ipmitool is not working well when running in high load network. We recommend to add extra timeout by using “-N 5”. Default is 1 second for RMCP+, which is not enough. –N 5 will set 5 second as timeout. So the command will look like: ipmitool –I lanplus –H ip –U user –P password –C 17 –N 5 command

I dug in the code for ipmipoller.c, and it appears that there is a constant value of 1 for the timeout value, and it doesn't look like I can change that easily without changing and compiling or submitting a WHOLE bunch of code to allow it to be a variable, and y'all do NOT want my code. I am a sysadmin, and it's been 30+ years since I did any C:
while (ZBX_IS_RUNNING())
{
const int ipc_timeout = 2;
const int ipmi_timeout = 1;

So I really need a way to change that, preferably on the IPMI tab of hosts. 






[ZBXNEXT-7670] Move IPMI settings/parameters from host to interface level Created: 2022 May 02  Updated: 2022 May 03

Status: Open
Project: ZABBIX FEATURE REQUESTS
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Change Request Priority: Trivial
Reporter: Nathan Liefting Assignee: Unassigned
Resolution: Unresolved Votes: 0
Labels: hostconfigform, interface, ipmi
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Description   

Like with SNMP I think setting up IPMI interfaces on the Interface level is more logical and useful than doing it host-wide by having a parameter tab on the top of the host configuration modal window.

Proposal to move it from host-wide to interface specific. 






[ZBXNEXT-3643] please add redfish (an alternative for IPMI) Created: 2017 Jan 04  Updated: 2019 Nov 15

Status: Open
Project: ZABBIX FEATURE REQUESTS
Component/s: Frontend (F), Server (S)
Affects Version/s: None
Fix Version/s: None

Type: New Feature Request Priority: Trivial
Reporter: Stefan Assignee: Unassigned
Resolution: Unresolved Votes: 14
Labels: IPMI, redfish
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Sub-task
depends on ZBXNEXT-1261 Enterprise Connector to Monitor Stora... Open

 Description   

it would be very nice if you add redfish support, that is a more powerfull alternative to IPMI..
In some Server manufactors like Supermicro, Dell, HP it can be already used, to read data and send commands
take a look at this page: http://redfish.dmtf.org/



 Comments   
Comment by Francine SAUVAGE [ 2019 Sep 07 ]

Please change Priority to Critical now as Redfish is growing : Dell; Intel, IBM, ...  and some constructors will soon deliver their products without IPMI for security reason

Comment by Stefan [ 2019 Nov 14 ]

since we've http-agent and a powerfull javascript, json thing in zabbix we can do this. I'm working on a template for this

Comment by Stefan [ 2019 Nov 15 ]

OK, it looks like that we need ZBXNEXT-1527 for this, because /redfish/v1/Chassis/<ID>/Thermal or /redfish/v1/Chassis/<ID>/Power/PowerSupplies/<ID> is an variable URL because <ID> can change from server to server. So we need nested LLD first to solve this or we create a Template per vendor/server

Comment by Francine SAUVAGE [ 2019 Nov 15 ]

Hi @Stefan, I am so happy that you care about Redfish !

I fully agree with you about "variable Urls".

I guess /redfish/v1/Chassis/ is a ChassisCollection schema so yes, we need a nested LLD following "Contains" members to obtain the list of Chassis.

If you care : I would prefer a solution without agent: http(s) requests could be done remotely by the server without agent, no ? May be on server agent ?

Tell me if you need help about Redfish specification, I know much about that standard.

Regards,

Francine





[ZBXNEXT-3519] Support more IPMI network function codes Created: 2016 Oct 29  Updated: 2016 Oct 29

Status: Open
Project: ZABBIX FEATURE REQUESTS
Component/s: Proxy (P), Server (S)
Affects Version/s: 3.2.1
Fix Version/s: None

Type: Change Request Priority: Minor
Reporter: Marc Assignee: Unassigned
Resolution: Unresolved Votes: 0
Labels: actionoperations, globalscripts, ipmi, remotecommands
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Description   

How about making IPMI remote commands more powerful by supporting raw commands as well?

raw <netfn> <cmd> [data]





[ZBXNEXT-2855] IPMI chassis/BMC monitoring Created: 2015 Jun 24  Updated: 2018 Oct 19

Status: Open
Project: ZABBIX FEATURE REQUESTS
Component/s: Agent (G)
Affects Version/s: None
Fix Version/s: None

Type: Change Request Priority: Major
Reporter: kylix.tan Assignee: Unassigned
Resolution: Unresolved Votes: 2
Labels: ipmi, macros
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Description   

for Server's hardware monitoring, mostly by IPMI. the following feature missing from IPMI Agent just make it not optable.
1. no autodiscovery, external check scripts is needed to discovery sensor
2. macro is not fully supported, such as IPMI.IP, IPMI.USER, IPMI.PASS. In 3.0, some improvement appeared, such as sensor and unit support for LLD macro, that is very welcome
3. NOt fully support other IPMI item beyond "sensor", such as chassis,bmc

Zabbix is an amazing monitoring solution, hope you could make it better



 Comments   
Comment by Stefan [ 2015 Jun 24 ]

its similiar to ZBXNEXT-1799

Comment by Aleksandrs Saveljevs [ 2015 Jun 25 ]

IPMI low-level discovery is being tracked at ZBXNEXT-1423.

Comment by Aleksandrs Saveljevs [ 2015 Jun 25 ]

Request for {IPMI.IP} is being tracked at ZBXNEXT-1154. Macros {IPMI.USER} and {IPMI.PASS} are also there in the comments.

Comment by Aleksandrs Saveljevs [ 2015 Jun 25 ]

So it seems like the only thing that remains in this issue is support for monitoring chassis and BMC.

Comment by Branden [ 2018 Oct 19 ]

Up-vote for Chassis support as it would be great to have overall health status.

 

 





[ZBXNEXT-2674] Add ipmi checks into network discovery Created: 2015 Jan 22  Updated: 2021 Dec 03

Status: Open
Project: ZABBIX FEATURE REQUESTS
Component/s: Proxy (P), Server (S)
Affects Version/s: None
Fix Version/s: None

Type: Change Request Priority: Trivial
Reporter: Ivan Bayan Assignee: Unassigned
Resolution: Unresolved Votes: 13
Labels: ipmi, networkdiscovery
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Description   

I have a lot of IPMI devices and i want to monitor them, first i thought that i can use IPMI checks in network discovery like for SNMP, but later noticed that zabbix does not have IPMI checks for discovery. I searched for that functionality in documentation for later version and found nothing, searched for same feature request in zabbix Jira and again found nothing.
I asked about that on community forum and got reply that this is not implemented yet, so i decided to create feature request.
Also it will be nice to add ability to fetch not only sensors data, but 'fru print', 'chassis status' and 'bmc info' because there may be placed critical information about PSU status and board manufacturer.


  1. ipmitool -H 10.41.0.30 -U ADMIN -E fru print 0
    Board Mfg Date : Mon Jan 1 09:00:00 1996
    Board Mfg : Super Micro
    Board Product : IPMI 2.0
    Board Serial :
    Board Part Number : Winbond Hermon
    Product Manufacturer : Super Micro
    Product Name : IPMI 2.0
    Product Part Number : Winbond Hermon
    Product Serial :
  1. ipmitool -H 10.41.0.30 -U ADMIN -E chassis status
    System Power : on
    Power Overload : false
    Power Interlock : inactive
    Main Power Fault : false
    Power Control Fault : false
    Power Restore Policy : always-off
    Last Power Event : ac-failed
    Chassis Intrusion : inactive
    Front-Panel Lockout : inactive
    Drive Fault : false
    Cooling/Fan Fault : false
    Front Panel Control : none

Link on the thread: https://www.zabbix.com/forum/showthread.php?p=160859



 Comments   
Comment by Darryn van Tonder [ 2018 Oct 09 ]

+1. We've got an ugly work around in scripts and would love to see IPMI discovery as an option.

Comment by dimir [ 2021 Feb 02 ]

With new Template for IPMI monitoring https://www.zabbix.com/integrations/ipmi this feature becomes even more needed. Because otherwise the template cannot be used together with Network Discovery.

Comment by Tim Hammond [ 2021 Dec 03 ]

I have search all over the place to figure out how to add a IPMI interface to a host during a network discovery and to no avail.

Has anyone figured out how to even add the IPMI interface during a discovery?

I have SNMPv3 working just fine.





[ZBXNEXT-1799] LLD: use {#MACROS} as "IPMI sensor" value for "IPMI agent" Created: 2013 Jun 19  Updated: 2019 Jul 16  Resolved: 2014 Oct 17

Status: Closed
Project: ZABBIX FEATURE REQUESTS
Component/s: Server (S)
Affects Version/s: 2.0.6
Fix Version/s: 2.5.0

Type: Change Request Priority: Trivial
Reporter: Yuriy Smetana Assignee: Unassigned
Resolution: Fixed Votes: 5
Labels: ipmi, lld, macros
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

Ubuntu 12.10


Attachments: PNG File 1.png     PNG File 2.png    

 Description   

Hi!

It seems user can not use Low Level Discovery macros for "IPMI agent" ("IPMI sensor" value) in item prototype.

I have discovery rule, which returns IMPI-sensor names (via corresponding LLD macros, i.e. {#SENSOR_ID}) .

I can use this macros as Item Name, as Item Key but I can not use it as "IMPI sensor" (item calculation error: "sensor or control {#SENSOR_ID}@[192.168.252.206]:623 does not exis").

This feature is must-have for IPMI low-level discovery.

Thank you.



 Comments   
Comment by Andras Fabian [ 2013 Dec 06 ]

Its a real pity, that this approach doesn't works! Especially IPMI sensors are a great thing to discovery via LLDs (as its often a pain to use "hard coded", because the available sensors can differ quite often between different servers etc:!) But if we can't use the discovered values in a simple field like "IPMI sensor",then it renders the whole LLD approach with "IPMI checks" unusable. The only solution is now to build our own IPMI check script and use it as "external script" ...
Please consider changing this behavior!

PS. and I just realized: overusing external scripts is not recommended ... so there are not even much ways to work around this!

Comment by Tatapoum [ 2014 May 14 ]

I agree. It would really be handy. The IPMI checks are quite useless without LLD.

Comment by Alexander Vladishev [ 2014 Oct 17 ]

Available in the development branch svn://svn.zabbix.com/branches/dev/ZBXNEXT-2079

Comment by Alexander Vladishev [ 2014 Oct 20 ]

Available in pre-2.5.0 (trunk) r50042.

Comment by richlv [ 2019 Jul 16 ]

If documentation was updated for this issue, links here might be useful (to what'snew, IPMI pages perhaps).

Comment by richlv [ 2019 Jul 16 ]

From the description, it seems like this did not implement LLD for IPMI, only added LLD macro support in sensor names.
For sensor discovery one might want to vote on ZBXNEXT-1423.





[ZBXNEXT-1671] provide cli utility to list ipmi sensors Created: 2013 Mar 18  Updated: 2013 Mar 18

Status: Open
Project: ZABBIX FEATURE REQUESTS
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: New Feature Request Priority: Major
Reporter: richlv Assignee: Unassigned
Resolution: Unresolved Votes: 2
Labels: ipmi
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Description   

our current ipmi discrete sensor documentation suggests to enable debuglevel4 to print out discrete sensors. this is not very user friendly... also not that easy to do, as it requires zabbix server restart and on a larger system - digging through hundreds of megabytes of logs.

it would be great to split out/reuse that code for a very simple cli utility to print out discrete sensors the way we need them

https://www.zabbix.com/documentation/2.2/manual/config/items/itemtypes/ipmi



 Comments   
Comment by Oleksii Zagorskyi [ 2013 Mar 18 ]

Isn't it possible to use any existing ipmi related tool ?

Comment by richlv [ 2013 Mar 18 ]

according to andris, they do not output data in the reliable format we need

Comment by Andris Mednis [ 2013 Mar 18 ]

For example, "ipmitool" shows about "CATERR" sensor:

$ ipmitool -v -U zabbix -H 192.168.xxx.xxx -I lanplus -L user sdr type "Processor"
Password: 
Sensor ID              : CATERR (0x68)
 Entity ID             : 3.96 (Processor)
 Sensor Type (Discrete): Processor
 Assertions Enabled    : Digital State
                         [State Asserted]
 Deassertions Enabled  : Digital State
                         [State Asserted]

or

$ ipmitool -U zabbix -H 192.168.xxx.xxx -I lanplus -L user sensor list all
...
CATERR           | 0x0        | discrete   | 0x0000| na        | na        | na        | na        | na        | na

Free-IPMI shows:

$ /usr/sbin/ipmi-sensors -l USER -h 192.168.xxx.xxx -u zabbix -P
ID | Name             | Type                   | Reading    | Units | Event
...
19 | CATERR           | Processor              | N/A        | N/A   | 'OK'

The output is nicely decoded and formatted but it was difficult for me to determine the value of "Reading type code" and "Sensor type code" without guessing. Even sensor name does not always have to be ASCII or UNICODE-encoded, according to IPMI specifications it could be a binary encoded string. Therefore I decided to show exactly numerical values of attributes in Zabbix logfile:

  8358:20130318:111122.170 Added sensor: host:'192.168.xxx.xxx:623' id_type:0 id_sz:7 id:'CATERR' reading_type:0x3 ('discrete_state') type:0x7 ('processor') full_name:'(r0.32.3.0).CATERR'

If we could find a utility which shows values precisely, the problem would be solved.





[ZBXNEXT-1578] Ability to use user macros for IPMI user and password Created: 2013 Jan 17  Updated: 2020 Apr 28  Resolved: 2020 Apr 28

Status: Closed
Project: ZABBIX FEATURE REQUESTS
Component/s: Proxy (P), Server (S)
Affects Version/s: 2.0.4
Fix Version/s: 5.0.0beta2, 5.0 (plan)

Type: New Feature Request Priority: Trivial
Reporter: Andrej Kacian Assignee: Andrejs Kozlovs
Resolution: Fixed Votes: 11
Labels: ipmi, macros
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: PNG File Selection_220.png    
Team: Team C
Sprint: Sprint 62 (Mar 2020), Sprint 63 (Apr 2020)
Story Points: 0.25

 Description   

I would appreciate to be able to define a macro on host or template level, and then use that macro in user and password field in IPMI interface configuration for any host. That way, if these access credentials need to be changed in the future, it will be possible to change them in just one place, instead of having to go through all hosts one by one.
Thanks!



 Comments   
Comment by evand [ 2016 Jun 13 ]

I'm using Zabbix 3.0.2. © 2001–2016, Zabbix SIA, found this func is also missing. Waiting for updates.

Comment by Darryn van Tonder [ 2018 Oct 09 ]

On Zabbix 3.4.14 and this is really a surprise. With close to 1000+ IPMI objects managed by various teams, this simple feature is a pain for us.

Comment by Andrey Shibanov [ 2019 Sep 04 ]

Actual

Comment by Kirill Sklyarov [ 2019 Sep 04 ]

very need this future hope you add it soon

Comment by Andrejs Kozlovs [ 2020 Apr 20 ]

Available in:

Documentation updated:





[ZBXNEXT-1423] IPMI sensor lld Created: 2012 Sep 13  Updated: 2020 Feb 12  Resolved: 2020 Feb 01

Status: Closed
Project: ZABBIX FEATURE REQUESTS
Component/s: Proxy (P), Server (S)
Affects Version/s: None
Fix Version/s: 5.0.0alpha1, 5.0 (plan)

Type: New Feature Request Priority: Minor
Reporter: richlv Assignee: Andrejs Kozlovs
Resolution: Fixed Votes: 23
Labels: ipmi, lld
Σ Remaining Estimate: Not Specified Remaining Estimate: Not Specified
Σ Time Spent: Not Specified Time Spent: Not Specified
Σ Original Estimate: Not Specified Original Estimate: Not Specified

Attachments: JPEG File ipmi_get.jpg    
Issue Links:
Causes
causes ZBX-17295 IPMI sensors LLD improvement Closed
Duplicate
Sub-task
part of ZBXNEXT-5527 Create IPMI template for Supermicro a... Closed
Sub-Tasks:
Key
Summary
Type
Status
Assignee
ZBXNEXT-5614 Frontend: IPMI sensor lld Specification change (Sub-task) Closed Andrejs Kozlovs  
Team: Team C
Sprint: Sprint 58 (Nov 2019), Sprint 59 (Dec 2019), Sprint 60 (Jan 2020)
Story Points: 3

 Description   

ability to discover (low level discovery) ipmi sensors would be nice. filter would be on sensor name, most likely



 Comments   
Comment by Alexei Vladishev [ 2012 Oct 09 ]

Also it would be great to have plugins supported for LLD (shared libs?), so that the LLD functionality could be extended by third parties.

Comment by Grigoriy Ermolaev [ 2015 Nov 10 ]

ZBXNEXT-1799

Comment by richlv [ 2015 Nov 10 ]

note that this is different from ZBXNEXT-1799 - while that issue allowed to use lld macros/variables in ipmi sensor names, there's still no ipmi discovery item available

Comment by Andrejs Kozlovs [ 2020 Jan 23 ]

Available in:

Comment by Alexei Vladishev [ 2020 Jan 24 ]

Since it is already implemented it would be great to hear your feedback. Feel free to grab alphas and betas of Zabbix 5.0 when available and report back if it works for you as expected.

Comment by Alexander Vladishev [ 2020 Feb 01 ]

Updated documentation:





[ZBXNEXT-300] Add support for ipmi discrete sensor values Created: 2010 Apr 15  Updated: 2013 Jul 23  Resolved: 2013 Apr 02

Status: Closed
Project: ZABBIX FEATURE REQUESTS
Component/s: None
Affects Version/s: 1.8.2
Fix Version/s: 2.1.0

Type: Change Request Priority: Major
Reporter: Mark Carbonaro Assignee: Unassigned
Resolution: Fixed Votes: 21
Labels: ipmi
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: Text File IBM_System_x3550M2_sdr_elist_all.txt     Text File IBM_System_x3550M2_sensor_list_all.txt     Text File IBM_System_x3550_IPMI_sensors.txt     Text File ProLiant-BL460c-G7-sensors-hpblade.txt     Text File ProLiant-BL685c-G7-sensors-sdr-hpblade.txt     Text File ProLiant_BL460c_G1_sdr_elist_all.txt     Text File ProLiant_DL360_G7_sdr_elist_all.txt     Text File ProLiant_DL360_G7_sensor_list_all.txt     Text File ProLiant_DL360_G7_v_sdr_elist_all.txt     Text File ProLiant_DL360_G7_v_sensor_list_all.txt     Text File checks_ipmi.c.patch    
Issue Links:
Duplicate
is duplicated by ZBX-6812 [IPMI agent] Error while getting valu... Reopened

 Description   

At present discrete sensor values are not supported and are needed to monitor drive failure information on most Sun x86 servers.

See this forum threat for some information: http://zabbix.com/forum/showthread.php?t=10762



 Comments   
Comment by Mark Carbonaro [ 2010 Apr 15 ]

Here is the patch from the forum thread that has been updated to apply to 1.8.2 although I don't think it is a complete and proper implementation, but unfortunately I don't poses the skills to fix it up.

Comment by Andy Lubel [ 2010 May 12 ]

We would prefer to monitor the discrete results that show things like whether the service LED is on or there is a fault rather than have 30+ objects monitored per physical server and the trigger complexity (each systems critical thresholds are different), we can read this discrete data universally on every sun system with recent ILOM.

example on a sun server:
PSx/PRSNT
PSx/VINOK
PSx/PWROK

Comment by Alex Deiter [ 2010 May 18 ]

Thanks a lot for patch!

To Alexei: could you please review this patch ?

Patch successfully tested on Sun Fire x4150/x4200/x4450:

14722:20100518:135325.027 In init_ipmi_host([10.1.1.2]:623)
14722:20100518:135325.027 In get_ipmi_host([10.1.1.2]:623)
14722:20100518:135325.027 In get_ipmi_sensor_by_name() FB0/FM0/PRSNT@[10.1.1.2]:623
14722:20100518:135325.027 In read_ipmi_sensor() FB0/FM0/PRSNT@[10.1.1.2]:623
14722:20100518:135325.195 In get_ipmi_sensor()

> sensor info test(30.0)
Sensor
Name: test(30.0).FB0/FM0/PRSNT
LUN: 0
Number: 161
Event Reading Type: 8
Event Reading Type Name: discrete_device_presense
Type: 37
Type Name: entity_presense
Event Support: per state
Init Scanning: true
Init Events: true
Init Thresholds: false
Init Hysteresis: false
Init Type: true
Init Power Up Events: true
Init Power Up Scanning: true
Ignore If No Entity: false
Auto Rearm: true
OEM1: 0
Id: FB0/FM0/PRSNT
Event
Offset: 0
Name: device removed/absent
Supports: assertion
Event
Offset: 1
Name: device inserted/present
Supports: assertion
>

Comment by Thomas Lohmüller [ 2011 Nov 14 ]

I tried to monitor the Service LED on HP servers. It's also a discrete value and does not work. Is there anything going on here? It's still not fixed in 1.8.8.

Comment by Oleksii Zagorskyi [ 2012 Jan 19 ]

An example of a sensor list. FAN sensors work well, PS Status sensor not.
root@zabbix:~# ipmitool -U ADMIN -H 10.10.10.10 -I lanplus -L user sensor list
Password:
FAN 1 | 5476.000 | RPM | ok | 400.000 | 576.000 | 784.000 | 33856.000 | 34225.000 | 34596.000
... trim
FAN 8 | na | RPM | na | na | na | na | na | na | na
CPU1 Vcore | 0.872 | Volts | ok | 0.776 | 0.800 | 0.824 | 1.352 | 1.376 | 1.400
... trim
VBAT | 3.216 | Volts | ok | 2.880 | 2.904 | 2.928 | 3.648 | 3.672 | 3.696
CPU1 Temp | 0x0 | discrete | 0x0000| na | na | na | na | na | na
CPU2 Temp | 0x0 | discrete | 0x0000| na | na | na | na | na | na
System Temp | 30.000 | degrees C | ok | -9.000 | -7.000 | -5.000 | 75.000 | 77.000 | 79.000
P1-DIMM1A | 37.000 | degrees C | ok | -9.000 | -7.000 | -5.000 | 65.000 | 70.000 | 75.000
... trim
Chassis Intru | 0x0 | discrete | 0x0000| na | na | na | na | na | na
PS Status | 0x1 | discrete | 0x01ff| na | na | na | na | na | na

Comment by Jc Duss [ 2012 Apr 17 ]

Hi all,

Is this ticket still in development and what is the release target please?

Thank you for your work on it.

JC.

Comment by Oleksii Zagorskyi [ 2012 Apr 17 ]

As I understood the implementation of this feature turned out to be absolutely NOT trivial.
It requires to find out a model/approach and then complex development.

Comment by richlv [ 2012 Jul 24 ]

i thought i described it somewhere, but just can't find it... so the situation is like this.

discrete sensors may return a value where any bit can be meaningful. so first bit may be some status of the first psu, second of the second one etc. or maybe one is intrusion sensor, another is humidity sensor etc.

patch returns position of the first set value (0001000011101100 would get you 4). but that would completely ignore the remaining set bits.

possible approaches and their drawbacks :

a) just allow to store discrete values in text items, and write triggers against them matching strings. triggers become very hard to manage, there is no separate history for individual bits and there is no possibility to graph this (up/down), in the history there is no way to figure out what's what (without knowing bits by heart)

b) for each discrete sensor auto-create individual items for each bit. trigger functions are the same, history for each bit is separate, graphing possible, value mapping can be set up and history is easy to use. but if all you need is a single bit from the discrete value, this is an overkill.

maybe the solution is some mix (or both) of the above, but that's far from trivial at that point

<Sasha>
c) just allow to store discrete values in numeric items and add support bitwise operators in trigger expressions. Not possible graphing and value mapping.

d) or add preprocessing for numeric items and store only useful bits.

<richlv> as per openipmi docs cited below, "each bit may represent the initialization state of a piece of software". in this case the suggested patch that just returns first set bit would be best, so it would be highly desirable to have such functionality as an option

Comment by richlv [ 2012 Jul 25 ]

http://openipmi.sourceforge.net/IPMI.pdf has some details as well. citing it :

Sensor monitor something about an object. IPMI defines many types of sensors, but groups them into two
main categories: Threshold and discrete. Threshold sensors are “analog”, they have continuous (or mostly
continuous) readings. Things like fans speed, voltage, or temperature.
Discrete sensors have a set of binary readings that may each be independently zero or one. In some
sensors, these may be independent. For instance, a power supply may have both an external power failure
and a predictive failure at the same time. In other cases they may be mutually exclusive. For instance, each
bit may represent the initialization state of a piece of software.

Discrete sensors report their readings in a 16-bit bitmask, each bit generally representing a discrete state.
For instance, consider the slot/connector sensor. Bit 0 tells if there is a fault. Bit 2 tells if a device is present
in the slot. Bit 5 tells if power is off on the slot. Each bit tells a completely independent state and they may
each be zero or one independently.

Comment by Oleksii Zagorskyi [ 2012 Aug 10 ]

bitwise operators in triggers requested in ZBXNEXT-1329

Comment by Andris Mednis [ 2013 Jan 11 ]

Document at http://openipmi.sourceforge.net/IPMI.pdf is from February 10, 2006.
To get a newer version download the latest source code archive from http://sourceforge.net/projects/openipmi/. For example, OpenIPMI-2.0.20-rc1/doc/IPMI.pdf is from July 2, 2012.

Comment by Andris Mednis [ 2013 Jan 15 ]

While setting up a Zabbix server (from trunk/) test system on Debian GNU/Linux 64-bit testing ("Wheezy"), I noticed that:

  • using Zabbix with Debian OpenIPMI 2.0.16-1.3 packages produces error:
    11206:20130115:112404.996 In setup_done() phost:0x2784470 host:'[xxx.xx.xx.xx]:623'
    11206:20130115:112404.996 setup_done() fail: [33554441] Unknown error 33554441
    11206:20130115:112404.996 End of setup_done():NETWORK_ERROR
    11206:20130115:112404.996 In domain_closed() phost:0x2784470 host:'[xxx.xxx.xx.xx]:623'
    11206:20130115:112404.996 End of domain_closed()
    11206:20130115:112404.996 End of init_ipmi_host():0x2784470
    11206:20130115:112404.996 Item [xxxxxxx:baseboard_temp] error: cannot connect to IPMI host: [33554441] Unknown error 33554441
  • using Zabbix with locally compiled OpenIPMI-2.0.19 works fine (with analog sensors).
Comment by Andris Mednis [ 2013 Feb 07 ]

1) Can you tell which discrete sensors are the most important to support in Zabbix ?

2) Things like "Cooling/Fan fault detected", "Drive Fault" do not show up under sensors, they belong to "chassis status". Should they be supported as if they were discrete sensors ?

Comment by Sergey Syreskin [ 2013 Feb 08 ]

Hi Andris,

Here is a list of sensors for IBM SystemX 3550 M2, which we use in our production environment. We definitely need to know about power supply and power supply fans failures. Then I think, Host Power, One of PCI Error, One of the DIMMs, One of the CPUs, Sys Board Fault, CPU 1 OverTemp, CPU 2 OverTemp.

Regards,
Sergey

Comment by Andris Mednis [ 2013 Feb 08 ]

Thanks, Sergey!

Your list of IBM SystemX sensors is very helpful as an example.
Can You also send an output of "ipmitool .... sdr elist all" ?

Does anybody have an examples of "ipmitool ... sensor list all" and "ipmitool .... sdr elist all" on a blade enclosure with multiple blades ?

Best regards,
Andris

Comment by Sergey Syreskin [ 2013 Feb 08 ]

This is not a blade server, however I upload both files, just in case.

andris Thanks! You helped me to understand more.

Comment by Tomasz Pawelczak [ 2013 Feb 11 ]

Ipmi sensors info for ProLiant BL460c G7 blades. For HP only blades have ipmi sensors - not whole chassis.

See ProLiant-BL460c-G7-sensors-hpblade.txt

Comment by Andrej Kacian [ 2013 Feb 11 ]

Attaching also sdr+sensors listing for a HP ProLiant 685c G7 blade.

Comment by Sergey Syreskin [ 2013 Feb 11 ]

Added ipmitool output for HP ProLiant DL360 G7.

Comment by Andris Mednis [ 2013 Feb 11 ]

Thanks for samples!

Comment by Andris Mednis [ 2013 Feb 14 ]

Hi!
Initial draft of specifications at https://www.zabbix.org/wiki/Docs/specs/ZBXNEXT-300.

Trunk-based development branch
svn://svn.zabbix.com/branches/dev/ZBXNEXT-300-trunk
contains work done so far (reading discrete sensors, bitwise function AND for testing bits in trigger expressions).
Waiting for your feedback...

Comment by Oleksii Zagorskyi [ 2013 Feb 15 ]

I think that new trigger function name (and) is not the best.
Maybe we could select something more intuitively clear ? For example "bit" or "bitwise" or something similar.

Maybe then mode (AND|OR) could be additional function parameter ? (gust an idea)

Also 2nd function parameter (sec or #num) should be optional as well.

Wouldn't it be more "readable" for users to use 1st function parameter in binary form ?
i.e. binary "00001000" instead of decimal "8"

Also a phrase "Sensor "Power Unit Stat" has "Event/Reading Type Code" 0x6f and "Sensor Type Code" 0x9" is not clear to me, I mean it contains twice "Sensor Type Code" with different values. Is this ok ?

Comment by Andris Mednis [ 2013 Feb 15 ]

How about using bitwise function names as in Lua: band(), bnot(), bor(), bxor() ?
bitwise(and, 1, #1) instead of band(1, #1) ? I don't know...

Ok, let's make 2nd function parameter (sec or #num) optional.

Is binary "00001000" sufficient ? Or somebody would want "0x08", too ?

"Event/Reading Type Code" and "Sensor Type Code" are two different codes which characterize a sensor.
Examples of "Event/Reading Type Code": 01h = Threshold (a.k.a. analog), 02h = Discrete(DMI-based “Usage State” STATES), 03h, ...06h= ‘digital’ Discrete(DIGITAL/DISCRETE EVENT STATES),...
Examples of "Sensor Type Code": 01h=Temperature, 02h=Voltage, 03h=Current, 04h=Fan, 05h=Physical Security (Chassis Intrusion), 06=Platform Security
Violation Attempt, 07h=Processor, 08h=Power Supply, ...

Comment by Oleksii Zagorskyi [ 2013 Feb 15 ]

Idea about band(), bnot(), bor(), bxor() looks good, in any case it's much better than just and().

If we would select "bitwise(and, 1, #1)" style then I'd set MODE (and|or) to not 1st position, but to 2nd or last, and there should be defined default value for MODE ("and" probably).
Then most often used function will look more readable, simple, say like: "bitwise(00001000)"

Please ignore my comment about "Sensor Type Code", i was wrong, that phrase in the spec is just a bit complicated.

Comment by Felipe de Moura Vieira [ 2013 Mar 08 ]

An output of ipmitool for a Proliant BL460C G1

Comment by Andris Mednis [ 2013 Mar 08 ]

Thanks, Felipe!
One of sensors is "Virtual Fan"

Comment by Andris Mednis [ 2013 Mar 14 ]

Hi!
You are welcome to read specifications at https://www.zabbix.org/wiki/Docs/specs/ZBXNEXT-300
and test development branch svn://svn.zabbix.com/branches/dev/ZBXNEXT-300-trunk
for:

  • support of discrete sensors,
  • a new function "band()" (bitwise AND) for trigger expressions and calculated items,
  • function "count()" with added operator "band" for trigger expressions and calculated items.

Known problems so far:

  • frontend's "Expression constructor" does not yet support entering "band" operator for count() function.
    Enter expressions without constructor.
  • documentation is not updated. Use specifications at
    https://www.zabbix.org/wiki/Docs/specs/ZBXNEXT-300
    as current documentation and see examples below.

As IPMI discrete sensors change their state rarely, I took a frequently changing item "system.cpu.switches" as an input for testing "band()" and "count()".
Examples I used for testing:

  • calculated items:
    band("system.cpu.switches",#1,18446744073709551360,0) <--- a 64 bit mask to set 8 least significant bits to "0".
    band("system.cpu.switches",#1,1) <--- a value "AND-ed" with mask 1 gives the least significant bit of value.
    band("system.cpu.switches",#1,1,1d)
    band("system.cpu.switches",#100,1,1d)
    band("system.cpu.switches",0,1)
    band("system.cpu.switches",0,1,1d)
    band("system.cpu.switches",100,1)
    band("system.cpu.switches",100,1,1d)
    count("system.cpu.switches",600,0,band) <--- mask 0 "matches" any value. Same as count("system.cpu.switches",600).
    count("system.cpu.switches",600,1,band) <--- how many values in last 10 minutes have "1" in the least significant bit.
    count("system.cpu.switches",600,1,band,60)
    count("system.cpu.switches",600,2/3,band) <--- how many values in last 10 minutes have "10" in 2 least significant bits.
    <--- Bitwise AND with mask 3 sets all bits to "0" except the 2 least significant bits.
    <--- The result is compared to 2 ("10" in binary).
  • trigger expressions:
    {localhost1:system.cpu.switches.count(0,6/7,band)}

    >0 <--- Does the last value contain "110" in the 3 least significant bits ?

    {test_db:Power_Unit_Stat.band(#1,1)}

    =1 <--- Does the last reading of discrete sensor Power_Unit_Stat have "1" in the least significant bit ?

    {test_db:Power_Unit_Stat.band(#1,1,1d)}

    =1

    {test_db:Power_Unit_Stat.band(#20,1)}

    =1

    {test_db:Power_Unit_Stat.band(#20,1,1d)}

    =1

    {test_db:Power_Unit_Stat.band(0,1)}

    =1

    {test_db:Power_Unit_Stat.band(0,1,1d)}

    =1

    {test_db:Power_Unit_Stat.band(20,1)}

    =1

    {test_db:Power_Unit_Stat.band(20,1,1d)}

    =1

Waiting for your thoughts...

Comment by Andris Mednis [ 2013 Mar 14 ]

(1) Documented a new function "band" and a new operator "band" for function "count" in
https://www.zabbix.com/documentation/2.2/manual/appendix/triggers/functions

<richlv> apparently some discrete sensor related documentation has also been added to https://www.zabbix.com/documentation/2.2/manual/config/items/itemtypes/ipmi - anywhere else ?
still missing - whatsnew

andris "whatsnew" documentation added at https://www.zabbix.com/documentation/2.2/manual/introduction/whatsnew220?&#support_for_ipmi_discrete_sensors

<richlv> thanks - i reordered it a bit and added a couple of links - please check that it still looks ok; if so, this can be closed

andris Thanks. CLOSED

Comment by Toms (Inactive) [ 2013 Mar 19 ]

Fronted ready for testing in development branch svn://svn.zabbix.com/branches/dev/ZBXNEXT-300-trunk r34462

Comment by Andris Mednis [ 2013 Mar 22 ]

Fixed in development branch svn://svn.zabbix.com/branches/dev/ZBXNEXT-300-trunk

Comment by Andris Zeila [ 2013 Mar 28 ]

Successfully tested

andris Thanks! Proposed changes accepted.

Comment by Andris Mednis [ 2013 Mar 28 ]

Fixed in version pre-2.1.0 rev.34705

Comment by richlv [ 2013 Apr 02 ]

reopen to close subissue





[ZBXNEXT-7] better ipmi error reporting in frontend Created: 2009 Jan 22  Updated: 2015 Oct 27

Status: Open
Project: ZABBIX FEATURE REQUESTS
Component/s: Frontend (F), Proxy (P), Server (S)
Affects Version/s: None
Fix Version/s: None

Type: Change Request Priority: Minor
Reporter: richlv Assignee: Unassigned
Resolution: Unresolved Votes: 2
Labels: errorreporting, ipmi
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

zabbix 1.6.2


Attachments: File ipmi.patch    

 Description   

when looking at zabbix server log with debuglevel 4, several error cases can be observed :

32414:20090122:124032 EINF: 0 ipmi_lan.c(got_rmcpp_open_session_rsp): BMC returned an auth algorithm that wasn't supported: 1
32414:20090122:124032 Cannot connect to IPMI host. Error 0x16 Invalid argument

32414:20090122:124132 EINF: 0 ipmi_lan.c(auth_cap_done): Requested authentication not supported
32414:20090122:124132 Cannot connect to IPMI host. Error 0x16 Invalid argument

frontend, on the other hand, only shows :
"Cannot connect to IPMI host. Error 0x16 Invalid argument"

it would be helpful, if frontend would also show EINF contents, if available - obviously, they tell more about the problem



 Comments   
Comment by Alexei Vladishev [ 2009 Jun 18 ]

Rich,

Can this be closed? I think it was already fixed, wasn't it?

Alexei

Comment by richlv [ 2009 Jun 18 ]

not sure - do ipmi errors now contain more information ?
i tried to mess with ipmi configuration, but i couldn't get any errors to appear at all
if error messages now contain more information, sure, this report can be closed

Comment by richlv [ 2009 Jun 18 ]

turned out i had managed to crash zabbix_server (yay), which was the reason for missing error message.
if i use some weird auth selection, i still see only the following error in the frontend :

Cannot connect to IPMI host. Error 0x16 Invalid argument

Comment by Sandis Neilands (Inactive) [ 2015 Oct 14 ]

Variation of the theme. In trunk (3.0alpha) I get:

cannot connect to IPMI host: [16777428] Unknown error 16777428

In hex this is 10000D4. We can decode it by looking at include/OpenIPMI/ipmi_err.h:

#define IPMI_IPMI_ERR_TOP       0x01000000
...
#define IPMI_INSUFFICIENT_PRIVILEGE_CC          0xD4

The reason for this problem is that we use zbx_strerror() in setup_done() callback for all errors. zbx_strerror() is a wrapper for POSIX strerror() which handles only standartised errno values. Instead we should use ipmi_get_error_string() as it handles both the standartised errno values and IPMI errors as well.

BTW, some of the standartised errno values returned by the library are quite strange. In case in it couldn't parse a response from IPMI agent it can return EINVAL which translates to "Invalid argument". The intended use of EINVAL is to indicate likely programming errors instead of protocol errors. See POSIX.

This issue gets my vote because I also had to enable debug logs and restart server just to troubleshoot a configuration issue.

Comment by Sandis Neilands (Inactive) [ 2015 Oct 27 ]

The attached patch fixes the problem at least for libopenipmi 2.0.18 on Mint:

  • added IPMI log mapping to Zabbix server log - now severe and fatal IPMI problems will be logged by default:
    • IPMI_LOG_INFO --> LOG_LEVEL_DEBUG;
    • IPMI_LOG_WARNING --> LOG_LEVEL_WARNING;
    • IPMI_LOG_SEVERE --> LOG_LEVEL_ERR;
    • IPMI_LOG_FATAL --> LOG_LEVEL_CRIT;
    • IPMI_LOG_ERR_INFO --> LOG_LEVEL_DEBUG;
    • PMI_LOG_DEBUG_START, IPMI_LOG_DEBUG_CONT, IPMI_LOG_DEBUG_END --> LOG_LEVEL_DEBUG;
    • others (future, past) --> LOG_LEVEL_DEBUG;
  • present the previous "error info" log to the user when an error is passed to the callback function. Unfortunately the "error info" can be as useless as the error code - this is an OpenIPMI issue. However it might be useful in web searches or to look up the problem in their source code.




[ZBX-24195] Fatal error is displayed when cloning a host with IPMI username and password that are too long in Standalone mode Created: 2024 Mar 06  Updated: 2024 Mar 28

Status: READY TO DEVELOP
Project: ZABBIX BUGS AND ISSUES
Component/s: Frontend (F)
Affects Version/s: None
Fix Version/s: 6.0.29rc1, 6.4.14rc1, 7.0.0beta3 (master), 7.0 (plan)

Type: Problem report Priority: Trivial
Reporter: Sergejs Olonkins Assignee: Zabbix Development Team
Resolution: Unresolved Votes: 0
Labels: clone, ipmi, validation
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: GIF File cloning_ipmi_with_failed_validation_issue.gif     PNG File image-2024-03-06-18-38-10-000.png    
Team: Team B
Sprint: Sprint candidates
Story Points: 0.125

 Description   

Problem description: In case if user attempts to clone a host in standalone mode, that has too long IPMI username and password the following fatal error isreturned:

In case if this is done from Host configuration dialog, cloning results in an "Unexpected server error".

Steps to reproduce:

  1. Open any host configuration in standalone mode
  2. Open IPMI tab
  3. Set "Username" and "Password" to some string, that is long enough to trigger the length validation
  4. Press "Clone" button
    Result 1: The above fatal error is returned.
  5. Repeat steps 1-4, but this time open host configuration form in an overlay dialog
    Result 2: "Unexpected server error." error is returned.

Expected: Same as for other fields if their value is not according to validation, just cloning the host should not result in Fatal or Unexpected errors, the new host form just needs to be opened.

Example:






[ZBX-22325] Failed to start IPMI pool on proxy (crashing) Created: 2023 Feb 08  Updated: 2023 Feb 18  Resolved: 2023 Feb 13

Status: Closed
Project: ZABBIX BUGS AND ISSUES
Component/s: Proxy (P)
Affects Version/s: 6.4.0beta5
Fix Version/s: 7.0 (plan)

Type: Problem report Priority: Trivial
Reporter: Ricardo Parras Assignee: Vladislavs Sokurenko
Resolution: Cannot Reproduce Votes: 0
Labels: IPMI, Proxy
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

Almalinux 9.1


Attachments: File zabbix_error_ipmi.rar     Text File zabbix_proxy.log     Text File zabbix_proxy4.log    
Team: Team A
Sprint: Support backlog

 Description   

Steps to reproduce:

  1. Change the proxy setting by enabling StartIPMIPollers to 1
    Restart proxy service

Result:
Reading the log files there will be an ipmi manager #1 process start error

You can also see the message: cannot read IPMI service request and
 Zabbix Proxy stopped. Zabbix 6.4.0beta5 (revision 89049fb7196)

Zabbix Server in the same version does not have this error. Only on proxies.



 Comments   
Comment by Vladislavs Sokurenko [ 2023 Feb 10 ]

Thank you for your report, does issue persist with rc1 ?

Comment by Rostislav Palivoda [ 2023 Feb 13 ]

ricardoparras can you reproduce it on 6.4rc1 version? We think its already fixed.

Comment by Ricardo Parras [ 2023 Feb 18 ]

Hello! Sorry for the delay in responding.
I installed a proxy with the rc1 version and the error was corrected.
Thank you very much.





[ZBX-21386] HP Proliant dl360p g8 ILO IPMI. Некорректные данные. Created: 2022 Jul 25  Updated: 2022 Jul 26

Status: Confirmed
Project: ZABBIX BUGS AND ISSUES
Component/s: Templates (T)
Affects Version/s: 6.0.6
Fix Version/s: None

Type: Problem report Priority: Major
Reporter: alkor80 Assignee: Zabbix Integration Team
Resolution: Unresolved Votes: 0
Labels: IPMI, ipmi.get, template
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

Linux Ubuntuu 20.04.4 5.4.0-109-generic #123-Ubuntu SMP


Attachments: File ILO_Json.json     JPEG File Снимок.JPG    

 Description   

Добрый день.
Столкнулся с такой проблемой.
Нужно мониторить HP dl360p g8 по ILO IPMI. 
Проблемы с получением Корректных значений сенсоров
Если запрашивать через IPMI.get то в json получаем некорректные данные.

Файл ответа в формате json приложен.

Но если запросить через консоль сервера данные 
#ipmitool -U user -P passwd -H ILO_IPAddress -I lanplus -L user sensors
получаем корректно. Cкрин экрана прикладываю



 Comments   
Comment by alkor80 [ 2022 Jul 25 ]

+ Вывод команды  

/usr/sbin/ipmi-sensors -D LAN2_0 -h 172.17.90.32 -u user -p passwd-l USER -W discretereading --no-header-output --quiet-cache --sdr-cache-recreate --comma-separated-output --entity-sensor-names

 

18,System Chassis 1 UID Light,OEM Reserved,N/A,N/A,'OEM Event = 0001h'
19,System Chassis 2 Sys. Health LED,OEM Reserved,N/A,N/A,N/A
20,Air Inlet 01-Inlet Ambient,Temperature,25.00,C,'OK'
21,Processor 1 02-CPU 1,Temperature,40.00,C,'OK'
22,Processor 2 03-CPU 2,Temperature,40.00,C,'OK'
23,Memory Device 1 04-P1 DIMM 1-6,Temperature,37.00,C,'OK'
24,Memory Device 2 05-P1 DIMM 7-12,Temperature,39.00,C,'OK'
25,Memory Device 3 06-P2 DIMM 1-6,Temperature,N/A,C,N/A
26,Memory Device 4 07-P2 DIMM 7-12,Temperature,N/A,C,N/A
27,Memory Device 5 08-P1 Mem Zone,Temperature,33.00,C,'OK'
28,Memory Device 6 09-P1 Mem Zone,Temperature,36.00,C,'OK'
29,Memory Device 7 10-P2 Mem Zone,Temperature,30.00,C,'OK'
30,Memory Device 8 11-P2 Mem Zone,Temperature,31.00,C,'OK'
31,Disk 12-HD Max,Temperature,35.00,C,'OK'
32,System Board 1 13-Chipset 1,Temperature,44.00,C,'OK'
33,System Board 2 14-Chipset1 Zone,Temperature,38.00,C,'OK'
34,Power Supply 4 15-P/S 1 Inlet,Temperature,27.00,C,'OK'
35,Power Supply 5 16-P/S 1 Zone,Temperature,30.00,C,'OK'
36,Power Supply 6 17-P/S 2 Inlet,Temperature,29.00,C,'OK'
37,Power Supply 7 18-P/S 2 Zone,Temperature,32.00,C,'OK'
38,Add-in Card 1 19-PCI #1,Temperature,N/A,C,N/A
39,Add-in Card 2 20-PCI #2,Temperature,N/A,C,N/A
40,Power Unit 1 21-VR P1,Temperature,40.00,C,'OK'
41,Power Unit 2 22-VR P2,Temperature,41.00,C,'OK'
42,Power Unit 3 23-VR P1 Mem,Temperature,30.00,C,'OK'
43,Power Unit 4 24-VR P1 Mem,Temperature,32.00,C,'OK'
44,Power Unit 5 25-VR P2 Mem,Temperature,29.00,C,'OK'
45,Power Unit 6 26-VR P2 Mem,Temperature,30.00,C,'OK'
46,Power Unit 7 27-VR P1Mem Zone,Temperature,28.00,C,'OK'
47,Power Unit 8 28-VR P1Mem Zone,Temperature,28.00,C,'OK'
48,Power Unit 9 29-VR P2Mem Zone,Temperature,26.00,C,'OK'
49,Power Unit 10 30-VR P2Mem Zone,Temperature,27.00,C,'OK'
50,System Board 3 31-HD Controller,Temperature,85.00,C,'OK'
51,System Board 4 32-HD Cntlr Zone,Temperature,49.00,C,'OK'
52,System Board 5 33-PCI 1 Zone,Temperature,37.00,C,'OK'
53,System Board 6 34-PCI 1 Zone,Temperature,39.00,C,'OK'
54,Add-in Card 3 35-LOM Card,Temperature,53.00,C,'OK'
55,System Board 7 36-PCI 2 Zone,Temperature,43.00,C,'OK'
56,System Board 8 37-System Board,Temperature,46.00,C,'OK'
57,System Board 9 38-System Board,Temperature,37.00,C,'OK'
58,System Board 10 39-Sys Exhaust,Temperature,35.00,C,'OK'
59,System Board 11 40-Sys Exhaust,Temperature,39.00,C,'OK'
60,System Board 12 41-Sys Exhaust,Temperature,37.00,C,'OK'
61,Battery 42-SuperCAP Max,Temperature,N/A,C,N/A
62,System Board 1 Fan Block 1,Fan,19.60,%,'transition to Running'
63,System Board 2 Fan Block 2,Fan,19.60,%,'transition to Running'
64,System Board 3 Fan Block 3,Fan,19.60,%,'transition to Running'
65,System Board 4 Fan Block 4,Fan,19.60,%,'transition to Running'
66,System Board 5 Fan Block 5,Fan,21.56,%,'transition to Running'
67,System Board 6 Fan Block 6,Fan,21.56,%,'transition to Running'
68,System Board 7 Fan Block 7,Fan,21.56,%,'transition to Running'
69,System Board 8 Fan Block 8,Fan,21.56,%,'transition to Running'
70,Power Supply 1,Power Supply,50.00,W,'Presence detected'
71,Power Supply 2,Power Supply,65.00,W,'Presence detected'
72,System Board 10 Power Meter,Current,140.00,W,'Device Enabled'
73,Power Supply 3 Power Supplies,Power Supply,N/A,N/A,'Fully Redundant'
75,System Board 9 Fans,Fan,N/A,N/A,'Fully Redundant'
77,System Board 11 Memory,Memory,N/A,N/A,'Presence detected'
78,Disk C1 P1I Bay 1,Drive Slot,N/A,N/A,'Drive Presence'





[ZBX-18964] ipmi.get trim required Created: 2021 Feb 03  Updated: 2023 Jan 03

Status: Reopened
Project: ZABBIX BUGS AND ISSUES
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Patch request Priority: Trivial
Reporter: Siahrei Pisarau Assignee: Zabbix Development Team
Resolution: Unresolved Votes: 0
Labels: IPMI, ipmi.get
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: File ip.get.json     PNG File ipmi1.png     PNG File ipmi2.png    

 Description   

Steps to reproduce:

  1. Create IPMI monitoring item
  2. Set sensors name HDD Temp 

Result:
sensor or control HDD Temp@[]:623 does not exist
Expected:
 item ":hdd.temp" became supported

if check JSON file created by ipmi.get  you can see that there are a lot of extra spaces in  sensor id, if take this id and create an item it works like expected

 

{"id":"HDD Temp        ","name":"(4.32).HDD Temp ","sensor":\{"type":1,"text":"temperature"}

,"reading":{"type":1,"text":"threshold"},"state":{"state":0,"text":""},"value":18.000000,"units":"C","threshold":{"lower":

{"non_crit":10.000000,"crit":5.000000,"non_recover":5.000000}

,"upper":{"non_crit":50.000000,"crit":55.000000,"non_recover":60.000000}}},{"id":"AOC_SAS Temp ","name":"(r0.32.46.0).AOC_SAS Temp

 

 

 

 

 



 Comments   
Comment by Igor Gorbach [ 2021 Feb 08 ]

Hello!

Thank you for reporting the issue!

Please, provide full json output and failed item (hdd.temp) configuration screenshot (with preprocessing)

 

 

Comment by Siahrei Pisarau [ 2021 Feb 09 ]

attached

Comment by Siahrei Pisarau [ 2021 Feb 09 ]

as sson as I add additional spaces  to IPMI sensor  field, all works as expected





[ZBX-17295] IPMI sensors LLD improvement Created: 2020 Feb 11  Updated: 2020 Mar 16  Resolved: 2020 Mar 16

Status: Closed
Project: ZABBIX BUGS AND ISSUES
Component/s: Proxy (P), Server (S)
Affects Version/s: 5.0.0alpha1
Fix Version/s: 5.0.0alpha3, 5.0 (plan)

Type: Problem report Priority: Trivial
Reporter: Andrejs Kozlovs Assignee: Andrejs Kozlovs
Resolution: Fixed Votes: 0
Labels: ipmi, lld
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Causes
caused by ZBXNEXT-1423 IPMI sensor lld Closed
Duplicate
is duplicated by ZBX-17289 Units and lower / upper thresholds sh... Closed
Team: Team C
Sprint: Sprint 61 (Feb 2020)
Story Points: 0.5

 Description   

Some improvements in ipmi.get output JSON are rquested by integration team.



 Comments   
Comment by Andrejs Kozlovs [ 2020 Feb 18 ]

Available in:
*5.0.0beta1 (master) 167b8ea99d5





[ZBX-17289] Units and lower / upper thresholds should be filled by ipmi.get Created: 2020 Feb 10  Updated: 2020 Feb 13  Resolved: 2020 Feb 13

Status: Closed
Project: ZABBIX BUGS AND ISSUES
Component/s: Server (S)
Affects Version/s: 5.0.0alpha1, 5.0 (plan)
Fix Version/s: None

Type: Problem report Priority: Trivial
Reporter: Vjaceslavs Bogdanovs Assignee: Unassigned
Resolution: Duplicate Votes: 0
Labels: ipmi, lld
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Duplicate
duplicates ZBX-17295 IPMI sensors LLD improvement Closed
Team: Team A

 Description   
  1. Units are not always present on the JSON. ipmi_sensor_get_base_unit_string() is used in Zabbix in zbx_got_thresh_reading_cb() and get_discovery_ipmi() - it needs to be examined is it used correctly.
  2. Returning "" (empty string) is bad for making triggers. Can we consider returning some number to signal "no data available"? Number 0 seems problematic, it could be a legitimate value. We need DIFFERENT fictious values for LOW and HIGH thresholds if their values are not available/defined (e.g. -9999999 and +9999999).

Please see additional discussion in ZBXNEXT-1423






[ZBX-17189] Zabbix doesn't see CPU Temp IPMI sensor Created: 2020 Jan 16  Updated: 2020 Jun 03  Resolved: 2020 Jun 03

Status: Closed
Project: ZABBIX BUGS AND ISSUES
Component/s: Server (S)
Affects Version/s: 4.2.8, 4.4.4
Fix Version/s: None

Type: Problem report Priority: Trivial
Reporter: Mariusz Gora Assignee: Arturs Lontons
Resolution: Incomplete Votes: 0
Labels: IBM, IPMI, cpu, imm2, x3550
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

IBM x3550 M4, libopenipmi0 (ver: 2.0.22),Ubuntu 18.04.1 LTS


Attachments: PNG File ipmi-not-exist.png     Text File ipmitool_result.txt     PNG File item.png     Text File zabbix_server.log    

 Description   

Description:

ipmitool shows  IPMI "CPU 2 Temp" sensor exist on this server and return value but zabbix server can't see it.
After run server with DebugLevel=4 and grep zabbix server logs for "CPU " I can see only discovered "CPU 1 Temp" sensor (despite the detection of other sensors for CPU2).**

I suspect more than listed above zabbix server version has is affected this issue

Steps to reproduce:

  1. Set StartIPMIPollers=3 in zabbix_server.conf and restart server.
  2. Prepare IBM x3550 M4 server's BMC (IMM2 in this case) to accept IPMI requests.
  3. Add host with x3650 M4 IMM2 address as IPMI interface and correct IPMI configuration.
  4. Create IPMI item with "IPMI Sensor" value as "CPU 2 Temp".

Result:
Zabbix says "sensor or control CPU 2 Temp@[192.168.yyy.xxx]:623 does not exist" (see attachment).

Expected:
Zabbix server show "CPU 2 Temp" value like it does for "CPU 1 Temp". (Tested server has CPU on botch sockets).

Thanks



 Comments   
Comment by Arturs Lontons [ 2020 Jan 17 ]

Hi,
As per our documentation - have you tried adding a "0" at the end of the sensor name?
https://www.zabbix.com/documentation/current/manual/config/items/itemtypes/ipmi#notes_on_discrete_sensor_names_in_openipmi-2016_2017_2018_and_2019

Comment by Mariusz Gora [ 2020 Jan 17 ]

Hi Arturs,

Yes, I tried but server sayed the same (sensor does not exist).
I created item for the same host with "IPMI Sensor" value as "CPU 1 Temp" and it's works fine. I can see values returned by this item.

Comment by Arturs Lontons [ 2020 Mar 18 ]

Could you please also provide the output of the sensor get via IPMI tool?

Comment by Vladislavs Boborikins (Inactive) [ 2020 Jun 03 ]

Unfortunately, we cannot reproduce the issue without the additional details we've requested earlier. With that in mind, we are closing the case. Please feel free to reopen the ticket if the problem persists.





[ZBX-16664] IPMI poller skips processing if one of the elements is missing information (not installed) Created: 2019 Sep 20  Updated: 2019 Oct 16  Resolved: 2019 Oct 15

Status: Closed
Project: ZABBIX BUGS AND ISSUES
Component/s: Proxy (P), Server (S)
Affects Version/s: 3.0.28, 4.0.12, 4.2.6
Fix Version/s: 4.0.14rc1, 4.2.8rc1, 4.4.1rc1, 5.0.0alpha1, 5.0 (plan)

Type: Incident report Priority: Trivial
Reporter: Edgar Akhmetshin Assignee: Andrejs Kozlovs
Resolution: Fixed Votes: 1
Labels: IPMI, PROXY, Server
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

Ubuntu 16.04 for Server, CentOS 7 for Proxy


Attachments: Zip Archive zbx_bmc_1.zip    
Issue Links:
Causes
Team: Team A
Sprint: Sprint 56 (Sep 2019), Sprint 57 (Oct 2019)
Story Points: 1

 Description   

Steps to reproduce:
Start IPMI monitoring, attach standard or custom IPMI template to the host with physical FAN3-FAN8 installed (or any absent component - not populated dimm slots or PSU), no FAN-1,2 installed:

 32173:20190920:130153.441 In zbx_read_ipmi_sensor() sensor:'FAN2@[x.x.x.x]:623'
 32173:20190920:130153.441 In zbx_perform_openipmi_ops() host:'[x.x.x.x]:623' phost:0x55f06adf59b0 from zbx_read_ipmi_sensor()
 32173:20190920:130153.442 In zbx_got_thresh_reading_cb()
 32173:20190920:130153.442 zbx_got_thresh_reading_cb() fail: [16777419] Unknown error 16777419
 32173:20190920:130153.442 End of zbx_got_thresh_reading_cb():NETWORK_ERROR
 32173:20190920:130153.442 End zbx_perform_openipmi_ops() from zbx_read_ipmi_sensor()
 32173:20190920:130153.442 End of zbx_read_ipmi_sensor():NETWORK_ERROR
 32173:20190920:130153.442 In zbx_ipc_async_socket_send()
 32173:20190920:130153.442 In zbx_ipc_client_send() clientid:0

Expected:
Skip uninstalled components.



 Comments   
Comment by Andrejs Kozlovs [ 2019 Oct 14 ]

Fixed in:

  • 4.0.14rc1 11050926347
  • 4.2.8rc1 f5c4b344a78
  • 4.4.1rc1 27b17f66f49
  • 5.0.0alpha1 (master) 75c022fe261




[ZBX-15935] zbx_perform_all_openipmi_ops can enter infinite loop Created: 2019 Apr 04  Updated: 2019 May 19  Resolved: 2019 May 19

Status: Closed
Project: ZABBIX BUGS AND ISSUES
Component/s: Proxy (P), Server (S)
Affects Version/s: 4.0.6, 4.2.0
Fix Version/s: 4.0.8rc1, 4.2.2rc1, 4.4.0alpha1, 4.4 (plan)

Type: Problem report Priority: Critical
Reporter: Eric A. Borisch Assignee: Andrejs Sitals
Resolution: Fixed Votes: 0
Labels: bug, ipmi
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

OpenIPMI >= 2.0.26


Issue Links:
Causes
caused by ZBX-15578 IPMI times out and fails to read valu... Closed
Team: Team I
Sprint: Sprint 51 (Apr 2019), Sprint 52 (May 2019)
Story Points: 0.5

 Description   

Steps to reproduce:

  1. Have IPMI checks which enter zbx_perform_all_openipmi_ops() and return from perform_one_op() before the timeout expires. perform_one_op() updates the remaining timeout, which will get driven down to 0.0, at which point we just sit and spin.

Result:
**

100% CPU usage on IPMI thread 

Expected:

Not this.

 

Further discussion:

Once perform_one_op() returns before timeout, we never break out of the loop, since we reset start_time each cycle, but we keep comparing duration against the original timeout, not the (updated by perform_one_op()) remaining timeout.

Since perform_one_op() updates the remaining timeout internally (and returns a timeout of {0,0} if it did timeout), skip the start_time tracking completely, and just loop while (tv.tv_sec + tv.tv_usec > 0) and reset the tv to the timeout at the start of each loop:

void	zbx_perform_all_openipmi_ops(int timeout)
{
	struct timeval	tv = {1, 0};

	while (tv.tv_sec + tv.tv_usec > 0)
	{
		int	res;

		tv.tv_sec = timeout;
		tv.tv_usec = 0;

		res = os_hnd->perform_one_op(os_hnd, &tv);

		/* perform_one_op() returns 0 on success, errno on failure (timeout means success) */
		if (0 != res)
		{
			zabbix_log(LOG_LEVEL_DEBUG, "IPMI error: %s", zbx_strerror(res));
			break;
		}
	}
}


 Comments   
Comment by Andrejs Sitals [ 2019 Apr 04 ]

Thanks for your report, eborisch.

perform_one_op() just passes timeout to sel_select() which is defined in selector.c. It doesn't do anything else with timeout.

sel_select() started updating timeout in version 2.0.26 which was released on 2018-12-14. It didn't modify timeout in 2.0.25 and older versions.

Comment by Eric A. Borisch [ 2019 Apr 04 ]

Aha; yes I am running 2.0.27, so that explains it. Perhaps just resetting tv after every call, then.

Thanks for digging into this!

Comment by Andrejs Sitals [ 2019 Apr 08 ]

Fixed in development branch svn://svn.zabbix.com/branches/dev/ZBX-15935

Comment by Eric A. Borisch [ 2019 Apr 23 ]

Doesn't appear to have made the window for 4.0.7...

Comment by Andrejs Sitals [ 2019 Apr 24 ]

Available in versions:

  • pre-4.0.8rc1 e9a4af8a397
  • pre-4.2.2rc1 c0eb9246d67
  • pre-4.4.0alpha1 60d301f0207
Comment by Eric A. Borisch [ 2019 Apr 26 ]

No surprise, but also encountered on FreeBSD with zabbix_proxy performing IPMI checks – pegged the CPU.

Current FreeBSD ports versions of openipmi and zabbix4 are 2.0.27 and 4.0.7, respectively.





[ZBX-15578] IPMI times out and fails to read values when polls aren't frequent enough Created: 2019 Feb 04  Updated: 2021 Dec 02  Resolved: 2019 Mar 22

Status: Closed
Project: ZABBIX BUGS AND ISSUES
Component/s: Proxy (P), Server (S)
Affects Version/s: 4.0.4rc2, 4.2.0alpha3
Fix Version/s: 4.0.6rc2, 4.2.0rc2, 4.2 (plan)

Type: Problem report Priority: Trivial
Reporter: Andrejs Sitals Assignee: Unassigned
Resolution: Fixed Votes: 1
Labels: ipmi
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

Ubuntu 18.04, OpenIPMI 2.0.22 from repository, OpenIPMI 2.0.25 from sources


Attachments: File zabbix_server.log    
Issue Links:
Causes
causes ZBX-15935 zbx_perform_all_openipmi_ops can ente... Closed
causes ZBX-16058 Zabbix server queue (IPMI agent) has ... Closed
Duplicate
Sub-task
Team: Team C
Sprint: Sprint 49 (Feb 2019), Sprint 50 (Mar 2019)
Story Points: 1

 Description   

Steps to reproduce:

  1. Set DebugLevel=4 in zabbix_server.conf
  2. Set StartIPMIPollers=1 in zabbix_server.conf
  3. Create host "IPMI timeouts" with a single IPMI interface
  4. Create item "Power", set update interval to 30s
  5. Start monitoring server's log by running
    tail -f /var/log/zabbix/zabbix_server.log | grep -i -E "(ipmi (poller|manager)|ipmi_lan\.c|network error|restored|power)"
  1. Run Zabbix server

Result:
See log file. There's a repeating pattern between "Connection 0 to the BMC is up" lines. Connection goes up, gets a value, then there are network errors with "Received IPMI error: c3" and "Received IPMI error: ff" log entries.

0xC3 is IPMI_TIMEOUT_CC in OpenIPMI. 0xFF is IPMI_UNKNOWN_ERR_CC.

There are also gaps in "Latest data" when these errors occur.

Expected:

  1. No "Received IPMI error" in log file
  2. No gaps in collected data

Proposed fix:
There are calls to os_hnd->perform_one_op(os_hnd) in some places. If I understand correctly, these calls allow OpenIPMI to do internal processing. If these calls aren't frequent enough, then OpenIPMI cannot do its work and timeouts happen.

To fix this, IPMI poller process should call os_hnd->perform_one_op(os_hnd) while it is waiting for messages from IPMI manager.



 Comments   
Comment by Andrejs Sitals [ 2019 Feb 04 ]

This issue was discovered while trying to reproduce ZBX-15152.

Comment by Andrejs Sitals [ 2019 Mar 19 ]

Fixed in development branch svn://svn.zabbix.com/branches/dev/ZBX-15578

Comment by Andris Mednis [ 2019 Mar 22 ]

Available in versions:

  • pre-4.0.6rc2 r91473
  • pre-4.2.0rc2 (trunk) r91475




[ZBX-14590] Get data error using ipmi interface Created: 2018 Jul 10  Updated: 2019 Jan 25  Resolved: 2019 Jan 25

Status: Closed
Project: ZABBIX BUGS AND ISSUES
Component/s: Server (S)
Affects Version/s: 3.4.11
Fix Version/s: None

Type: Incident report Priority: Major
Reporter: ftdream Assignee: Unassigned
Resolution: Unsupported version Votes: 0
Labels: ipmi, sensors
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

Ubuntu 16.04.1 HP DL388 G7


Attachments: PNG File ipmi.fan.png     PNG File ipmi.fan1.png     PNG File ipmi.power.png    

 Description   

Sensors captured using "ipmitool -I lanplus -H ip -U user -P password sensor" on zabbix server are as follows


UID Light | 0x0 | discrete | 0x0080| na | na | na | na | na | na 
Sys. Health LED | 0x0 | discrete | 0x0080| na | na | na | na | na | na 
Power Supply 1 | 115 | Watts | ok | na | na | na | na | na | na 
Power Supply 2 | na | discrete | na | na | na | na | na | na | na 
Power Supplies | 0x0 | discrete | 0x0880| na | na | na | na | na | na 
Fan 1 | 29.400 | percent | ok | na | na | na | na | na | na 
Fan 2 | 30.576 | percent | ok | na | na | na | na | na | na 
Fan 3 | 36.456 | percent | ok | na | na | na | na | na | na 
Fan 4 | 36.456 | percent | ok | na | na | na | na | na | na 
Fan 5 | 30.968 | percent | ok | na | na | na | na | na | na 
Fan 6 | 13.720 | percent | ok | na | na | na | na | na | na 
Fans | 0x0 | discrete | 0x0180| na | na | na | na | na | na 
Temp 1 | 22.000 | degrees C | ok | na | na | na | na | 41.000 | 45.000 
Temp 2 | 40.000 | degrees C | ok | na | na | na | na | 82.000 | 83.000 
Temp 3 | 40.000 | degrees C | ok | na | na | na | na | 82.000 | 83.000 
Temp 4 | 25.000 | degrees C | ok | na | na | na | na | 87.000 | 92.000 
Temp 5 | 25.000 | degrees C | ok | na | na | na | na | 87.000 | 92.000 
Temp 6 | 28.000 | degrees C | ok | na | na | na | na | 87.000 | 92.000 
Temp 7 | 29.000 | degrees C | ok | na | na | na | na | 87.000 | 92.000 
Temp 8 | 37.000 | degrees C | ok | na | na | na | na | 90.000 | 95.000 
Temp 9 | 31.000 | degrees C | ok | na | na | na | na | 65.000 | 70.000 
Temp 10 | 43.000 | degrees C | ok | na | na | na | na | 90.000 | 95.000 
Temp 11 | 33.000 | degrees C | ok | na | na | na | na | 70.000 | 75.000 
Temp 12 | 40.000 | degrees C | ok | na | na | na | na | 90.000 | 95.000 
Temp 13 | 28.000 | degrees C | ok | na | na | na | na | 70.000 | 75.000 
Temp 14 | 35.000 | degrees C | ok | na | na | na | na | 70.000 | 75.000 
Temp 15 | 32.000 | degrees C | ok | na | na | na | na | 70.000 | 75.000 
Temp 16 | na | | na | na | na | na | na | 70.000 | 75.000 
Temp 17 | na | | na | na | na | na | na | 70.000 | 75.000 
Temp 18 | na | | na | na | na | na | na | 70.000 | 75.000 
Temp 19 | 24.000 | degrees C | ok | na | na | na | na | 70.000 | 75.000 
Temp 20 | 28.000 | degrees C | ok | na | na | na | na | 70.000 | 75.000 
Temp 21 | 29.000 | degrees C | ok | na | na | na | na | 80.000 | 85.000 
Temp 22 | 29.000 | degrees C | ok | na | na | na | na | 80.000 | 85.000 
Temp 23 | 34.000 | degrees C | ok | na | na | na | na | 77.000 | 82.000 
Temp 24 | 30.000 | degrees C | ok | na | na | na | na | 70.000 | 75.000 
Temp 25 | 29.000 | degrees C | ok | na | na | na | na | 70.000 | 75.000 
Temp 26 | 30.000 | degrees C | ok | na | na | na | na | 70.000 | 75.000 
Temp 27 | na | | na | na | na | na | na | 70.000 | 75.000 
Temp 28 | na | | na | na | na | na | na | 70.000 | 75.000 
Temp 29 | 35.000 | degrees C | ok | na | na | na | na | 60.000 | 65.000 
Temp 30 | 66.000 | degrees C | ok | na | na | na | na | 110.000 | 115.000 
Memory | 0x0 | discrete | 0x4080| na | na | na | na | na | na 
Power Meter | 110 | Watts | ok | na | na | na | na | na | na

The zabbix server's log level is 4,“grep 'Added sensor' /tmp/zabbix_server.log”


2789:20180710:110649.140 Added sensor: host:'10.0.0.54:623' id_type:0 id_sz:12 id:'Power Meter' reading_type:0x9 ('discrete_device_enable') type:0x3 ('current') full_name:'0(7.9).Power Meter'
 2789:20180710:110649.140 Added sensor: host:'10.0.0.54:623' id_type:0 id_sz:7 id:'Memory' reading_type:0x6f ('sensor specific') type:0xc ('memory') full_name:'0(7.8).Memory'
 2789:20180710:110649.141 Added sensor: host:'10.0.0.54:623' id_type:0 id_sz:8 id:'Temp 30' reading_type:0x1 ('threshold') type:0x1 ('temperature') full_name:'0(66.12).Temp 30'
 2789:20180710:110649.141 Added sensor: host:'10.0.0.54:623' id_type:0 id_sz:8 id:'Temp 29' reading_type:0x1 ('threshold') type:0x1 ('temperature') full_name:'0(15.1).Temp 29'
 2789:20180710:110649.142 Added sensor: host:'10.0.0.54:623' id_type:0 id_sz:8 id:'Temp 28' reading_type:0x1 ('threshold') type:0x1 ('temperature') full_name:'0(16.8).Temp 28'
 2789:20180710:110649.142 Added sensor: host:'10.0.0.54:623' id_type:0 id_sz:8 id:'Temp 27' reading_type:0x1 ('threshold') type:0x1 ('temperature') full_name:'0(16.7).Temp 27'
 2789:20180710:110649.142 Added sensor: host:'10.0.0.54:623' id_type:0 id_sz:8 id:'Temp 26' reading_type:0x1 ('threshold') type:0x1 ('temperature') full_name:'0(66.11).Temp 26'
 2789:20180710:110649.143 Added sensor: host:'10.0.0.54:623' id_type:0 id_sz:8 id:'Temp 25' reading_type:0x1 ('threshold') type:0x1 ('temperature') full_name:'0(66.10).Temp 25'
 2789:20180710:110649.143 Added sensor: host:'10.0.0.54:623' id_type:0 id_sz:8 id:'Temp 24' reading_type:0x1 ('threshold') type:0x1 ('temperature') full_name:'0(66.9).Temp 24'
 2789:20180710:110649.144 Added sensor: host:'10.0.0.54:623' id_type:0 id_sz:8 id:'Temp 23' reading_type:0x1 ('threshold') type:0x1 ('temperature') full_name:'0(66.8).Temp 23'
 2789:20180710:110649.144 Added sensor: host:'10.0.0.54:623' id_type:0 id_sz:8 id:'Temp 22' reading_type:0x1 ('threshold') type:0x1 ('temperature') full_name:'0(66.7).Temp 22'
 2789:20180710:110649.145 Added sensor: host:'10.0.0.54:623' id_type:0 id_sz:8 id:'Temp 21' reading_type:0x1 ('threshold') type:0x1 ('temperature') full_name:'0(66.6).Temp 21'
 2789:20180710:110649.145 Added sensor: host:'10.0.0.54:623' id_type:0 id_sz:8 id:'Temp 20' reading_type:0x1 ('threshold') type:0x1 ('temperature') full_name:'0(66.5).Temp 20'
 2789:20180710:110649.146 Added sensor: host:'10.0.0.54:623' id_type:0 id_sz:8 id:'Temp 19' reading_type:0x1 ('threshold') type:0x1 ('temperature') full_name:'0(66.4).Temp 19'
 2789:20180710:110649.146 Added sensor: host:'10.0.0.54:623' id_type:0 id_sz:8 id:'Temp 18' reading_type:0x1 ('threshold') type:0x1 ('temperature') full_name:'0(16.6).Temp 18'
 2789:20180710:110649.147 Added sensor: host:'10.0.0.54:623' id_type:0 id_sz:8 id:'Temp 17' reading_type:0x1 ('threshold') type:0x1 ('temperature') full_name:'0(16.5).Temp 17'
 2789:20180710:110649.147 Added sensor: host:'10.0.0.54:623' id_type:0 id_sz:8 id:'Temp 16' reading_type:0x1 ('threshold') type:0x1 ('temperature') full_name:'0(16.4).Temp 16'
 2789:20180710:110649.148 Added sensor: host:'10.0.0.54:623' id_type:0 id_sz:8 id:'Temp 15' reading_type:0x1 ('threshold') type:0x1 ('temperature') full_name:'0(16.3).Temp 15'
 2789:20180710:110649.148 Added sensor: host:'10.0.0.54:623' id_type:0 id_sz:8 id:'Temp 14' reading_type:0x1 ('threshold') type:0x1 ('temperature') full_name:'0(16.2).Temp 14'
 2789:20180710:110649.148 Added sensor: host:'10.0.0.54:623' id_type:0 id_sz:8 id:'Temp 13' reading_type:0x1 ('threshold') type:0x1 ('temperature') full_name:'0(16.1).Temp 13'
 2789:20180710:110649.149 Added sensor: host:'10.0.0.54:623' id_type:0 id_sz:8 id:'Temp 12' reading_type:0x1 ('threshold') type:0x1 ('temperature') full_name:'0(66.3).Temp 12'
 2789:20180710:110649.150 Added sensor: host:'10.0.0.54:623' id_type:0 id_sz:8 id:'Temp 11' reading_type:0x1 ('threshold') type:0x1 ('temperature') full_name:'0(66.2).Temp 11'
 2789:20180710:110649.150 Added sensor: host:'10.0.0.54:623' id_type:0 id_sz:8 id:'Temp 10' reading_type:0x1 ('threshold') type:0x1 ('temperature') full_name:'0(66.1).Temp 10'
 2789:20180710:110649.150 Added sensor: host:'10.0.0.54:623' id_type:0 id_sz:7 id:'Temp 9' reading_type:0x1 ('threshold') type:0x1 ('temperature') full_name:'0(19.2).Temp 9'
 2789:20180710:110649.151 Added sensor: host:'10.0.0.54:623' id_type:0 id_sz:7 id:'Temp 8' reading_type:0x1 ('threshold') type:0x1 ('temperature') full_name:'0(19.1).Temp 8'
 2789:20180710:110649.151 Added sensor: host:'10.0.0.54:623' id_type:0 id_sz:7 id:'Temp 7' reading_type:0x1 ('threshold') type:0x1 ('temperature') full_name:'0(8.4).Temp 7'
 2789:20180710:110649.152 Added sensor: host:'10.0.0.54:623' id_type:0 id_sz:7 id:'Temp 6' reading_type:0x1 ('threshold') type:0x1 ('temperature') full_name:'0(8.3).Temp 6'
 2789:20180710:110649.152 Added sensor: host:'10.0.0.54:623' id_type:0 id_sz:7 id:'Temp 5' reading_type:0x1 ('threshold') type:0x1 ('temperature') full_name:'0(8.2).Temp 5'
 2789:20180710:110649.153 Added sensor: host:'10.0.0.54:623' id_type:0 id_sz:7 id:'Temp 4' reading_type:0x1 ('threshold') type:0x1 ('temperature') full_name:'0(8.1).Temp 4'
 2789:20180710:110649.153 Added sensor: host:'10.0.0.54:623' id_type:0 id_sz:7 id:'Temp 3' reading_type:0x1 ('threshold') type:0x1 ('temperature') full_name:'0(65.2).Temp 3'
 2789:20180710:110649.154 Added sensor: host:'10.0.0.54:623' id_type:0 id_sz:7 id:'Temp 2' reading_type:0x1 ('threshold') type:0x1 ('temperature') full_name:'0(65.1).Temp 2'
 2789:20180710:110649.154 Added sensor: host:'10.0.0.54:623' id_type:0 id_sz:7 id:'Temp 1' reading_type:0x1 ('threshold') type:0x1 ('temperature') full_name:'0(64.1).Temp 1'
 2789:20180710:110649.155 Added sensor: host:'10.0.0.54:623' id_type:0 id_sz:5 id:'Fans' reading_type:0xb ('discrete_redundancy') type:0x4 ('fan') full_name:'0(7.7).Fans'
 2789:20180710:110649.155 Added sensor: host:'10.0.0.54:623' id_type:0 id_sz:6 id:'Fan 6' reading_type:0xa ('discrete_availability') type:0x4 ('fan') full_name:'0(7.6).Fan 6'
 2789:20180710:110649.155 Added sensor: host:'10.0.0.54:623' id_type:0 id_sz:6 id:'Fan 5' reading_type:0xa ('discrete_availability') type:0x4 ('fan') full_name:'0(7.5).Fan 5'
 2789:20180710:110649.156 Added sensor: host:'10.0.0.54:623' id_type:0 id_sz:6 id:'Fan 4' reading_type:0xa ('discrete_availability') type:0x4 ('fan') full_name:'0(7.4).Fan 4'
 2789:20180710:110649.156 Added sensor: host:'10.0.0.54:623' id_type:0 id_sz:6 id:'Fan 3' reading_type:0xa ('discrete_availability') type:0x4 ('fan') full_name:'0(7.3).Fan 3'
 2789:20180710:110649.156 Added sensor: host:'10.0.0.54:623' id_type:0 id_sz:6 id:'Fan 2' reading_type:0xa ('discrete_availability') type:0x4 ('fan') full_name:'0(7.2).Fan 2'
 2789:20180710:110649.157 Added sensor: host:'10.0.0.54:623' id_type:0 id_sz:6 id:'Fan 1' reading_type:0xa ('discrete_availability') type:0x4 ('fan') full_name:'0(7.1).Fan 1'
 2789:20180710:110649.157 Added sensor: host:'10.0.0.54:623' id_type:0 id_sz:15 id:'Power Supplies' reading_type:0xb ('discrete_redundancy') type:0x8 ('power_supply') full_name:'0(10.3).Power Supplies'
 2789:20180710:110649.157 Added sensor: host:'10.0.0.54:623' id_type:0 id_sz:15 id:'Power Supply 2' reading_type:0x6f ('sensor specific') type:0x8 ('power_supply') full_name:'0(10.2).Power Supply 2'
 2789:20180710:110649.158 Added sensor: host:'10.0.0.54:623' id_type:0 id_sz:15 id:'Power Supply 1' reading_type:0x6f ('sensor specific') type:0x8 ('power_supply') full_name:'0(10.1).Power Supply 1'
 2789:20180710:110649.158 Added sensor: host:'10.0.0.54:623' id_type:0 id_sz:16 id:'Sys. Health LED' reading_type:0x71 ('invalid') type:0xc0 ('invalid') full_name:'0(23.2).Sys. Health LED'
 2789:20180710:110649.159 Added sensor: host:'10.0.0.54:623' id_type:0 id_sz:10 id:'UID Light' reading_type:0x70 ('invalid') type:0xc0 ('invalid') full_name:'0(23.1).UID Light'

"tail -f /tmp/zabbix_server.log|grep "State""


2789:20180710:110823.884 State [Power Supplies | power_supply | power_supply | discrete_redundancy | state 0 value is 0]
 2789:20180710:110823.884 State [Power Supplies | power_supply | power_supply | discrete_redundancy | state 1 value is 0]
 2789:20180710:110823.884 State [Power Supplies | power_supply | power_supply | discrete_redundancy | state 3 value is 1]
 2789:20180710:110823.884 State [Power Supplies | power_supply | power_supply | discrete_redundancy | state 4 value is 0]
 2789:20180710:110823.884 State [Power Supplies | power_supply | power_supply | discrete_redundancy | state 5 value is 0]
 2789:20180710:110824.899 State [Power Supply 1 | power_supply | power_supply | sensor specific | state 0 value is 1]
 2789:20180710:110824.899 State [Power Supply 1 | power_supply | power_supply | sensor specific | state 1 value is 0]
 2789:20180710:110824.911 State [Fan 1 | system_board | fan | discrete_availability | state 0 value is 1]
 2789:20180710:110824.911 State [Fan 1 | system_board | fan | discrete_availability | state 3 value is 0]
 2789:20180710:110824.911 State [Fan 1 | system_board | fan | discrete_availability | state 4 value is 0]
 2789:20180710:110824.911 State [Fan 1 | system_board | fan | discrete_availability | state 8 value is 0]
 2789:20180710:110825.923 State [Fan 2 | system_board | fan | discrete_availability | state 0 value is 1]
 2789:20180710:110825.923 State [Fan 2 | system_board | fan | discrete_availability | state 3 value is 0]
 2789:20180710:110825.924 State [Fan 2 | system_board | fan | discrete_availability | state 4 value is 0]
 2789:20180710:110825.924 State [Fan 2 | system_board | fan | discrete_availability | state 8 value is 0]
 2789:20180710:110826.936 State [Fan 3 | system_board | fan | discrete_availability | state 0 value is 1]
 2789:20180710:110826.936 State [Fan 3 | system_board | fan | discrete_availability | state 3 value is 0]
 2789:20180710:110826.936 State [Fan 3 | system_board | fan | discrete_availability | state 4 value is 0]
 2789:20180710:110826.936 State [Fan 3 | system_board | fan | discrete_availability | state 8 value is 0]
 2789:20180710:110826.948 State [Sys. Health LED | system_chassis | invalid | invalid | state 0 value is 0]
 2789:20180710:110826.948 State [Sys. Health LED | system_chassis | invalid | invalid | state 1 value is 0]
 2789:20180710:110826.948 State [Sys. Health LED | system_chassis | invalid | invalid | state 2 value is 0]
 2789:20180710:110827.973 State [Fan 4 | system_board | fan | discrete_availability | state 0 value is 1]
 2789:20180710:110827.973 State [Fan 4 | system_board | fan | discrete_availability | state 3 value is 0]
 2789:20180710:110827.973 State [Fan 4 | system_board | fan | discrete_availability | state 4 value is 0]
 2789:20180710:110827.973 State [Fan 4 | system_board | fan | discrete_availability | state 8 value is 0]

 



 Comments   
Comment by Alexey Pustovalov [ 2018 Jul 10 ]

Please provide screenshot with item configuration details.

Comment by ftdream [ 2018 Jul 10 ]
Type of information is float ,the result is the same.
Comment by ftdream [ 2018 Jul 10 ]
Only temperature monitoring is normal. The official explanation is that temperature is a threshold sensor and the others are discrete sensors.But I don't know exactly what to do to get the data right.
Comment by ftdream [ 2018 Jul 10 ]
A screenshot of the item configuration information has been submitted,eg:ipmi.fan1.png
Comment by Aigars Kadikis [ 2019 Jan 25 ]

Closing due to an unsupported version.





[ZBX-13470] IPMI: allow searching sensors by full name as well as id Created: 2018 Feb 14  Updated: 2018 Mar 27  Resolved: 2018 Mar 25

Status: Closed
Project: ZABBIX BUGS AND ISSUES
Component/s: Server (S)
Affects Version/s: 3.4.6
Fix Version/s: 4.0.0alpha5, 4.0 (plan)

Type: Problem report Priority: Major
Reporter: Alexey Asemov Assignee: Michael Veksler
Resolution: Fixed Votes: 0
Labels: IPMI, patch
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

Linux, OpenIPMI


Attachments: Text File zabbix-ipmi-fullname.patch    
Team: Team C
Sprint: Sprint 28, Sprint 29, Sprint 30
Story Points: 1

 Description   

Some BMC cards provide multiple sensors belonging to different entities but with the same ID. Only full name differs for these sensors. A good example is Dell iDRAC 6 card. Zabbix sensor debug output looks like the following:

7395:20180213:131244.993 Added sensor: host:'10.0.4.74:623' id_type:0 id_sz:5 id:'Temp' reading_type:0x1 ('threshold') type:0x1 ('temperature') full_name:'1(10.1).Temp'
7395:20180213:131244.994 Added sensor: host:'10.0.4.74:623' id_type:0 id_sz:5 id:'Temp' reading_type:0x1 ('threshold') type:0x1 ('temperature') full_name:'1(3.2).Temp'
7395:20180213:131244.995 Added sensor: host:'10.0.4.74:623' id_type:0 id_sz:5 id:'Temp' reading_type:0x1 ('threshold') type:0x1 ('temperature') full_name:'1(3.1).Temp'

Here you can see three 'Temp' readings which cannot be distinguished one from other easily.

Proposed and included patch stores sensors full name in the sensors list and adds a search function that finds sensor by full name, allowing to use full_name field values as IPMI sensor IDs.



 Comments   
Comment by Alexey Asemov [ 2018 Feb 14 ]

Worth mentioning that OpenIPMI library prefixes full name with ever changing domain name. This patch does remove one so full names are consistent.





[ZBX-13468] IPMI: solve RMCP+ connecton issues for iDRAC/others by using BMC selected authentication strategy Created: 2018 Feb 13  Updated: 2018 Jun 08  Resolved: 2018 Jun 08

Status: Closed
Project: ZABBIX BUGS AND ISSUES
Component/s: Server (S)
Affects Version/s: 3.4.6
Fix Version/s: None

Type: Patch request Priority: Major
Reporter: Alexey Asemov Assignee: Unassigned
Resolution: Won't fix Votes: 0
Labels: IPMI, RMCP+, iDRAC, patch
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

Linux, OpenIPMI


Attachments: Text File zabbix-ipmi-lanp.patch    
Sprint: Sprint 29, Sprint 30, Sprint 31, Sprint 32, Sprint 33, Sprint 34, Sprint 35

 Description   

Currently Zabbix has issues with BMCs supporting only RMCP+ but a limited list of RMCP+ authentication/integrity/confidentiality schemes not matching OpenIPMI defaults. It manifests as 'Invalid argument (22)' error with RMCP+, error 16777441 when not using RMCP+, and so on.

A good example is Dell iDRAC 6 module. It cannot be queried by Zabbix IPMI checks because of that.

I am attaching a patch that enforces all RMCP+ channel authentication/integrity/confidentiality parameters to BMC specified.

This patch is better extended by Zabbix team to add real options for RMCP+ authentication/integrity/confidentiality modes to the web interface. This is just 'proof of concept' patch making BMC rule the communication options that indeed works when using RMCP+ at least on iDRAC 6.



 Comments   
Comment by Alexey Asemov [ 2018 Jun 08 ]

So I assume the only way for now is using external scripts for iDRAC.





[ZBX-12563] Outdated information regarding unreachable pollers and IPMI checks Created: 2017 Aug 22  Updated: 2018 Oct 09  Resolved: 2017 Oct 10

Status: Closed
Project: ZABBIX BUGS AND ISSUES
Component/s: Documentation (D)
Affects Version/s: 3.4.0
Fix Version/s: 4.0 (plan)

Type: Documentation task Priority: Trivial
Reporter: Glebs Ivanovskis (Inactive) Assignee: Martins Valkovskis
Resolution: Fixed Votes: 0
Labels: ipmi, unreachable
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Team: Team C
Sprint: Sprint 15, Sprint 16, Sprint 17, Sprint 18
Story Points: 0.25

 Description   

As far as I know, after ZBXNEXT-3386 all IPMI checks are handled by IPMI manager process and IPMI workers, there is no unreachable poller for IPMI items. This piece of documentation still says:

From that moment UnreachableDelay defines how often a host is rechecked using one of the items (including LLD rules) in this unreachability situation and such rechecks will be performed already by unreachable pollers.



 Comments   
Comment by Martins Valkovskis [ 2017 Oct 04 ]

Thanks for reporting, fixed in 3.4. RESOLVED

glebs.ivanovskis Thank you! Looks good to me.
CLOSED





[ZBX-12493] Move depends on uninitialised value when using IPMI Created: 2017 Aug 09  Updated: 2017 Sep 07  Resolved: 2017 Sep 07

Status: Closed
Project: ZABBIX BUGS AND ISSUES
Component/s: Proxy (P), Server (S)
Affects Version/s: 2.2.20rc1, 3.0.11rc1, 3.2.8rc1, 3.4.0rc1
Fix Version/s: 2.2.20rc1, 3.0.11rc1, 3.2.8rc1, 3.4.2rc1, 4.0.0alpha1

Type: Problem report Priority: Trivial
Reporter: Vladislavs Sokurenko Assignee: Unassigned
Resolution: Fixed Votes: 0
Labels: ipmi
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: File ZBX-12483.diff    
Team: Team A
Sprint: Sprint 16
Story Points: 1.5

 Description   

Valgrind produces following error:

==10886== Conditional jump or move depends on uninitialised value(s)
==10886==    at 0x5481205: ipmi_lanp_setup_con (in /usr/lib64/libOpenIPMI.so.0.0.5)
==10886==    by 0x54820C6: ipmi_ip_setup_con (in /usr/lib64/libOpenIPMI.so.0.0.5)
==10886==    by 0x42B4BB: zbx_init_ipmi_host (checks_ipmi.c:1236)
==10886==    by 0x42CD22: get_value_ipmi (checks_ipmi.c:1420)
==10886==    by 0x42945B: ipmi_poller_process_value_request (ipmi_poller.c:111)
==10886==    by 0x42945B: ipmi_poller_thread (ipmi_poller.c:242)
==10886==    by 0x474459: zbx_thread_start (threads.c:128)
==10886==    by 0x418DE9: MAIN_ZABBIX_ENTRY (proxy.c:1097)
==10886==    by 0x469352: daemon_start (daemon.c:392)
==10886==    by 0x417A56: main (proxy.c:847)
==10886==  Uninitialised value was created by a stack allocation
==10886==    at 0x42B260: zbx_init_ipmi_host (checks_ipmi.c:1211

It appears that OpenIPMI sets port without respect of port count variable, it only checks that port is not NULL:

5583     for (i=0; i<MAX_IP_ADDR; i++) {
5584         if (!ports[i])
5585             ports[i] = IPMI_LAN_STD_PORT_STR;
5586     }

Please also see patch attached, though OpenIPMI code must be carefully analyzed before changes are made.



 Comments   
Comment by Andris Mednis [ 2017 Sep 04 ]

Confirmed with OpenIPMI-2.0.22.

Comment by Andris Mednis [ 2017 Sep 05 ]

Fixed in development branches:

  • svn://svn.zabbix.com/branches/dev/ZBX-12493-22
  • svn://svn.zabbix.com/branches/dev/ZBX-12493-30
Comment by Vladislavs Sokurenko [ 2017 Sep 07 ]

Successfully tested

Comment by Andris Mednis [ 2017 Sep 07 ]

Thanks! See improved comment in svn://svn.zabbix.com/branches/dev/ZBX-12493-22, r72320.

vso looks good.

Comment by Andris Mednis [ 2017 Sep 07 ]

Available in versions:

  • pre-2.2.20rc1 r72324
  • pre-3.0.11rc1 r72325
  • pre-3.2.8rc1 r72326
  • pre-3.4.2rc1 r72328
  • pre-4.0.0alpha1 r72330
Comment by Andris Mednis [ 2017 Sep 07 ]

No documentation changes required.





[ZBX-12360] IPMI Agent get wrong value Created: 2017 Jul 07  Updated: 2020 Feb 18

Status: Reopened
Project: ZABBIX BUGS AND ISSUES
Component/s: Server (S)
Affects Version/s: 3.2.6
Fix Version/s: None

Type: Problem report Priority: Trivial
Reporter: BM Assignee: Unassigned
Resolution: Unresolved Votes: 0
Labels: ipmi
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: PNG File ZBX_fan_Template_Conf.PNG     PNG File ZBX_fan_value_wrong.PNG    
Issue Links:
Sub-task
part of ZBX-8395 Zabbix display wrong values with IPMI... Closed
Sprint: Sprint 12, Sprint 13, Sprint 14, Sprint 15, Sprint 16, Sprint 17, Sprint 18
Story Points: 4

 Description   

Hello,

I've got an issue where Zabbix IPMI agent get wrong values.
Joined screenshot of a faulty value for fan and the item configuration.

With openipmi commands I've got the rights values

sofdevzabbix01:/home/bob# ipmitool -I lanplus -H 172.18.131.80 -U -P sdr
UID Light | 0x00 | ok
Sys. Health LED | no reading | ns
01-Inlet Ambient | 20 degrees C | ok
02-CPU 1 | 40 degrees C | ok
03-CPU 2 | 40 degrees C | ok
04-P1 DIMM 1-3 | disabled | ns
05-P1 DIMM 4-6 | disabled | ns
06-P1 DIMM 7-9 | 33 degrees C | ok
07-P1 DIMM 10-12 | 34 degrees C | ok
08-P2 DIMM 1-3 | disabled | ns
09-P2 DIMM 4-6 | disabled | ns
10-P2 DIMM 7-9 | 32 degrees C | ok
11-P2 DIMM 10-12 | 29 degrees C | ok
12-HD Max | 35 degrees C | ok
13-Chipset | 50 degrees C | ok
14-P/S 1 | 34 degrees C | ok
15-P/S 2 | 31 degrees C | ok
16-P/S 2 Zone | 36 degrees C | ok
17-VR P1 | 39 degrees C | ok
18-VR P2 | 39 degrees C | ok
19-VR P1 Mem | 37 degrees C | ok
20-VR P1 Mem | 36 degrees C | ok
21-VR P2 Mem | 35 degrees C | ok
22-VR P2 Mem | 37 degrees C | ok
23-VR P1Vtt Zone | 37 degrees C | ok
24-VR P2Vtt Zone | 35 degrees C | ok
25-HD Controller | 85 degrees C | ok
26-iLO Zone | 38 degrees C | ok
27-LOM Card | 64 degrees C | ok
28-PCI 1 | disabled | ns
29-PCI 2 | disabled | ns
30-PCI 3 | 78 degrees C | ok
31-PCI 4 | disabled | ns
32-PCI 5 | disabled | ns
33-PCI 6 | disabled | ns
34-PCI 1 Zone | 34 degrees C | ok
35-PCI 2 Zone | 37 degrees C | ok
36-PCI 3 Zone | 38 degrees C | ok
37-PCI 4 Zone | disabled | ns
38-PCI 5 Zone | disabled | ns
39-PCI 6 Zone | disabled | ns
40-I/O Board 1 | 39 degrees C | ok
41-I/O Board 2 | disabled | ns
42-VR P1 Zone | 29 degrees C | ok
43-BIOS Zone | 52 degrees C | ok
44-System Board | 38 degrees C | ok
45-SuperCap Max | 26 degrees C | ok
46-Chipset Zone | 39 degrees C | ok
47-Battery Zone | 38 degrees C | ok
48-I/O Zone | 44 degrees C | ok
49-Sys Exhaust | 40 degrees C | ok
50-Sys Exhaust | 37 degrees C | ok
Fan 1 | 6.27 percent | ok
Fan 2 | 6.27 percent | ok
Fan 3 | 6.27 percent | ok
Fan 4 | 7.84 percent | ok
Fan 5 | 14.90 percent | ok
Fan 6 | 18.82 percent | ok
Power Supply 1 | 140 Watts | ok
Power Supply 2 | 35 Watts | ok
Power Meter | 154 Watts | ok
Power Supplies | 0x00 | ok
Fans | 0x00 | ok
Memory | 0x00 | ok
C1 P1I Bay 1 | 0x01 | ok
C1 P1I Bay 2 | 0x01 | ok
C1 P1I Bay 3 | 0x01 | ok
C1 P1I Bay 4 | 0x01 | ok
C1 P2I Bay 5 | 0x01 | ok
C1 P2I Bay 6 | 0x01 | ok
C1 P2I Bay 7 | 0x01 | ok
C1 P2I Bay 8 | 0x01 | ok

Only temperature sensors get good values.



 Comments   
Comment by Andrea Biscuola (Inactive) [ 2017 Jul 17 ]

Can't reproduce this. On our devices that have IPMI, the fan speed is reported in RPM.

BM Can you try to remove from the fan item configuration the unit and see what are the raw values without any potential conversion?

Comment by BM [ 2017 Jul 17 ]

Unit have been removed from the template for Fan, The template have been "unlinked and clear" then "link" again to the host.

The raw values are the same, I've got "1" for fan speed.

I want to precise that the values are from an ILO DL380p Gen8
And that the values problem are also for the power meter of the ILO. As you already see with "ipmitool" the values are good.

Comment by Andrea Biscuola (Inactive) [ 2017 Jul 17 ]

BM Can you also post the output of the "sensor" command instead of "sdr"?

Comment by BM [ 2017 Jul 17 ]

Yes here it is

bob@sofdevzabbix01:~$ ipmitool -I lanplus -H 172.18.131.80 -U -P usensor
UID Light | 0x0 | discrete | 0x0180| na | na | na | na | na | na
Sys. Health LED | na | discrete | na | na | na | na | na | na | na
01-Inlet Ambient | 21.000 | degrees C | ok | na | na | na | na | 42.000 | 46.000
02-CPU 1 | 40.000 | degrees C | ok | na | na | na | na | 70.000 | na
03-CPU 2 | 40.000 | degrees C | ok | na | na | na | na | 70.000 | na
04-P1 DIMM 1-3 | na | | na | na | na | na | na | 87.000 | na
05-P1 DIMM 4-6 | na | | na | na | na | na | na | 87.000 | na
06-P1 DIMM 7-9 | 33.000 | degrees C | ok | na | na | na | na | 87.000 | na
07-P1 DIMM 10-12 | 34.000 | degrees C | ok | na | na | na | na | 87.000 | na
08-P2 DIMM 1-3 | na | | na | na | na | na | na | 87.000 | na
09-P2 DIMM 4-6 | na | | na | na | na | na | na | 87.000 | na
10-P2 DIMM 7-9 | 32.000 | degrees C | ok | na | na | na | na | 87.000 | na
11-P2 DIMM 10-12 | 29.000 | degrees C | ok | na | na | na | na | 87.000 | na
12-HD Max | 35.000 | degrees C | ok | na | na | na | na | 60.000 | na
13-Chipset | 50.000 | degrees C | ok | na | na | na | na | 105.000 | na
14-P/S 1 | 34.000 | degrees C | ok | na | na | na | na | na | na
15-P/S 2 | 31.000 | degrees C | ok | na | na | na | na | na | na
16-P/S 2 Zone | 36.000 | degrees C | ok | na | na | na | na | 75.000 | 80.000
17-VR P1 | 39.000 | degrees C | ok | na | na | na | na | 115.000 | 120.000
18-VR P2 | 39.000 | degrees C | ok | na | na | na | na | 115.000 | 120.000
19-VR P1 Mem | 37.000 | degrees C | ok | na | na | na | na | 115.000 | 120.000
20-VR P1 Mem | 36.000 | degrees C | ok | na | na | na | na | 115.000 | 120.000
21-VR P2 Mem | 35.000 | degrees C | ok | na | na | na | na | 115.000 | 120.000
22-VR P2 Mem | 37.000 | degrees C | ok | na | na | na | na | 115.000 | 120.000
23-VR P1Vtt Zone | 37.000 | degrees C | ok | na | na | na | na | 90.000 | 95.000
24-VR P2Vtt Zone | 35.000 | degrees C | ok | na | na | na | na | 90.000 | 95.000
25-HD Controller | 85.000 | degrees C | ok | na | na | na | na | 100.000 | na
26-iLO Zone | 38.000 | degrees C | ok | na | na | na | na | 90.000 | 95.000
27-LOM Card | 65.000 | degrees C | ok | na | na | na | na | 100.000 | na
28-PCI 1 | na | | na | na | na | na | na | 100.000 | na
29-PCI 2 | na | | na | na | na | na | na | 100.000 | na
30-PCI 3 | 79.000 | degrees C | ok | na | na | na | na | 100.000 | na
31-PCI 4 | na | | na | na | na | na | na | 100.000 | na
32-PCI 5 | na | | na | na | na | na | na | 100.000 | na
33-PCI 6 | na | | na | na | na | na | na | 100.000 | na
34-PCI 1 Zone | 34.000 | degrees C | ok | na | na | na | na | 65.000 | 70.000
35-PCI 2 Zone | 37.000 | degrees C | ok | na | na | na | na | 66.000 | 71.000
36-PCI 3 Zone | 38.000 | degrees C | ok | na | na | na | na | 66.000 | 71.000
37-PCI 4 Zone | na | | na | na | na | na | na | 65.000 | 70.000
38-PCI 5 Zone | na | | na | na | na | na | na | 65.000 | 70.000
39-PCI 6 Zone | na | | na | na | na | na | na | 65.000 | 70.000
40-I/O Board 1 | 40.000 | degrees C | ok | na | na | na | na | 66.000 | 71.000
41-I/O Board 2 | na | | na | na | na | na | na | 66.000 | 71.000
42-VR P1 Zone | 29.000 | degrees C | ok | na | na | na | na | 95.000 | 100.000
43-BIOS Zone | 52.000 | degrees C | ok | na | na | na | na | 90.000 | 95.000
44-System Board | 38.000 | degrees C | ok | na | na | na | na | 80.000 | 85.000
45-SuperCap Max | 26.000 | degrees C | ok | na | na | na | na | 65.000 | na
46-Chipset Zone | 40.000 | degrees C | ok | na | na | na | na | 75.000 | 80.000
47-Battery Zone | 38.000 | degrees C | ok | na | na | na | na | 75.000 | 80.000
48-I/O Zone | 45.000 | degrees C | ok | na | na | na | na | 75.000 | 80.000
49-Sys Exhaust | 40.000 | degrees C | ok | na | na | na | na | 75.000 | 80.000
50-Sys Exhaust | 37.000 | degrees C | ok | na | na | na | na | 75.000 | 80.000
Fan 1 | 6.272 | percent | ok | na | na | na | na | na | na
Fan 2 | 6.272 | percent | ok | na | na | na | na | na | na
Fan 3 | 6.272 | percent | ok | na | na | na | na | na | na
Fan 4 | 7.840 | percent | ok | na | na | na | na | na | na
Fan 5 | 14.504 | percent | ok | na | na | na | na | na | na
Fan 6 | 18.424 | percent | ok | na | na | na | na | na | na
Power Supply 1 | 140 | Watts | ok | na | na | na | na | na | na
Power Supply 2 | 35 | Watts | ok | na | na | na | na | na | na
Power Meter | 154 | Watts | ok | na | na | na | na | na | na
Power Supplies | 0x0 | discrete | 0x0180| na | na | na | na | na | na
Fans | 0x0 | discrete | 0x0180| na | na | na | na | na | na
Memory | 0x0 | discrete | 0x4080| na | na | na | na | na | na
C1 P1I Bay 1 | 0x1 | discrete | 0x0180| na | na | na | na | na | na
C1 P1I Bay 2 | 0x1 | discrete | 0x0180| na | na | na | na | na | na
C1 P1I Bay 3 | 0x1 | discrete | 0x0180| na | na | na | na | na | na
C1 P1I Bay 4 | 0x1 | discrete | 0x0180| na | na | na | na | na | na
C1 P2I Bay 5 | 0x1 | discrete | 0x0180| na | na | na | na | na | na
C1 P2I Bay 6 | 0x1 | discrete | 0x0180| na | na | na | na | na | na
C1 P2I Bay 7 | 0x1 | discrete | 0x0180| na | na | na | na | na | na
C1 P2I Bay 8 | 0x1 | discrete | 0x0180| na | na | na | na | na | na

Comment by Andrea Biscuola (Inactive) [ 2017 Jul 17 ]

BM I have a theory about how the bug is originated. But I need from you to run the server with increased logging level (Change the DebugLevel parameter to 4 in the zabbix_server.conf file).
Run the server until some IPMI checks are performed and, if you can, post the resulting log files here. Maybe We will be lucky to not need special hardware for debugging this

Thanks

Comment by BM [ 2017 Jul 18 ]

In link you have the log with a level 4 https://drive.google.com/open?id=0BxYYuq2hl7h9TWRzdVdiejY4R2c
I wasn't able to use the attachment, the file is 116Mo

Comment by Andrea Biscuola (Inactive) [ 2017 Jul 20 ]

So, After a lot of thinking I understood (I think) why BM receive wrong values. It took me a while and also a lot of reading of the IPMI version 2.0 specification. The Power Supply and Fan sensors in BM hardware are 'discrete' (see section 42.1 of the IPMI 2.0 specification).
Those discrete reads report one or more states of the sensor and along those states, also the currently measured value. In our code, that check is performed in src/zabbix_server/ipmi/checks_ipmi.c in the zbx_read_ipmi_sensor() function. Inside the function you can find this code snippet:

	switch (s->reading_type)
	{
		case IPMI_EVENT_READING_TYPE_THRESHOLD:
			if (0 != (ret = ipmi_sensor_get_reading(s->sensor, zbx_got_thresh_reading_cb, h)))
			{
				/* do not use pointer to sensor here - the sensor may have disappeared during */
				/* ipmi_sensor_get_reading(), as domain might be closed due to communication failure */
				h->err = zbx_dsprintf(h->err, "Cannot read sensor \"%s\"."
						" ipmi_sensor_get_reading() return error: 0x%x", id_str, ret);
				h->ret = NOTSUPPORTED;
				goto out;
			}
			break;
		case IPMI_EVENT_READING_TYPE_DISCRETE_USAGE:
		case IPMI_EVENT_READING_TYPE_DISCRETE_STATE:
		case IPMI_EVENT_READING_TYPE_DISCRETE_PREDICTIVE_FAILURE:
		case IPMI_EVENT_READING_TYPE_DISCRETE_LIMIT_EXCEEDED:
		case IPMI_EVENT_READING_TYPE_DISCRETE_PERFORMANCE_MET:
		case IPMI_EVENT_READING_TYPE_DISCRETE_SEVERITY:
		case IPMI_EVENT_READING_TYPE_DISCRETE_DEVICE_PRESENCE:
		case IPMI_EVENT_READING_TYPE_DISCRETE_DEVICE_ENABLE:
		case IPMI_EVENT_READING_TYPE_DISCRETE_AVAILABILITY:
		case IPMI_EVENT_READING_TYPE_DISCRETE_REDUNDANCY:
		case IPMI_EVENT_READING_TYPE_DISCRETE_ACPI_POWER:
		case IPMI_EVENT_READING_TYPE_SENSOR_SPECIFIC:
		case 0x70:	/* reading types 70h-7Fh are for OEM discrete sensors */
		case 0x71:
		case 0x72:
		case 0x73:
		case 0x74:
		case 0x75:
		case 0x76:
		case 0x77:
		case 0x78:
		case 0x79:
		case 0x7a:
		case 0x7b:
		case 0x7c:
		case 0x7d:
		case 0x7e:
		case 0x7f:
			if (0 != (ret = ipmi_sensor_get_states(s->sensor, zbx_got_discrete_states_cb, h)))
			{
				/* do not use pointer to sensor here - the sensor may have disappeared during */
				/* ipmi_sensor_get_states(), as domain might be closed due to communication failure */
				h->err = zbx_dsprintf(h->err, "Cannot read sensor \"%s\"."
						" ipmi_sensor_get_states() return error: 0x%x", id_str, ret);
				h->ret = NOTSUPPORTED;
				goto out;
			}
			break;
		default:
			s_reading_type_string = ipmi_sensor_get_event_reading_type_string(s->sensor);

			h->err = zbx_dsprintf(h->err, "Cannot read sensor \"%s\"."
					" IPMI reading type \"%s\" is not supported", id_str, s_reading_type_string);
			h->ret = NOTSUPPORTED;
			goto out;
	}

This switch statement process the various type of 'readings' that a sensor expose. In the case of BM we can see from the log provided that, while the temperature sensors (all the ones with correct values) have a type of 0x1, so purely "threshold-based", the others are 'generic discrete' (for example the Fan sensors have type 0xa and it fall in the 'Generic discrete' category).
What we are doing, is reading the actual state of the sensor and reporting it as the value instead of reading the state and storing it in the right place AND reading the value and storing it in the sensor structure.
So this is a bug in zabbix and not a non-compliant equipment or faulty hardware.
What we should do is:

  • Read the sensor state and store it.
  • Check if the reading have also a data value in it
  • Read the data and store it in the sensor structure.

I hope it's clear where the problem lies





[ZBX-12334] IPMI Monitoring Returning Incorrect Result Created: 2017 Jul 03  Updated: 2017 Jul 23  Resolved: 2017 Jul 21

Status: Closed
Project: ZABBIX BUGS AND ISSUES
Component/s: Server (S)
Affects Version/s: 3.2.6
Fix Version/s: 3.2.6

Type: Incident report Priority: Trivial
Reporter: Leandro Moreira Assignee: Unassigned
Resolution: Won't fix Votes: 0
Labels: IPMI, Zabbixm
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

Debian 9, Zabbix 3.2.6 and IPMI 2.0.22


Attachments: PNG File zabbix_ipmi_interface.PNG     PNG File zabbix_ipmi_sensor.PNG     PNG File zabbix_ipmi_version.PNG    

 Description   

Good night dear!
I'm trying to monitor two HP servers (360 G5 and one ML 310eG2) by IPMI the zabbix server connects to the servers by IPMI without problems, but the readings are different than the ipmitool command returns in cosole (prints attached) I've already compiled the packages IPMI by enabling SSL, I upgraded my operating system (debian to version 9) in the expectation that these issues would be resolved by upgrading the ipmi package version (2.0.16 to 2.0.22) but the problems still persist.
An example of the "Power Meter" sensor that on the zabbix server console returns 218 Watts at the zabbix interface returns 2 Watts.
Can you help me?



 Comments   
Comment by Rostislav Palivoda [ 2017 Jul 21 ]

Please turn to IRC channel for help from community - http://zabbix.org/wiki/Getting_help

Comment by Leandro Moreira [ 2017 Jul 23 ]

Thank's for your help.

Att.





[ZBX-11697] zabbix server randomly crashes Created: 2017 Jan 11  Updated: 2017 May 30  Resolved: 2017 Jan 23

Status: Closed
Project: ZABBIX BUGS AND ISSUES
Component/s: Server (S)
Affects Version/s: 3.2.3
Fix Version/s: None

Type: Incident report Priority: Blocker
Reporter: Dmitry Burtsev Assignee: Unassigned
Resolution: Duplicate Votes: 0
Labels: crash, ipmi
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

Zabbix Appliance 3.2.3 MySQL version, on physical server
Xeon 5130 x2, 4 Gb RAM, SCSI RAID 300 Gb
~130 hosts, 3800 items, 1500 triggers


Attachments: Text File 2017-03-31-crash.log     File 2017-03-31-crash.log_analysis.odt     File OpenIPMI-2.0.22-Andris_debug.tgz     File checks_ipmi.c     File checks_ipmi_protection_added.c     Text File compile_modified_on_appliance.txt     File zabbix_server1-debug4_21553.txt.gz     File zabbix_server1-debug4_21553_en.odt     File zabbix_server2-debug4_2775.txt.gz    
Issue Links:
Sub-task
depends on ZBX-10940 *** buffer overflow detected ***: /us... Closed

 Description   

Hello,

I've got fresh install of Zabbix Appliance 3.2.3 with restored database from Zabbix version 3.0.7 (zabbix 3.0.7 worked fine with this database).

Zabbix server starts fine, works for some time (10-60 min) then randomly crashes with following messages in log:

*** buffer overflow detected ***: /usr/sbin/zabbix_server: unreachable poller #16 [got 0 values in 0.000004 sec, getting values] terminated
======= Backtrace: =========
/lib/x86_64-linux-gnu/libc.so.6(+0x7329f)[0x7f9c4803d29f]
/lib/x86_64-linux-gnu/libc.so.6(__fortify_fail+0x5c)[0x7f9c480d4bbc]
/lib/x86_64-linux-gnu/libc.so.6(+0x109a90)[0x7f9c480d3a90]

In zabbix_server.conf there is following setting:
StartPollersUnreachable=20

I tried to decrease it to 10 and even 3, but that didn't help.

I'm attached some log files (zabbix_server-debug4.zip is a log with Debug level 4).



 Comments   
Comment by Dmitry Burtsev [ 2017 Jan 11 ]

Forgot to say, after that crash I can start zabbix server manually and it works fine until eventually crash again:
sudo service zabbix-server start

Comment by Andris Mednis [ 2017 Jan 11 ]

Hi, Dmitry!
Thanks for sending good debug logs !
Seems like a duplicate of ZBX-10940, but with improved debug info in logs from ZBX-10983 fix.

Comment by Dmitry Burtsev [ 2017 Jan 11 ]

Hello Andris,

I played a bit more with different config settings and found out following info that may help:

With this setup my Zabbix server still crashes sooner or later (10-60 min):
StartIPMIPollers=1
StartPollersUnreachable=1

And with this setup my Zabbix server works fine (but it doesn't check IPMI hosts, of course):
StartIPMIPollers=0
StartPollersUnreachable=20

So looks like my problem is IPMI.

Comment by Andris Mednis [ 2017 Jan 12 ]

Hi, Dmitry!
Thanks! Your comment confirms what is known - the crash occurs when "unreachable poller" polls an IPMI host and communication fails ... and some yet unknown combination of factors also happen.
What is your OpenIPMI library version ?

Comment by Dmitry Burtsev [ 2017 Jan 13 ]

Hello Andris,

appliance@zabbix:~$ apt list --installed | grep ipmi
libopenipmi0/trusty-updates,now 2.0.18-0ubuntu7.2 amd64 [installed,automatic]

It's Zabbix Appliance, I added/changed nothing, only did "apt-get update" and "apt-get upgrade".

Comment by Andris Mednis [ 2017 Jan 18 ]

Relevant part of log files are extracted into attachments zabbix_server1-debug4_21553.txt.gz and zabbix_server2-debug4_2775.txt.gz.
Dmitry, you can remove full log attachments.

Comment by Dmitry Burtsev [ 2017 Jan 18 ]

Andris,
Ok, done. Thanks.

Comment by Andris Mednis [ 2017 Jan 23 ]

Closing as duplicate of ZBX-10940. Thanks to kerberos464 for debug logs !

Comment by Andris Mednis [ 2017 Feb 02 ]

So far no luck - cannot reproduce this crash in unreachable poller.
Testing with Zabbix Appliance 3.2.3 in virtual machine.

Comment by Dmitry Burtsev [ 2017 Feb 02 ]

I put 'service zabbix-server start' to cron job every 5 min, and forgot about issue, so I'm fine with that now.
Today I checked zabbix server logs and discovered that zabbix server crashes very rarely now (1-2 times per 24 hours), I don't know why.

If I can help with fixing issue, tell me how, please. Maybe debug logs level 5 may help?

Comment by Andris Mednis [ 2017 Feb 02 ]

Thanks for feedback, Dmitry!
DebugLevel=5 unlikely will help with IPMI.
Would it be possible for you to compile Zabbix server and OpenIPMI from sources ?
I can provide instructions how I do that and OpenIPMI 2.0.22 sources with my additional debug logging added (it goes into Zabbix log).

Comment by Andris Mednis [ 2017 Feb 03 ]

Attachment zabbix_server1-debug4_21553_en.odt is a commented analysis of what happened in attached log file zabbix_server1-debug4_21553.txt.gz.
In short - a callback function which belongs to Zabbix is called with a corrupted pointer from OpenIPMI library.
It is not clear is it a bug in Zabbix or in OpenIPMI library.
I could not reproduce the crash in test environment.

Comment by Dmitry Burtsev [ 2017 Feb 03 ]

Andris,
I know almost nothing about Linux (this why I choosed Appliance version of Zabbix), so I'm not sure if I can compile from source, but I can try.
Please provide instructions.

Comment by Andris Mednis [ 2017 Feb 03 ]

Thanks, Dmitry!
I will try to make description how to do it on appliance.

Comment by Andris Mednis [ 2017 Feb 06 ]

Hi, Dmitry!
Instructions how to compile OpenIPMI library and Zabbix server with additional debugging are in attachment "compile_modified_on_appliance.txt". Feel free to ask if something is not clear.

Comment by Dmitry Burtsev [ 2017 Feb 07 ]

Hello Andris,
Thank you for really great instruction. Just finished following it, so my new Zabbix server is working now. Cron task is disabled as well.
zabbix_server.conf config has following settings now:
StartIPMIPollers=20
StartPollersUnreachable=20
DebugLevel is commented, so it has default value 3

When (if) Zabbix server crash again, I will upload log file.
I checked previous logs, last crash occured on February 3rd, so we have to wait some time till next crash. Anyway I will report when it will happen.
Thanks.

Comment by Andris Mednis [ 2017 Feb 07 ]

Hi, Dmitry!
Great that you managed it
DebugLevel=3 ir also the right thing - but you increased it after start to DebugLevel=4 for 'unreachable pollers' with

 /home/appliance/zabbix_323/sbin/zabbix_server -c /etc/zabbix/zabbix_server.conf -R log_level_increase="unreachable poller"

right ?
Also as 'root' you can check that our modified OpenIPMI library is in use:

# lsof -p $(ps -fu zabbix | grep zabbix_server.conf | awk '{print $2}') | grep OpenIPMI
zabbix_se 1236 zabbix  mem    REG  252,0   140138 165704 /home/appliance/OpenIPMI-2.0.22-debug-install/lib/libOpenIPMIutils.so.0.0.1
zabbix_se 1236 zabbix  mem    REG  252,0    92041 165715 /home/appliance/OpenIPMI-2.0.22-debug-install/lib/libOpenIPMIposix.so.0.0.1
zabbix_se 1236 zabbix  mem    REG  252,0  3747361 165708 /home/appliance/OpenIPMI-2.0.22-debug-install/lib/libOpenIPMI.so.0.0.5
Comment by Dmitry Burtsev [ 2017 Feb 07 ]

Yes, I was running that command for increasing debug level.

OpenIPMI library:

root@zabbix:/home/appliance# lsof -p $(ps -fu zabbix | grep zabbix_server.conf | awk '{print $2}') | grep OpenIPMI
zabbix_se 15471 zabbix  mem    REG    252,0   140138  6947935 /home/appliance/OpenIPMI-2.0.22-debug-install/lib/libOpenIPMIutils.so.0.0.1
zabbix_se 15471 zabbix  mem    REG    252,0    92041  6947947 /home/appliance/OpenIPMI-2.0.22-debug-install/lib/libOpenIPMIposix.so.0.0.1
zabbix_se 15471 zabbix  mem    REG    252,0  3805854  6947941 /home/appliance/OpenIPMI-2.0.22-debug-install/lib/libOpenIPMI.so.0.0.5
Comment by Andris Mednis [ 2017 Feb 07 ]

Great!
Waiting for crash....

Comment by Oleksii Zagorskyi [ 2017 Feb 13 ]

Andris, prepare please patch for "checks_ipmi.c" but for version 3.0.7.
diff taken between the attached file and 3.2.3 - cannot be successfully applied on 3.0.7

Comment by Andris Mednis [ 2017 Feb 13 ]

Patch is not used here. You take a fresh 3.0.7 source code and replace src/zabbix_server/poller/checks_ipmi.c with the file checks_ipmi.c from ZBX-11697 attachment https://support.zabbix.com/secure/attachment/47767/checks_ipmi.c .
If possible, you follow instructions "compile_modified_on_appliance.txt" in attachment and compile a private installation of OpenIPMI with more debug logging added.

Comment by Dmitry Burtsev [ 2017 Mar 31 ]

Hello Andris,
Looks like zabbix server crashed finally. I'm attaching last 200 lines of log file.

Comment by Andris Mednis [ 2017 Apr 04 ]

Thanks, Dmitry!
Is it possible for you to compress and attach a longer log file, not just 200 lines ? Then I can extract relevant info and you can delete it soon after.
In this case (process 2110 crashed) you can use a command like

   grep '^  2110' 2017-03-31-crash.log | gzip > 2017-03-31-crash_2110.log.gz

to get only info about crashed process to upload.

Comment by Dmitry Burtsev [ 2017 Apr 04 ]

Hello Andris,
Sure. I'm attaching log file with last 10000 lines.

Comment by Dmitry Burtsev [ 2017 Apr 04 ]

Ok, I'm attaching log from process 2110 as well (250 mb when unpacked)
I repacked log file into 7z archive, because Support engine doesn't let me to upload file more than 15 mb.

Comment by Andris Mednis [ 2017 Apr 04 ]

Thanks! That was unbelievably fast
You can remove them now, I copied them.

Comment by Dmitry Burtsev [ 2017 Apr 04 ]

Ok, done. Hope it will help to fix issue. Thanks!

Comment by Andris Mednis [ 2017 Apr 07 ]

Uploaded some analysis of crash log - attachment 2017-03-31-crash.log_analysis.odt .

Comment by Andris Mednis [ 2017 Apr 13 ]

Hi, kerberos464!

As I still cannot find the root cause of crash, I added protection (see attachment
checks_ipmi_protection_added.c) - detect the invalid pointer and ignore it. Hopefully it will prevent crash, but I do not know as the crash is not reproducible here.
You can make a backup copy of your current "src/zabbix_server/poller/checks_ipmi.c" file and replace it with file "checks_ipmi_protection_added.c", like:

$ cp src/zabbix_server/poller/checks_ipmi.c src/zabbix_server/poller/checks_ipmi.c.20170413
$ cp /tmp/checks_ipmi_protection_added.c src/zabbix_server/poller/checks_ipmi.c

and recompile.
Be careful, this is experimental, it could not be tested in my environment. I do not know how OpenIPMI library behaves when invalid pointers start to appear and we ignore them (will there be a memory leak instead of crash ?)

Comment by Dmitry Burtsev [ 2017 Apr 14 ]

Hello Andris,
Ok, I recompiled Zabbix server with new file, now let's wait till next crash.
Thanks.

Comment by Andris Mednis [ 2017 Apr 18 ]

Thanks, Dmitry!





[ZBX-11406] Point out clearly what commands can be done via IPMI Created: 2016 Oct 29  Updated: 2017 May 30  Resolved: 2017 Feb 14

Status: Closed
Project: ZABBIX BUGS AND ISSUES
Component/s: Documentation (D)
Affects Version/s: 2.0.19, 2.2.15, 3.2.1
Fix Version/s: None

Type: Incident report Priority: Trivial
Reporter: Marc Assignee: Unassigned
Resolution: Fixed Votes: 0
Labels: actionoperations, globalscripts, ipmi, remotecommands
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Team: Team A
Sprint: Sprint 1

 Description   

It is not obvious from the documentation what IPMI commands are supported by Zabbix.
It seems that only "Basic controls" are supported:

<command> <value>

Where <value> is an integer resp. "on" or "off" and <command> is (theoretically) one of these:

  • relay
  • alarm
  • reset
  • power
  • fan speed
  • one-shot reset
  • output
  • one-shot output

Since blanks are not supported for commands, commands like "fan speed" can neither be used.



 Comments   
Comment by Martins Valkovskis [ 2016 Nov 10 ]

Documented in Remote commands for 3.2 (also in versions 2.0, 2.2, 2.4, 3.0, 3.4).

Linked to from action operations and global scripts.

Comment by Marc [ 2016 Nov 10 ]

Just wondering whether it might be worth to also refer to chapter "9.1.3 Basic Type Controls" of IPMI – A Gentle Introduction with OpenIPMI.

Anyway, it's clearer now. Feel free to close the issue

Comment by Alexander Vladishev [ 2016 Dec 15 ]

<value> is optional parameter. If it is missing the default value "on" will be used.

Second example can be corrected:

"reset on" => "reset"

martins-v Thanks, I've corrected the example.

Is value optional with all commands or just some? How about when integer is used as value?

sasha value is optional for all commands. Syntax of the remote command must be corrected:

<command> <value>    =>    <command> [<value>]

value for IPMI commands is always an integer value. "on" is equal 1, "off" - 0.

martins-v Updated in Remote commands.

sasha CLOSED





[ZBX-11384] Configuration error does not affect item/host status Created: 2016 Oct 24  Updated: 2018 Jan 31  Resolved: 2018 Jan 28

Status: Closed
Project: ZABBIX BUGS AND ISSUES
Component/s: Proxy (P), Server (S)
Affects Version/s: 2.2.16rc1, 3.0.6rc1, 3.2.2rc1, 3.4.0alpha1
Fix Version/s: 4.0 (plan)

Type: Incident report Priority: Minor
Reporter: Glebs Ivanovskis (Inactive) Assignee: Viktors Tjarve
Resolution: Won't fix Votes: 0
Labels: encryption, ipmi, odbc
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Team: Team A
Sprint: Sprint 25, Sprint 26
Story Points: 2

 Description   

Common source of configuration errors are missing (not compiled in) features in server/proxy binary. Therefore it does not make sense to continue attempts to poll certain type of items or certain hosts and litter log file with error messages.



 Comments   
Comment by Glebs Ivanovskis (Inactive) [ 2018 Jan 18 ]

Similarly there is no point checking Simple checks with keys not supported (neither by Zabbix, nor by loaded modules), Internal checks with invalid keys, Calculated items with broken formulas and stuff like this until configuration update.

Comment by Viktors Tjarve [ 2018 Jan 22 ]

Potentially this could be solved similar to how it was done with proxy misconfiguration from server side ZBX-13060.
Currently it is decided not to fix this. Complexity added to solve the issues described in this ticket is not worth what we would gain from it.

WON'T FIX





[ZBX-11169] Zabbix server 3.0 keeps crashing Created: 2016 Sep 07  Updated: 2017 May 30  Resolved: 2016 Dec 17

Status: Closed
Project: ZABBIX BUGS AND ISSUES
Component/s: Server (S)
Affects Version/s: 3.0.4
Fix Version/s: None

Type: Incident report Priority: Critical
Reporter: Mohannad Assignee: Unassigned
Resolution: Duplicate Votes: 0
Labels: crash, ipmi
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

Centralized environment depending on a single zabbix server as a master hosting a number of local and remote nodes using a combination of proxies and agents to poll the data.


Attachments: Text File zabbix-sever-logs(14-9-2016).txt     Text File zabbix-sever-logs(15-9-2016).txt     Text File zabbix-sever-logs.txt     Text File zabbix-sever-logs.txt    
Issue Links:
Duplicate
duplicates ZBX-10983 zabbix server 3.0 crash - ipmi problem Closed

 Description   

Zabbix server keeps crashing unexpectedly, When I restart the service it works for a couple of hours and then crashes.

The attached is zabbix server logs.

Thanks
Mohannad Swaid



 Comments   
Comment by richlv [ 2016 Sep 07 ]

backtrace :

  9333:20160907:024441.784 === Backtrace: ===
  9333:20160907:024441.785 3: /usr/sbin/zabbix_server: ipmi poller #21 [got 0 values in 0.000004 sec, getting values](print_fatal_info+0xae) [0x47e20e]
  9333:20160907:024441.785 2: /usr/sbin/zabbix_server: ipmi poller #21 [got 0 values in 0.000004 sec, getting values]() [0x47e4e7]
  9333:20160907:024441.785 1: /lib/x86_64-linux-gnu/libc.so.6(+0x36cb0) [0x7fe10ab51cb0]
  9333:20160907:024441.785 0: [0x194ee78]
Comment by richlv [ 2016 Sep 07 ]

seems to be ipmi related - which version of libopenipmi is that ?

Comment by Mohannad [ 2016 Sep 08 ]

On Zabbix server it is
libopenipmi0 2.0.18-0ubuntu7.2 amd64 Intelligent Platform Management Interface - runtime

The server is running Ubuntu trusty 14.04.5 LTS

Comment by Aleksandrs Saveljevs [ 2016 Sep 08 ]

Since the problem happens quite regularly, could you please attach some DebugLevel=4 logs for IPMI pollers (i.e., a log that shows what IPMI pollers were doing prior to the crash)? Remember that you have the possibility to increase logging level for IPMI pollers only, see https://www.zabbix.com/documentation/2.4/manual/introduction/whatsnew240#daemon_improvements .

Comment by Mohannad [ 2016 Sep 12 ]

For some reason since I changed the Debug level to 4 it hasn't crashed, Also I have changed the server.log size, I don't think it is related
but I will post the logs once it crashes again.

Comment by Mohannad [ 2016 Sep 12 ]

Apologies, it looks like that it did crash but the service status shows as running but no data is being pulled.
I'v attached the server's log.

Comment by richlv [ 2016 Sep 12 ]

the latest log still doesn't seem to show anything crash related. are you sure that's the part where it crashed ?

Comment by Mohannad [ 2016 Sep 13 ]

Yes I've noticed that as well, but zabbix wasn't writing any logs after these lines and data wasn't being pulled; however, the service status showed as running on both the
front-end and the CLI

11511:20160910:133455.000 __zbx_zbx_setproctitle() title:'ipmi poller #64 [got 0 values in 0.000185 sec, getting values]'
11511:20160910:133455.001 __zbx_zbx_setproctitle() title:'ipmi poller #64 [got 0 values in 0.000114 sec, idle 4 sec]'
11441:20160910:133455.297 In evaluate_function() function:'super-content.vod.net.au:ipmi.system.processor_vcc1.last(#1)'

The server hasn't crashed since yesterday, I will keep monitoring it till the end of the day and see if it crashes.

Comment by Mohannad [ 2016 Sep 14 ]

Hi it crashed again , the attached is the full log

19974:20160914:140628.411 ====== Fatal information: ======
20042:20160914:140628.411 __zbx_zbx_setproctitle() title:'ipmi poller #83 [got 0 values in 0.000154 sec, getting values]'
19974:20160914:140628.411 Program counter: 0x2318ce0
20042:20160914:140628.411 In get_values()
19974:20160914:140628.412 === Registers: ===
20042:20160914:140628.412 In DCconfig_get_poller_items() poller_type:2
19974:20160914:140628.412 r8 = 2323e00 = 36847104 = 36847104
20042:20160914:140628.412 End of DCconfig_get_poller_items():0
19974:20160914:140628.412 r9 = 2327ff0 = 36863984 = 36863984
20042:20160914:140628.412 In DCconfig_get_poller_nextcheck() poller_type:2
19974:20160914:140628.412 r10 = 8 = 8 = 8
20042:20160914:140628.412 End of DCconfig_get_poller_nextcheck():1473826019
19974:20160914:140628.412 r11 = 0 = 0 = 0
20042:20160914:140628.412 End of get_values():0
19974:20160914:140628.412 r12 = 57d8ccc4 = 1473825988 = 1473825988
20047:20160914:140628.412 __zbx_zbx_setproctitle() title:'ipmi poller #88 [got 0 values in 0.000183 sec, getting values]'
20042:20160914:140628.412 In delete_inactive_ipmi_hosts()
19974:20160914:140628.412 r13 = 0 = 0 = 0
20047:20160914:140628.413 In get_values()
20042:20160914:140628.413 End of delete_inactive_ipmi_hosts()
19974:20160914:140628.413 r14 = 0 = 0 = 0
20047:20160914:140628.413 In DCconfig_get_poller_items() poller_type:2
20042:20160914:140628.413 __zbx_zbx_setproctitle() title:'ipmi poller #83 [got 0 values in 0.000994 sec, idle 5 sec]'

Comment by Aleksandrs Saveljevs [ 2016 Sep 14 ]

It seems that it is still not the full log. If we grep it by 19974 (which is the PID of the crashing process), then we will get this:

 19974:20160914:140628.411 ====== Fatal information: ======
 19974:20160914:140628.411 Program counter: 0x2318ce0
 19974:20160914:140628.412 === Registers: ===
 19974:20160914:140628.412 r8      =          2323e00 =             36847104 =             36847104
 19974:20160914:140628.412 r9      =          2327ff0 =             36863984 =             36863984
 19974:20160914:140628.412 r10     =                8 =                    8 =                    8
 19974:20160914:140628.412 r11     =                0 =                    0 =                    0
 19974:20160914:140628.412 r12     =         57d8ccc4 =           1473825988 =           1473825988
 19974:20160914:140628.412 r13     =                0 =                    0 =                    0
 19974:20160914:140628.413 r14     =                0 =                    0 =                    0
 19974:20160914:140628.413 r15     =                2 =                    2 =                    2
 19974:20160914:140628.413 rdi     =          2318be0 =             36801504 =             36801504
 19974:20160914:140628.413 rsi     =          22eed20 =             36629792 =             36629792
 19974:20160914:140628.413 rbp     =          22e4460 =             36586592 =             36586592
 19974:20160914:140628.413 rbx     =          22e4460 =             36586592 =             36586592
 19974:20160914:140628.414 rdx     =                1 =                    1 =                    1
 19974:20160914:140628.414 rax     =          2318be0 =             36801504 =             36801504
 19974:20160914:140628.414 rcx     =                0 =                    0 =                    0
 19974:20160914:140628.414 rsp     =     7ffcd8392978 =      140723936110968 =      140723936110968
 19974:20160914:140628.414 rip     =          2318ce0 =             36801760 =             36801760
 19974:20160914:140628.414 efl     =            10202 =                66050 =                66050
 19974:20160914:140628.414 csgsfs  =               33 =                   51 =                   51
 19974:20160914:140628.414 err     =               15 =                   21 =                   21
 19974:20160914:140628.414 trapno  =                e =                   14 =                   14
 19974:20160914:140628.415 oldmask =                0 =                    0 =                    0
 19974:20160914:140628.415 cr2     =          2318ce0 =             36801760 =             36801760
 19974:20160914:140628.415 === Backtrace: ===
 19974:20160914:140628.415 3: /usr/sbin/zabbix_server: ipmi poller #43 [got 0 values in 0.000121 sec, getting values](print_fatal_info+0xae) [0x47e20e]
 19974:20160914:140628.415 2: /usr/sbin/zabbix_server: ipmi poller #43 [got 0 values in 0.000121 sec, getting values]() [0x47e4e7]
 19974:20160914:140628.415 1: /lib/x86_64-linux-gnu/libc.so.6(+0x36cb0) [0x7fbc4b98ecb0]
 19974:20160914:140628.415 0: [0x2318ce0]
 ...

Unfortunately, it still does not show what the process was doing prior to the crash. Please post a longer part of the log.

Comment by Mohannad [ 2016 Sep 14 ]

this was the top of the log file, it looks like it got overwritten. I will increase the log size

Comment by Mohannad [ 2016 Sep 15 ]

Hi the service crashed again, I have attached the full log.

2008:20160915:063034.722 sensor 'P2-DIMM1A@[192.168.1.137]:623' deleted
1960:20160915:063034.722 End of domain_closed()
2021:20160915:063034.723 In delete_inactive_ipmi_hosts()
2008:20160915:063034.723 End of delete_ipmi_sensor()
1960:20160915:063034.723 Got signal [signal:11(SIGSEGV),reason:2,refaddr:0xc38db0]. Crashing ...
2021:20160915:063034.723 End of delete_inactive_ipmi_hosts()
2008:20160915:063034.723 End of sensor_change()
1960:20160915:063034.723 ====== Fatal information: ======
2021:20160915:063034.723 __zbx_zbx_setproctitle() title:'ipmi poller #56 [got 0 values in 0.001439 sec, idle 2 sec]'
2008:20160915:063034.723 In entity_change() phost:0xcb0870 host:'[192.168.1.137]:623'
1960:20160915:063034.723 Program counter: 0xc38db0
2008:20160915:063034.723 End of entity_change()

Comment by Aleksandrs Saveljevs [ 2016 Sep 20 ]

Thank you! The relevant part of the log (grepping by PID 1960) looks like this:

  1960:20160915:063034.677 __zbx_zbx_setproctitle() title:'ipmi poller #5 [got 0 values in 0.000192 sec, getting values]'
  1960:20160915:063034.678 In get_values()
  1960:20160915:063034.678 In DCconfig_get_poller_items() poller_type:2
  1960:20160915:063034.679 End of DCconfig_get_poller_items():0
  1960:20160915:063034.679 In DCconfig_get_poller_nextcheck() poller_type:2
  1960:20160915:063034.680 End of DCconfig_get_poller_nextcheck():1473885036
  1960:20160915:063034.680 End of get_values():0
  1960:20160915:063034.681 In delete_inactive_ipmi_hosts()
  1960:20160915:063034.681 In close_inactive_host(): 192.168.1.137
  1960:20160915:063034.682 EINF: 1(f.f)(m,0) sdr.c(handle_sdr_info): IPMI Error getting SDR info: ff
  1960:20160915:063034.682 In sensor_change() phost:0xcc6210 host:'[192.168.1.250]:623'
  1960:20160915:063034.683 In delete_ipmi_sensor() phost:0xcc6210 psensor:0xc94730
  1960:20160915:063034.684 sensor 'System Temp@[192.168.1.250]:623' deleted
  1960:20160915:063034.684 End of delete_ipmi_sensor()
  1960:20160915:063034.685 End of sensor_change()
  1960:20160915:063034.686 In sensor_change() phost:0xcc6210 host:'[192.168.1.250]:623'
  1960:20160915:063034.687 In delete_ipmi_sensor() phost:0xcc6210 psensor:0xc96ab0
  1960:20160915:063034.688 sensor 'CPU Temp@[192.168.1.250]:623' deleted
  1960:20160915:063034.689 End of delete_ipmi_sensor()
  1960:20160915:063034.689 End of sensor_change()
  1960:20160915:063034.690 In sensor_change() phost:0xcc6210 host:'[192.168.1.250]:623'
  1960:20160915:063034.691 In delete_ipmi_sensor() phost:0xcc6210 psensor:0xcc8960
  1960:20160915:063034.692 sensor 'FAN 1@[192.168.1.250]:623' deleted
  1960:20160915:063034.692 End of delete_ipmi_sensor()
  1960:20160915:063034.693 End of sensor_change()
  1960:20160915:063034.694 In entity_change() phost:0xcc6210 host:'[192.168.1.250]:623'
  1960:20160915:063034.694 End of entity_change()
  1960:20160915:063034.695 In sensor_change() phost:0xcc6210 host:'[192.168.1.250]:623'
  1960:20160915:063034.696 In delete_ipmi_sensor() phost:0xcc6210 psensor:0xcc9380
  1960:20160915:063034.697 sensor 'FAN 2@[192.168.1.250]:623' deleted
  1960:20160915:063034.697 End of delete_ipmi_sensor()
  1960:20160915:063034.697 End of sensor_change()
  1960:20160915:063034.698 In entity_change() phost:0xcc6210 host:'[192.168.1.250]:623'
  1960:20160915:063034.698 End of entity_change()
  1960:20160915:063034.699 In sensor_change() phost:0xcc6210 host:'[192.168.1.250]:623'
  1960:20160915:063034.699 In delete_ipmi_sensor() phost:0xcc6210 psensor:0xcc9dc0
  1960:20160915:063034.699 sensor 'FAN 3@[192.168.1.250]:623' deleted
  1960:20160915:063034.700 End of delete_ipmi_sensor()
  1960:20160915:063034.700 End of sensor_change()
  1960:20160915:063034.700 In entity_change() phost:0xcc6210 host:'[192.168.1.250]:623'
  1960:20160915:063034.701 End of entity_change()
  1960:20160915:063034.701 In sensor_change() phost:0xcc6210 host:'[192.168.1.250]:623'
  1960:20160915:063034.701 In delete_ipmi_sensor() phost:0xcc6210 psensor:0xcca990
  1960:20160915:063034.702 sensor 'FAN 4@[192.168.1.250]:623' deleted
  1960:20160915:063034.702 End of delete_ipmi_sensor()
  1960:20160915:063034.702 End of sensor_change()
  1960:20160915:063034.702 In entity_change() phost:0xcc6210 host:'[192.168.1.250]:623'
  1960:20160915:063034.702 End of entity_change()
  1960:20160915:063034.703 In sensor_change() phost:0xcc6210 host:'[192.168.1.250]:623'
  1960:20160915:063034.703 In delete_ipmi_sensor() phost:0xcc6210 psensor:0xccb560
  1960:20160915:063034.703 sensor 'FAN A@[192.168.1.250]:623' deleted
  1960:20160915:063034.703 End of delete_ipmi_sensor()
  1960:20160915:063034.703 End of sensor_change()
  1960:20160915:063034.703 In entity_change() phost:0xcc6210 host:'[192.168.1.250]:623'
  1960:20160915:063034.704 End of entity_change()
  1960:20160915:063034.704 In sensor_change() phost:0xcc6210 host:'[192.168.1.250]:623'
  1960:20160915:063034.704 In delete_ipmi_sensor() phost:0xcc6210 psensor:0xccc130
  1960:20160915:063034.704 sensor 'Vcore@[192.168.1.250]:623' deleted
  1960:20160915:063034.705 End of delete_ipmi_sensor()
  1960:20160915:063034.705 End of sensor_change()
  1960:20160915:063034.705 In entity_change() phost:0xcc6210 host:'[192.168.1.250]:623'
  1960:20160915:063034.705 End of entity_change()
  1960:20160915:063034.706 In sensor_change() phost:0xcc6210 host:'[192.168.1.250]:623'
  1960:20160915:063034.706 In delete_ipmi_sensor() phost:0xcc6210 psensor:0xcccd00
  1960:20160915:063034.706 sensor '3.3VCC@[192.168.1.250]:623' deleted
  1960:20160915:063034.706 End of delete_ipmi_sensor()
  1960:20160915:063034.706 End of sensor_change()
  1960:20160915:063034.707 In sensor_change() phost:0xcc6210 host:'[192.168.1.250]:623'
  1960:20160915:063034.707 In delete_ipmi_sensor() phost:0xcc6210 psensor:0xccd8d0
  1960:20160915:063034.707 sensor '12V@[192.168.1.250]:623' deleted
  1960:20160915:063034.707 End of delete_ipmi_sensor()
  1960:20160915:063034.707 End of sensor_change()
  1960:20160915:063034.708 In sensor_change() phost:0xcc6210 host:'[192.168.1.250]:623'
  1960:20160915:063034.708 In delete_ipmi_sensor() phost:0xcc6210 psensor:0xcce4a0
  1960:20160915:063034.708 sensor 'VDIMM@[192.168.1.250]:623' deleted
  1960:20160915:063034.709 End of delete_ipmi_sensor()
  1960:20160915:063034.709 End of sensor_change()
  1960:20160915:063034.709 In sensor_change() phost:0xcc6210 host:'[192.168.1.250]:623'
  1960:20160915:063034.709 In delete_ipmi_sensor() phost:0xcc6210 psensor:0xccf070
  1960:20160915:063034.710 sensor '5VCC@[192.168.1.250]:623' deleted
  1960:20160915:063034.710 End of delete_ipmi_sensor()
  1960:20160915:063034.710 End of sensor_change()
  1960:20160915:063034.711 In sensor_change() phost:0xcc6210 host:'[192.168.1.250]:623'
  1960:20160915:063034.711 In delete_ipmi_sensor() phost:0xcc6210 psensor:0xccfc40
  1960:20160915:063034.711 sensor '-12V@[192.168.1.250]:623' deleted
  1960:20160915:063034.711 End of delete_ipmi_sensor()
  1960:20160915:063034.712 End of sensor_change()
  1960:20160915:063034.712 In sensor_change() phost:0xcc6210 host:'[192.168.1.250]:623'
  1960:20160915:063034.713 In delete_ipmi_sensor() phost:0xcc6210 psensor:0xcd0810
  1960:20160915:063034.713 sensor 'VBAT@[192.168.1.250]:623' deleted
  1960:20160915:063034.713 End of delete_ipmi_sensor()
  1960:20160915:063034.713 End of sensor_change()
  1960:20160915:063034.713 In sensor_change() phost:0xcc6210 host:'[192.168.1.250]:623'
  1960:20160915:063034.714 In delete_ipmi_sensor() phost:0xcc6210 psensor:0xcd13e0
  1960:20160915:063034.714 sensor 'VSB@[192.168.1.250]:623' deleted
  1960:20160915:063034.714 End of delete_ipmi_sensor()
  1960:20160915:063034.714 End of sensor_change()
  1960:20160915:063034.714 In sensor_change() phost:0xcc6210 host:'[192.168.1.250]:623'
  1960:20160915:063034.715 In delete_ipmi_sensor() phost:0xcc6210 psensor:0xcd1fb0
  1960:20160915:063034.715 sensor 'AVCC@[192.168.1.250]:623' deleted
  1960:20160915:063034.715 End of delete_ipmi_sensor()
  1960:20160915:063034.715 End of sensor_change()
  1960:20160915:063034.715 In entity_change() phost:0xcc6210 host:'[192.168.1.250]:623'
  1960:20160915:063034.715 End of entity_change()
  1960:20160915:063034.716 In sensor_change() phost:0xcc6210 host:'[192.168.1.250]:623'
  1960:20160915:063034.716 In delete_ipmi_sensor() phost:0xcc6210 psensor:0xcd2b80
  1960:20160915:063034.716 sensor 'Chassis Intru@[192.168.1.250]:623' deleted
  1960:20160915:063034.716 End of delete_ipmi_sensor()
  1960:20160915:063034.717 End of sensor_change()
  1960:20160915:063034.717 In sensor_change() phost:0xcc6210 host:'[192.168.1.250]:623'
  1960:20160915:063034.717 End of sensor_change()
  1960:20160915:063034.718 In sensor_change() phost:0xcc6210 host:'[192.168.1.250]:623'
  1960:20160915:063034.718 End of sensor_change()
  1960:20160915:063034.718 In sensor_change() phost:0xcc6210 host:'[192.168.1.250]:623'
  1960:20160915:063034.719 End of sensor_change()
  1960:20160915:063034.719 In sensor_change() phost:0xcc6210 host:'[192.168.1.250]:623'
  1960:20160915:063034.719 End of sensor_change()
  1960:20160915:063034.719 In entity_change() phost:0xcc6210 host:'[192.168.1.250]:623'
  1960:20160915:063034.719 End of entity_change()
  1960:20160915:063034.720 In control_change() phost:0xcc6210 host:'[192.168.1.250]:623'
  1960:20160915:063034.720 In delete_ipmi_control() phost:0xcc6210 pcontrol:0xcbfbe0
  1960:20160915:063034.720 control 'power@[192.168.1.250]:623' deleted
  1960:20160915:063034.720 End of delete_ipmi_control()
  1960:20160915:063034.720 End of control_change()
  1960:20160915:063034.721 In control_change() phost:0xcc6210 host:'[192.168.1.250]:623'
  1960:20160915:063034.721 In delete_ipmi_control() phost:0xcc6210 pcontrol:0xcc05c0
  1960:20160915:063034.721 control 'reset@[192.168.1.250]:623' deleted
  1960:20160915:063034.721 End of delete_ipmi_control()
  1960:20160915:063034.722 End of control_change()
  1960:20160915:063034.722 In entity_change() phost:0xcc6210 host:'[192.168.1.250]:623'
  1960:20160915:063034.722 End of entity_change()
  1960:20160915:063034.722 In domain_closed() phost:0xc85460 host:'[192.168.1.137]:623'
  1960:20160915:063034.722 End of domain_closed()
  1960:20160915:063034.723 Got signal [signal:11(SIGSEGV),reason:2,refaddr:0xc38db0]. Crashing ...
  1960:20160915:063034.723 ====== Fatal information: ======
  1960:20160915:063034.723 Program counter: 0xc38db0
  1960:20160915:063034.723 === Registers: ===
  1960:20160915:063034.723 r8      =           cbf800 =             13367296 =             13367296
  1960:20160915:063034.723 r9      =           c91fc0 =             13180864 =             13180864
  1960:20160915:063034.723 r10     =                8 =                    8 =                    8
  1960:20160915:063034.724 r11     =                0 =                    0 =                    0
  1960:20160915:063034.724 r12     =         57d9b36a =           1473885034 =           1473885034
  1960:20160915:063034.724 r13     =           cc6210 =             13394448 =             13394448
  1960:20160915:063034.724 r14     =                0 =                    0 =                    0
  1960:20160915:063034.724 r15     =                2 =                    2 =                    2
  1960:20160915:063034.724 rdi     =           cc7050 =             13398096 =             13398096
  1960:20160915:063034.724 rsi     =                0 =                    0 =                    0
  1960:20160915:063034.724 rbp     =           c85460 =             13128800 =             13128800
  1960:20160915:063034.724 rbx     =           c85460 =             13128800 =             13128800
  1960:20160915:063034.725 rdx     =                1 =                    1 =                    1
  1960:20160915:063034.725 rax     =           cc7050 =             13398096 =             13398096
  1960:20160915:063034.725 rcx     =                0 =                    0 =                    0
  1960:20160915:063034.725 rsp     =     7ffee09f3398 =      140732666950552 =      140732666950552
  1960:20160915:063034.725 rip     =           c38db0 =             12815792 =             12815792
  1960:20160915:063034.725 efl     =            10202 =                66050 =                66050
  1960:20160915:063034.725 csgsfs  =               33 =                   51 =                   51
  1960:20160915:063034.725 err     =               15 =                   21 =                   21
  1960:20160915:063034.726 trapno  =                e =                   14 =                   14
  1960:20160915:063034.726 oldmask =                0 =                    0 =                    0
  1960:20160915:063034.726 cr2     =           c38db0 =             12815792 =             12815792
  1960:20160915:063034.726 === Backtrace: ===
  1960:20160915:063034.760 3: /usr/sbin/zabbix_server: ipmi poller #5 [got 0 values in 0.000192 sec, getting values](print_fatal_info+0xae) [0x47e20e]
  1960:20160915:063034.760 2: /usr/sbin/zabbix_server: ipmi poller #5 [got 0 values in 0.000192 sec, getting values]() [0x47e4e7]
  1960:20160915:063034.760 1: /lib/x86_64-linux-gnu/libc.so.6(+0x36cb0) [0x7f384533ccb0]
  1960:20160915:063034.760 0: [0xc38db0]
  1960:20160915:063034.760 === Memory map: ===
  1960:20160915:063034.760 00400000-0056c000 r-xp 00000000 fc:00 3020799                            /usr/sbin/zabbix_server
  1960:20160915:063034.760 0076b000-0076c000 r--p 0016b000 fc:00 3020799                            /usr/sbin/zabbix_server
  1960:20160915:063034.760 0076c000-00773000 rw-p 0016c000 fc:00 3020799                            /usr/sbin/zabbix_server
  1960:20160915:063034.760 00773000-00779000 rw-p 00000000 00:00 0 
  1960:20160915:063034.761 00c38000-00c59000 rw-p 00000000 00:00 0                                  [heap]
  1960:20160915:063034.761 00c59000-00c93000 rw-p 00000000 00:00 0                                  [heap]
  1960:20160915:063034.761 00c93000-00d17000 rw-p 00000000 00:00 0                                  [heap]
  1960:20160915:063034.761 7f3678000000-7f3678021000 rw-p 00000000 00:00 0 
  1960:20160915:063034.761 7f3678021000-7f367c000000 ---p 00000000 00:00 0 
  1960:20160915:063034.761 7f367fbc0000-7f367fbd6000 r-xp 00000000 fc:00 5636289                    /lib/x86_64-linux-gnu/libgcc_s.so.1
  1960:20160915:063034.761 7f367fbd6000-7f367fdd5000 ---p 00016000 fc:00 5636289                    /lib/x86_64-linux-gnu/libgcc_s.so.1
  1960:20160915:063034.761 7f367fdd5000-7f367fdd6000 rw-p 00015000 fc:00 5636289                    /lib/x86_64-linux-gnu/libgcc_s.so.1
  1960:20160915:063034.761 7f367fdd6000-7f367fdd7000 ---p 00000000 00:00 0 
  1960:20160915:063034.761 7f367fdd7000-7f36805d7000 rw-p 00000000 00:00 0 
  1960:20160915:063034.761 7f36805d7000-7f36c05d7000 rw-s 00000000 00:05 4718600                    /SYSV76000018 (deleted)
  1960:20160915:063034.762 7f36c05d7000-7f37005d7000 rw-s 00000000 00:05 4685831                    /SYSV77000018 (deleted)
  1960:20160915:063034.762 7f37005d7000-7f371390b000 rw-s 00000000 00:05 4620293                    /SYSV73000018 (deleted)
  1960:20160915:063034.762 7f371390b000-7f37805d8000 rw-s 00000000 00:05 4587524                    /SYSV67000018 (deleted)
  1960:20160915:063034.762 7f37805d8000-7f37c05d8000 rw-s 00000000 00:05 4554755                    /SYSV74000018 (deleted)
  1960:20160915:063034.762 7f37c05d8000-7f38005d8000 rw-s 00000000 00:05 4521986                    /SYSV48000018 (deleted)
  1960:20160915:063034.762 7f38005d8000-7f38405d8000 rw-s 00000000 00:05 4489217                    /SYSV68000018 (deleted)
  1960:20160915:063034.762 7f38405d8000-7f38405e2000 r-xp 00000000 fc:00 5640214                    /lib/x86_64-linux-gnu/libnss_files-2.19.so
  1960:20160915:063034.762 7f38405e2000-7f38407e1000 ---p 0000a000 fc:00 5640214                    /lib/x86_64-linux-gnu/libnss_files-2.19.so
  1960:20160915:063034.762 7f38407e1000-7f38407e2000 r--p 00009000 fc:00 5640214                    /lib/x86_64-linux-gnu/libnss_files-2.19.so
  1960:20160915:063034.762 7f38407e2000-7f38407e3000 rw-p 0000a000 fc:00 5640214                    /lib/x86_64-linux-gnu/libnss_files-2.19.so
  1960:20160915:063034.762 7f38407e3000-7f38407ee000 r-xp 00000000 fc:00 5640206                    /lib/x86_64-linux-gnu/libnss_nis-2.19.so
  1960:20160915:063034.762 7f38407ee000-7f38409ed000 ---p 0000b000 fc:00 5640206                    /lib/x86_64-linux-gnu/libnss_nis-2.19.so
  1960:20160915:063034.762 7f38409ed000-7f38409ee000 r--p 0000a000 fc:00 5640206                    /lib/x86_64-linux-gnu/libnss_nis-2.19.so
  1960:20160915:063034.762 7f38409ee000-7f38409ef000 rw-p 0000b000 fc:00 5640206                    /lib/x86_64-linux-gnu/libnss_nis-2.19.so
  1960:20160915:063034.762 7f38409ef000-7f3840a06000 r-xp 00000000 fc:00 5640202                    /lib/x86_64-linux-gnu/libnsl-2.19.so
  1960:20160915:063034.762 7f3840a06000-7f3840c05000 ---p 00017000 fc:00 5640202                    /lib/x86_64-linux-gnu/libnsl-2.19.so
  1960:20160915:063034.762 7f3840c05000-7f3840c06000 r--p 00016000 fc:00 5640202                    /lib/x86_64-linux-gnu/libnsl-2.19.so
  1960:20160915:063034.762 7f3840c06000-7f3840c07000 rw-p 00017000 fc:00 5640202                    /lib/x86_64-linux-gnu/libnsl-2.19.so
  1960:20160915:063034.762 7f3840c07000-7f3840c09000 rw-p 00000000 00:00 0 
  1960:20160915:063034.762 7f3840c09000-7f3840c12000 r-xp 00000000 fc:00 5640201                    /lib/x86_64-linux-gnu/libnss_compat-2.19.so
  1960:20160915:063034.762 7f3840c12000-7f3840e11000 ---p 00009000 fc:00 5640201                    /lib/x86_64-linux-gnu/libnss_compat-2.19.so
  1960:20160915:063034.762 7f3840e11000-7f3840e12000 r--p 00008000 fc:00 5640201                    /lib/x86_64-linux-gnu/libnss_compat-2.19.so
  1960:20160915:063034.762 7f3840e12000-7f3840e13000 rw-p 00009000 fc:00 5640201                    /lib/x86_64-linux-gnu/libnss_compat-2.19.so
  1960:20160915:063034.762 7f3840e13000-7f3840e15000 r-xp 00000000 fc:00 5640042                    /lib/x86_64-linux-gnu/libkeyutils.so.1.4
  1960:20160915:063034.762 7f3840e15000-7f3841015000 ---p 00002000 fc:00 5640042                    /lib/x86_64-linux-gnu/libkeyutils.so.1.4
  1960:20160915:063034.762 7f3841015000-7f3841016000 r--p 00002000 fc:00 5640042                    /lib/x86_64-linux-gnu/libkeyutils.so.1.4
  1960:20160915:063034.762 7f3841016000-7f3841017000 rw-p 00003000 fc:00 5640042                    /lib/x86_64-linux-gnu/libkeyutils.so.1.4
  1960:20160915:063034.762 7f3841017000-7f3841020000 r-xp 00000000 fc:00 5640199                    /lib/x86_64-linux-gnu/libcrypt-2.19.so
  1960:20160915:063034.762 7f3841020000-7f3841220000 ---p 00009000 fc:00 5640199                    /lib/x86_64-linux-gnu/libcrypt-2.19.so
  1960:20160915:063034.762 7f3841220000-7f3841221000 r--p 00009000 fc:00 5640199                    /lib/x86_64-linux-gnu/libcrypt-2.19.so
  1960:20160915:063034.762 7f3841221000-7f3841222000 rw-p 0000a000 fc:00 5640199                    /lib/x86_64-linux-gnu/libcrypt-2.19.so
  1960:20160915:063034.762 7f3841222000-7f3841250000 rw-p 00000000 00:00 0 
  1960:20160915:063034.762 7f3841250000-7f3841304000 r-xp 00000000 fc:00 3017526                    /usr/lib/x86_64-linux-gnu/libsqlite3.so.0.8.6
  1960:20160915:063034.762 7f3841304000-7f3841504000 ---p 000b4000 fc:00 3017526                    /usr/lib/x86_64-linux-gnu/libsqlite3.so.0.8.6
  1960:20160915:063034.762 7f3841504000-7f3841506000 r--p 000b4000 fc:00 3017526                    /usr/lib/x86_64-linux-gnu/libsqlite3.so.0.8.6
  1960:20160915:063034.762 7f3841506000-7f3841508000 rw-p 000b6000 fc:00 3017526                    /usr/lib/x86_64-linux-gnu/libsqlite3.so.0.8.6
  1960:20160915:063034.762 7f3841508000-7f3841509000 rw-p 00000000 00:00 0 
  1960:20160915:063034.762 7f3841509000-7f384154e000 r-xp 00000000 fc:00 3015170                    /usr/lib/x86_64-linux-gnu/libhx509.so.5.0.0
  1960:20160915:063034.762 7f384154e000-7f384174d000 ---p 00045000 fc:00 3015170                    /usr/lib/x86_64-linux-gnu/libhx509.so.5.0.0
  1960:20160915:063034.762 7f384174d000-7f384174f000 r--p 00044000 fc:00 3015170                    /usr/lib/x86_64-linux-gnu/libhx509.so.5.0.0
  1960:20160915:063034.762 7f384174f000-7f3841751000 rw-p 00046000 fc:00 3015170                    /usr/lib/x86_64-linux-gnu/libhx509.so.5.0.0
  1960:20160915:063034.762 7f3841751000-7f3841752000 rw-p 00000000 00:00 0 
  1960:20160915:063034.763 7f3841752000-7f384175f000 r-xp 00000000 fc:00 3015166                    /usr/lib/x86_64-linux-gnu/libheimbase.so.1.0.0
  1960:20160915:063034.763 7f384175f000-7f384195e000 ---p 0000d000 fc:00 3015166                    /usr/lib/x86_64-linux-gnu/libheimbase.so.1.0.0
  1960:20160915:063034.763 7f384195e000-7f384195f000 r--p 0000c000 fc:00 3015166                    /usr/lib/x86_64-linux-gnu/libheimbase.so.1.0.0
  1960:20160915:063034.763 7f384195f000-7f3841960000 rw-p 0000d000 fc:00 3015166                    /usr/lib/x86_64-linux-gnu/libheimbase.so.1.0.0
  1960:20160915:063034.763 7f3841960000-7f3841987000 r-xp 00000000 fc:00 3015168                    /usr/lib/x86_64-linux-gnu/libwind.so.0.0.0
  1960:20160915:063034.763 7f3841987000-7f3841b87000 ---p 00027000 fc:00 3015168                    /usr/lib/x86_64-linux-gnu/libwind.so.0.0.0
  1960:20160915:063034.763 7f3841b87000-7f3841b88000 r--p 00027000 fc:00 3015168                    /usr/lib/x86_64-linux-gnu/libwind.so.0.0.0
  1960:20160915:063034.763 7f3841b88000-7f3841b89000 rw-p 00028000 fc:00 3015168                    /usr/lib/x86_64-linux-gnu/libwind.so.0.0.0
  1960:20160915:063034.763 7f3841b89000-7f3841b90000 r-xp 00000000 fc:00 3017488                    /usr/lib/x86_64-linux-gnu/libffi.so.6.0.1
  1960:20160915:063034.763 7f3841b90000-7f3841d8f000 ---p 00007000 fc:00 3017488                    /usr/lib/x86_64-linux-gnu/libffi.so.6.0.1
  1960:20160915:063034.763 7f3841d8f000-7f3841d90000 r--p 00006000 fc:00 3017488                    /usr/lib/x86_64-linux-gnu/libffi.so.6.0.1
  1960:20160915:063034.763 7f3841d90000-7f3841d91000 rw-p 00007000 fc:00 3017488                    /usr/lib/x86_64-linux-gnu/libffi.so.6.0.1
  1960:20160915:063034.763 7f3841d91000-7f3841d9b000 r-xp 00000000 fc:00 3015147                    /usr/lib/x86_64-linux-gnu/libkrb5support.so.0.1
  1960:20160915:063034.763 7f3841d9b000-7f3841f9a000 ---p 0000a000 fc:00 3015147                    /usr/lib/x86_64-linux-gnu/libkrb5support.so.0.1
  1960:20160915:063034.763 7f3841f9a000-7f3841f9b000 r--p 00009000 fc:00 3015147                    /usr/lib/x86_64-linux-gnu/libkrb5support.so.0.1
  1960:20160915:063034.763 7f3841f9b000-7f3841f9c000 rw-p 0000a000 fc:00 3015147                    /usr/lib/x86_64-linux-gnu/libkrb5support.so.0.1
  1960:20160915:063034.763 7f3841f9c000-7f3841fc8000 r-xp 00000000 fc:00 3015149                    /usr/lib/x86_64-linux-gnu/libk5crypto.so.3.1
  1960:20160915:063034.763 7f3841fc8000-7f38421c7000 ---p 0002c000 fc:00 3015149                    /usr/lib/x86_64-linux-gnu/libk5crypto.so.3.1
  1960:20160915:063034.763 7f38421c7000-7f38421c9000 r--p 0002b000 fc:00 3015149                    /usr/lib/x86_64-linux-gnu/libk5crypto.so.3.1
  1960:20160915:063034.763 7f38421c9000-7f38421ca000 rw-p 0002d000 fc:00 3015149                    /usr/lib/x86_64-linux-gnu/libk5crypto.so.3.1
  1960:20160915:063034.763 7f38421ca000-7f38421cb000 rw-p 00000000 00:00 0 
  1960:20160915:063034.763 7f38421cb000-7f3842287000 r-xp 00000000 fc:00 3015151                    /usr/lib/x86_64-linux-gnu/libkrb5.so.3.3
  1960:20160915:063034.763 7f3842287000-7f3842487000 ---p 000bc000 fc:00 3015151                    /usr/lib/x86_64-linux-gnu/libkrb5.so.3.3
  1960:20160915:063034.763 7f3842487000-7f3842494000 r--p 000bc000 fc:00 3015151                    /usr/lib/x86_64-linux-gnu/libkrb5.so.3.3
  1960:20160915:063034.763 7f3842494000-7f3842496000 rw-p 000c9000 fc:00 3015151                    /usr/lib/x86_64-linux-gnu/libkrb5.so.3.3
  1960:20160915:063034.763 7f3842496000-7f38424aa000 r-xp 00000000 fc:00 3015160                    /usr/lib/x86_64-linux-gnu/libroken.so.18.1.0
  1960:20160915:063034.763 7f38424aa000-7f38426a9000 ---p 00014000 fc:00 3015160                    /usr/lib/x86_64-linux-gnu/libroken.so.18.1.0
  1960:20160915:063034.764 7f38426a9000-7f38426aa000 r--p 00013000 fc:00 3015160                    /usr/lib/x86_64-linux-gnu/libroken.so.18.1.0
  1960:20160915:063034.764 7f38426aa000-7f38426ab000 rw-p 00014000 fc:00 3015160                    /usr/lib/x86_64-linux-gnu/libroken.so.18.1.0
  1960:20160915:063034.764 7f38426ab000-7f38426db000 r-xp 00000000 fc:00 3015164                    /usr/lib/x86_64-linux-gnu/libhcrypto.so.4.1.0
  1960:20160915:063034.764 7f38426db000-7f38428db000 ---p 00030000 fc:00 3015164                    /usr/lib/x86_64-linux-gnu/libhcrypto.so.4.1.0
  1960:20160915:063034.764 7f38428db000-7f38428dc000 r--p 00030000 fc:00 3015164                    /usr/lib/x86_64-linux-gnu/libhcrypto.so.4.1.0
  1960:20160915:063034.764 7f38428dc000-7f38428dd000 rw-p 00031000 fc:00 3015164                    /usr/lib/x86_64-linux-gnu/libhcrypto.so.4.1.0
  1960:20160915:063034.764 7f38428dd000-7f38428de000 rw-p 00000000 00:00 0 
  1960:20160915:063034.764 7f38428de000-7f38428e1000 r-xp 00000000 fc:00 5636274                    /lib/x86_64-linux-gnu/libcom_err.so.2.1
  1960:20160915:063034.764 7f38428e1000-7f3842ae0000 ---p 00003000 fc:00 5636274                    /lib/x86_64-linux-gnu/libcom_err.so.2.1
  1960:20160915:063034.764 7f3842ae0000-7f3842ae1000 r--p 00002000 fc:00 5636274                    /lib/x86_64-linux-gnu/libcom_err.so.2.1
  1960:20160915:063034.764 7f3842ae1000-7f3842ae2000 rw-p 00003000 fc:00 5636274                    /lib/x86_64-linux-gnu/libcom_err.so.2.1
  1960:20160915:063034.764 7f3842ae2000-7f3842b7f000 r-xp 00000000 fc:00 3015162                    /usr/lib/x86_64-linux-gnu/libasn1.so.8.0.0
  1960:20160915:063034.764 7f3842b7f000-7f3842d7f000 ---p 0009d000 fc:00 3015162                    /usr/lib/x86_64-linux-gnu/libasn1.so.8.0.0
  1960:20160915:063034.764 7f3842d7f000-7f3842d80000 r--p 0009d000 fc:00 3015162                    /usr/lib/x86_64-linux-gnu/libasn1.so.8.0.0
  1960:20160915:063034.764 7f3842d80000-7f3842d83000 rw-p 0009e000 fc:00 3015162                    /usr/lib/x86_64-linux-gnu/libasn1.so.8.0.0
  1960:20160915:063034.764 7f3842d83000-7f3842e05000 r-xp 00000000 fc:00 3015172                    /usr/lib/x86_64-linux-gnu/libkrb5.so.26.0.0
  1960:20160915:063034.764 7f3842e05000-7f3843004000 ---p 00082000 fc:00 3015172                    /usr/lib/x86_64-linux-gnu/libkrb5.so.26.0.0
  1960:20160915:063034.764 7f3843004000-7f3843007000 r--p 00081000 fc:00 3015172                    /usr/lib/x86_64-linux-gnu/libkrb5.so.26.0.0
  1960:20160915:063034.764 7f3843007000-7f384300a000 rw-p 00084000 fc:00 3015172                    /usr/lib/x86_64-linux-gnu/libkrb5.so.26.0.0
  1960:20160915:063034.764 7f384300a000-7f384300b000 rw-p 00000000 00:00 0 
  1960:20160915:063034.764 7f384300b000-7f3843013000 r-xp 00000000 fc:00 3015174                    /usr/lib/x86_64-linux-gnu/libheimntlm.so.0.1.0
  1960:20160915:063034.764 7f3843013000-7f3843212000 ---p 00008000 fc:00 3015174                    /usr/lib/x86_64-linux-gnu/libheimntlm.so.0.1.0
  1960:20160915:063034.764 7f3843212000-7f3843213000 r--p 00007000 fc:00 3015174                    /usr/lib/x86_64-linux-gnu/libheimntlm.so.0.1.0
  1960:20160915:063034.764 7f3843213000-7f3843214000 rw-p 00008000 fc:00 3015174                    /usr/lib/x86_64-linux-gnu/libheimntlm.so.0.1.0
  1960:20160915:063034.764 7f3843214000-7f3843218000 r-xp 00000000 fc:00 5636293                    /lib/x86_64-linux-gnu/libgpg-error.so.0.10.0
  1960:20160915:063034.764 7f3843218000-7f3843417000 ---p 00004000 fc:00 5636293                    /lib/x86_64-linux-gnu/libgpg-error.so.0.10.0
  1960:20160915:063034.764 7f3843417000-7f3843418000 r--p 00003000 fc:00 5636293                    /lib/x86_64-linux-gnu/libgpg-error.so.0.10.0
  1960:20160915:063034.764 7f3843418000-7f3843419000 rw-p 00004000 fc:00 5636293                    /lib/x86_64-linux-gnu/libgpg-error.so.0.10.0
  1960:20160915:063034.764 7f3843419000-7f3843454000 r-xp 00000000 fc:00 3017516                    /usr/lib/x86_64-linux-gnu/libp11-kit.so.0.0.0
  1960:20160915:063034.764 7f3843454000-7f3843653000 ---p 0003b000 fc:00 3017516                    /usr/lib/x86_64-linux-gnu/libp11-kit.so.0.0.0
  1960:20160915:063034.764 7f3843653000-7f3843659000 r--p 0003a000 fc:00 3017516                    /usr/lib/x86_64-linux-gnu/libp11-kit.so.0.0.0
  1960:20160915:063034.764 7f3843659000-7f384365b000 rw-p 00040000 fc:00 3017516                    /usr/lib/x86_64-linux-gnu/libp11-kit.so.0.0.0
  1960:20160915:063034.764 7f384365b000-7f384366d000 r-xp 00000000 fc:00 3017270                    /usr/lib/x86_64-linux-gnu/libtasn1.so.6.2.0
  1960:20160915:063034.764 7f384366d000-7f384386d000 ---p 00012000 fc:00 3017270                    /usr/lib/x86_64-linux-gnu/libtasn1.so.6.2.0
  1960:20160915:063034.764 7f384386d000-7f384386e000 r--p 00012000 fc:00 3017270                    /usr/lib/x86_64-linux-gnu/libtasn1.so.6.2.0
  1960:20160915:063034.764 7f384386e000-7f384386f000 rw-p 00013000 fc:00 3017270                    /usr/lib/x86_64-linux-gnu/libtasn1.so.6.2.0
  1960:20160915:063034.764 7f384386f000-7f38438b3000 r-xp 00000000 fc:00 3015156                    /usr/lib/x86_64-linux-gnu/libgssapi_krb5.so.2.2
  1960:20160915:063034.764 7f38438b3000-7f3843ab3000 ---p 00044000 fc:00 3015156                    /usr/lib/x86_64-linux-gnu/libgssapi_krb5.so.2.2
  1960:20160915:063034.764 7f3843ab3000-7f3843ab4000 r--p 00044000 fc:00 3015156                    /usr/lib/x86_64-linux-gnu/libgssapi_krb5.so.2.2
  1960:20160915:063034.764 7f3843ab4000-7f3843ab6000 rw-p 00045000 fc:00 3015156                    /usr/lib/x86_64-linux-gnu/libgssapi_krb5.so.2.2
  1960:20160915:063034.764 7f3843ab6000-7f3843acf000 r-xp 00000000 fc:00 3015201                    /usr/lib/x86_64-linux-gnu/librtmp.so.0
  1960:20160915:063034.764 7f3843acf000-7f3843cce000 ---p 00019000 fc:00 3015201                    /usr/lib/x86_64-linux-gnu/librtmp.so.0
  1960:20160915:063034.764 7f3843cce000-7f3843ccf000 r--p 00018000 fc:00 3015201                    /usr/lib/x86_64-linux-gnu/librtmp.so.0
  1960:20160915:063034.764 7f3843ccf000-7f3843cd0000 rw-p 00019000 fc:00 3015201                    /usr/lib/x86_64-linux-gnu/librtmp.so.0
  1960:20160915:063034.764 7f3843cd0000-7f3843d01000 r-xp 00000000 fc:00 3014760                    /usr/lib/x86_64-linux-gnu/libidn.so.11.6.11
  1960:20160915:063034.764 7f3843d01000-7f3843f01000 ---p 00031000 fc:00 3014760                    /usr/lib/x86_64-linux-gnu/libidn.so.11.6.11
  1960:20160915:063034.764 7f3843f01000-7f3843f02000 r--p 00031000 fc:00 3014760                    /usr/lib/x86_64-linux-gnu/libidn.so.11.6.11
  1960:20160915:063034.765 7f3843f02000-7f3843f03000 rw-p 00032000 fc:00 3014760                    /usr/lib/x86_64-linux-gnu/libidn.so.11.6.11
  1960:20160915:063034.765 7f3843f03000-7f3843f3d000 r-xp 00000000 fc:00 3015176                    /usr/lib/x86_64-linux-gnu/libgssapi.so.3.0.0
  1960:20160915:063034.765 7f3843f3d000-7f384413d000 ---p 0003a000 fc:00 3015176                    /usr/lib/x86_64-linux-gnu/libgssapi.so.3.0.0
  1960:20160915:063034.765 7f384413d000-7f384413e000 r--p 0003a000 fc:00 3015176                    /usr/lib/x86_64-linux-gnu/libgssapi.so.3.0.0
  1960:20160915:063034.765 7f384413e000-7f3844140000 rw-p 0003b000 fc:00 3015176                    /usr/lib/x86_64-linux-gnu/libgssapi.so.3.0.0
  1960:20160915:063034.765 7f3844140000-7f3844141000 rw-p 00000000 00:00 0 
  1960:20160915:063034.765 7f3844141000-7f384415a000 r-xp 00000000 fc:00 3015183                    /usr/lib/x86_64-linux-gnu/libsasl2.so.2.0.25
  1960:20160915:063034.765 7f384415a000-7f384435a000 ---p 00019000 fc:00 3015183                    /usr/lib/x86_64-linux-gnu/libsasl2.so.2.0.25
  1960:20160915:063034.765 7f384435a000-7f384435b000 r--p 00019000 fc:00 3015183                    /usr/lib/x86_64-linux-gnu/libsasl2.so.2.0.25
  1960:20160915:063034.765 7f384435b000-7f384435c000 rw-p 0001a000 fc:00 3015183                    /usr/lib/x86_64-linux-gnu/libsasl2.so.2.0.25
  1960:20160915:063034.765 7f384435c000-7f3844363000 r-xp 00000000 fc:00 3020782                    /usr/lib/libOpenIPMIutils.so.0.0.1
  1960:20160915:063034.765 7f3844363000-7f3844563000 ---p 00007000 fc:00 3020782                    /usr/lib/libOpenIPMIutils.so.0.0.1
  1960:20160915:063034.765 7f3844563000-7f3844564000 r--p 00007000 fc:00 3020782                    /usr/lib/libOpenIPMIutils.so.0.0.1
  1960:20160915:063034.765 7f3844564000-7f3844565000 rw-p 00008000 fc:00 3020782                    /usr/lib/libOpenIPMIutils.so.0.0.1
  1960:20160915:063034.765 7f3844565000-7f38445e1000 r-xp 00000000 fc:00 5636245                    /lib/x86_64-linux-gnu/libgcrypt.so.11.8.2
  1960:20160915:063034.765 7f38445e1000-7f38447e1000 ---p 0007c000 fc:00 5636245                    /lib/x86_64-linux-gnu/libgcrypt.so.11.8.2
  1960:20160915:063034.765 7f38447e1000-7f38447e2000 r--p 0007c000 fc:00 5636245                    /lib/x86_64-linux-gnu/libgcrypt.so.11.8.2
  1960:20160915:063034.765 7f38447e2000-7f38447e5000 rw-p 0007d000 fc:00 5636245                    /lib/x86_64-linux-gnu/libgcrypt.so.11.8.2
  1960:20160915:063034.765 7f38447e5000-7f38447ee000 r-xp 00000000 fc:00 3020772                    /usr/lib/x86_64-linux-gnu/libltdl.so.7.3.0
  1960:20160915:063034.765 7f38447ee000-7f38449ed000 ---p 00009000 fc:00 3020772                    /usr/lib/x86_64-linux-gnu/libltdl.so.7.3.0
  1960:20160915:063034.765 7f38449ed000-7f38449ee000 r--p 00008000 fc:00 3020772                    /usr/lib/x86_64-linux-gnu/libltdl.so.7.3.0
  1960:20160915:063034.765 7f38449ee000-7f38449ef000 rw-p 00009000 fc:00 3020772                    /usr/lib/x86_64-linux-gnu/libltdl.so.7.3.0
  1960:20160915:063034.765 7f38449ef000-7f3844a10000 r-xp 00000000 fc:00 5636302                    /lib/x86_64-linux-gnu/liblzma.so.5.0.0
  1960:20160915:063034.765 7f3844a10000-7f3844c0f000 ---p 00021000 fc:00 5636302                    /lib/x86_64-linux-gnu/liblzma.so.5.0.0
  1960:20160915:063034.765 7f3844c0f000-7f3844c10000 r--p 00020000 fc:00 5636302                    /lib/x86_64-linux-gnu/liblzma.so.5.0.0
  1960:20160915:063034.765 7f3844c10000-7f3844c11000 rw-p 00021000 fc:00 5636302                    /lib/x86_64-linux-gnu/liblzma.so.5.0.0
  1960:20160915:063034.765 7f3844c11000-7f3844cc8000 r-xp 00000000 fc:00 3017296                    /usr/lib/x86_64-linux-gnu/libgnutls.so.26.22.6
  1960:20160915:063034.765 7f3844cc8000-7f3844ec7000 ---p 000b7000 fc:00 3017296                    /usr/lib/x86_64-linux-gnu/libgnutls.so.26.22.6
  1960:20160915:063034.765 7f3844ec7000-7f3844ecd000 r--p 000b6000 fc:00 3017296                    /usr/lib/x86_64-linux-gnu/libgnutls.so.26.22.6
  1960:20160915:063034.765 7f3844ecd000-7f3844ece000 rw-p 000bc000 fc:00 3017296                    /usr/lib/x86_64-linux-gnu/libgnutls.so.26.22.6
  1960:20160915:063034.765 7f3844ece000-7f3844ecf000 rw-p 00000000 00:00 0 
  1960:20160915:063034.765 7f3844ecf000-7f3844ee8000 r-xp 00000000 fc:00 5640203                    /lib/x86_64-linux-gnu/libpthread-2.19.so
  1960:20160915:063034.765 7f3844ee8000-7f38450e7000 ---p 00019000 fc:00 5640203                    /lib/x86_64-linux-gnu/libpthread-2.19.so
  1960:20160915:063034.765 7f38450e7000-7f38450e8000 r--p 00018000 fc:00 5640203                    /lib/x86_64-linux-gnu/libpthread-2.19.so
  1960:20160915:063034.765 7f38450e8000-7f38450e9000 rw-p 00019000 fc:00 5640203                    /lib/x86_64-linux-gnu/libpthread-2.19.so
  1960:20160915:063034.765 7f38450e9000-7f38450ed000 rw-p 00000000 00:00 0 
  1960:20160915:063034.765 7f38450ed000-7f3845105000 r-xp 00000000 fc:00 5636383                    /lib/x86_64-linux-gnu/libz.so.1.2.8
  1960:20160915:063034.765 7f3845105000-7f3845304000 ---p 00018000 fc:00 5636383                    /lib/x86_64-linux-gnu/libz.so.1.2.8
  1960:20160915:063034.765 7f3845304000-7f3845305000 r--p 00017000 fc:00 5636383                    /lib/x86_64-linux-gnu/libz.so.1.2.8
  1960:20160915:063034.766 7f3845305000-7f3845306000 rw-p 00018000 fc:00 5636383                    /lib/x86_64-linux-gnu/libz.so.1.2.8
  1960:20160915:063034.766 7f3845306000-7f38454c0000 r-xp 00000000 fc:00 5640211                    /lib/x86_64-linux-gnu/libc-2.19.so
  1960:20160915:063034.766 7f38454c0000-7f38456c0000 ---p 001ba000 fc:00 5640211                    /lib/x86_64-linux-gnu/libc-2.19.so
  1960:20160915:063034.766 7f38456c0000-7f38456c4000 r--p 001ba000 fc:00 5640211                    /lib/x86_64-linux-gnu/libc-2.19.so
  1960:20160915:063034.766 7f38456c4000-7f38456c6000 rw-p 001be000 fc:00 5640211                    /lib/x86_64-linux-gnu/libc-2.19.so
  1960:20160915:063034.766 7f38456c6000-7f38456cb000 rw-p 00000000 00:00 0 
  1960:20160915:063034.766 7f38456cb000-7f38456e2000 r-xp 00000000 fc:00 5640194                    /lib/x86_64-linux-gnu/libresolv-2.19.so
  1960:20160915:063034.766 7f38456e2000-7f38458e2000 ---p 00017000 fc:00 5640194                    /lib/x86_64-linux-gnu/libresolv-2.19.so
  1960:20160915:063034.766 7f38458e2000-7f38458e3000 r--p 00017000 fc:00 5640194                    /lib/x86_64-linux-gnu/libresolv-2.19.so
  1960:20160915:063034.766 7f38458e3000-7f38458e4000 rw-p 00018000 fc:00 5640194                    /lib/x86_64-linux-gnu/libresolv-2.19.so
  1960:20160915:063034.766 7f38458e4000-7f38458e6000 rw-p 00000000 00:00 0 
  1960:20160915:063034.766 7f38458e6000-7f38458e9000 r-xp 00000000 fc:00 5640198                    /lib/x86_64-linux-gnu/libdl-2.19.so
  1960:20160915:063034.766 7f38458e9000-7f3845ae8000 ---p 00003000 fc:00 5640198                    /lib/x86_64-linux-gnu/libdl-2.19.so
  1960:20160915:063034.766 7f3845ae8000-7f3845ae9000 r--p 00002000 fc:00 5640198                    /lib/x86_64-linux-gnu/libdl-2.19.so
  1960:20160915:063034.766 7f3845ae9000-7f3845aea000 rw-p 00003000 fc:00 5640198                    /lib/x86_64-linux-gnu/libdl-2.19.so
  1960:20160915:063034.766 7f3845aea000-7f3845bef000 r-xp 00000000 fc:00 5640196                    /lib/x86_64-linux-gnu/libm-2.19.so
  1960:20160915:063034.766 7f3845bef000-7f3845dee000 ---p 00105000 fc:00 5640196                    /lib/x86_64-linux-gnu/libm-2.19.so
  1960:20160915:063034.766 7f3845dee000-7f3845def000 r--p 00104000 fc:00 5640196                    /lib/x86_64-linux-gnu/libm-2.19.so
  1960:20160915:063034.766 7f3845def000-7f3845df0000 rw-p 00105000 fc:00 5640196                    /lib/x86_64-linux-gnu/libm-2.19.so
  1960:20160915:063034.766 7f3845df0000-7f3845e54000 r-xp 00000000 fc:00 3015202                    /usr/lib/x86_64-linux-gnu/libcurl.so.4.3.0
  1960:20160915:063034.766 7f3845e54000-7f3846054000 ---p 00064000 fc:00 3015202                    /usr/lib/x86_64-linux-gnu/libcurl.so.4.3.0
  1960:20160915:063034.766 7f3846054000-7f3846056000 r--p 00064000 fc:00 3015202                    /usr/lib/x86_64-linux-gnu/libcurl.so.4.3.0
  1960:20160915:063034.766 7f3846056000-7f3846057000 rw-p 00066000 fc:00 3015202                    /usr/lib/x86_64-linux-gnu/libcurl.so.4.3.0
  1960:20160915:063034.766 7f3846057000-7f3846064000 r-xp 00000000 fc:00 3015185                    /usr/lib/x86_64-linux-gnu/liblber-2.4.so.2.8.3
  1960:20160915:063034.766 7f3846064000-7f3846264000 ---p 0000d000 fc:00 3015185                    /usr/lib/x86_64-linux-gnu/liblber-2.4.so.2.8.3
  1960:20160915:063034.766 7f3846264000-7f3846265000 r--p 0000d000 fc:00 3015185                    /usr/lib/x86_64-linux-gnu/liblber-2.4.so.2.8.3
  1960:20160915:063034.766 7f3846265000-7f3846266000 rw-p 0000e000 fc:00 3015185                    /usr/lib/x86_64-linux-gnu/liblber-2.4.so.2.8.3
  1960:20160915:063034.766 7f3846266000-7f38462b3000 r-xp 00000000 fc:00 3015186                    /usr/lib/x86_64-linux-gnu/libldap_r-2.4.so.2.8.3
  1960:20160915:063034.766 7f38462b3000-7f38464b2000 ---p 0004d000 fc:00 3015186                    /usr/lib/x86_64-linux-gnu/libldap_r-2.4.so.2.8.3
  1960:20160915:063034.766 7f38464b2000-7f38464b4000 r--p 0004c000 fc:00 3015186                    /usr/lib/x86_64-linux-gnu/libldap_r-2.4.so.2.8.3
  1960:20160915:063034.766 7f38464b4000-7f38464b5000 rw-p 0004e000 fc:00 3015186                    /usr/lib/x86_64-linux-gnu/libldap_r-2.4.so.2.8.3
  1960:20160915:063034.766 7f38464b5000-7f38464b7000 rw-p 00000000 00:00 0 
  1960:20160915:063034.766 7f38464b7000-7f384666a000 r-xp 00000000 fc:00 5636285                    /lib/x86_64-linux-gnu/libcrypto.so.1.0.0
  1960:20160915:063034.766 7f384666a000-7f3846869000 ---p 001b3000 fc:00 5636285                    /lib/x86_64-linux-gnu/libcrypto.so.1.0.0
  1960:20160915:063034.767 7f3846869000-7f3846884000 r--p 001b2000 fc:00 5636285                    /lib/x86_64-linux-gnu/libcrypto.so.1.0.0
  1960:20160915:063034.767 7f3846884000-7f384688f000 rw-p 001cd000 fc:00 5636285                    /lib/x86_64-linux-gnu/libcrypto.so.1.0.0
  1960:20160915:063034.767 7f384688f000-7f3846893000 rw-p 00000000 00:00 0 
  1960:20160915:063034.767 7f3846893000-7f38468e8000 r-xp 00000000 fc:00 5636396                    /lib/x86_64-linux-gnu/libssl.so.1.0.0
  1960:20160915:063034.767 7f38468e8000-7f3846ae8000 ---p 00055000 fc:00 5636396                    /lib/x86_64-linux-gnu/libssl.so.1.0.0
  1960:20160915:063034.767 7f3846ae8000-7f3846aeb000 r--p 00055000 fc:00 5636396                    /lib/x86_64-linux-gnu/libssl.so.1.0.0
  1960:20160915:063034.767 7f3846aeb000-7f3846af2000 rw-p 00058000 fc:00 5636396                    /lib/x86_64-linux-gnu/libssl.so.1.0.0
  1960:20160915:063034.767 7f3846af2000-7f3846af7000 r-xp 00000000 fc:00 3020785                    /usr/lib/libOpenIPMIposix.so.0.0.1
  1960:20160915:063034.767 7f3846af7000-7f3846cf6000 ---p 00005000 fc:00 3020785                    /usr/lib/libOpenIPMIposix.so.0.0.1
  1960:20160915:063034.767 7f3846cf6000-7f3846cf7000 r--p 00004000 fc:00 3020785                    /usr/lib/libOpenIPMIposix.so.0.0.1
  1960:20160915:063034.767 7f3846cf7000-7f3846cf8000 rw-p 00005000 fc:00 3020785                    /usr/lib/libOpenIPMIposix.so.0.0.1
  1960:20160915:063034.767 7f3846cf8000-7f3846de0000 r-xp 00000000 fc:00 3020786                    /usr/lib/libOpenIPMI.so.0.0.5
  1960:20160915:063034.767 7f3846de0000-7f3846fe0000 ---p 000e8000 fc:00 3020786                    /usr/lib/libOpenIPMI.so.0.0.5
  1960:20160915:063034.767 7f3846fe0000-7f3846ff6000 r--p 000e8000 fc:00 3020786                    /usr/lib/libOpenIPMI.so.0.0.5
  1960:20160915:063034.767 7f3846ff6000-7f3846ffc000 rw-p 000fe000 fc:00 3020786                    /usr/lib/libOpenIPMI.so.0.0.5
  1960:20160915:063034.767 7f3846ffc000-7f3847000000 rw-p 00000000 00:00 0 
  1960:20160915:063034.767 7f3847000000-7f3847027000 r-xp 00000000 fc:00 3015338                    /usr/lib/x86_64-linux-gnu/libssh2.so.1.0.1
  1960:20160915:063034.767 7f3847027000-7f3847226000 ---p 00027000 fc:00 3015338                    /usr/lib/x86_64-linux-gnu/libssh2.so.1.0.1
  1960:20160915:063034.767 7f3847226000-7f3847227000 r--p 00026000 fc:00 3015338                    /usr/lib/x86_64-linux-gnu/libssh2.so.1.0.1
  1960:20160915:063034.767 7f3847227000-7f3847228000 rw-p 00027000 fc:00 3015338                    /usr/lib/x86_64-linux-gnu/libssh2.so.1.0.1
  1960:20160915:063034.767 7f3847228000-7f38472c9000 r-xp 00000000 fc:00 3015309                    /usr/lib/x86_64-linux-gnu/libnetsnmp.so.30.0.2
  1960:20160915:063034.767 7f38472c9000-7f38474c9000 ---p 000a1000 fc:00 3015309                    /usr/lib/x86_64-linux-gnu/libnetsnmp.so.30.0.2
  1960:20160915:063034.767 7f38474c9000-7f38474ca000 r--p 000a1000 fc:00 3015309                    /usr/lib/x86_64-linux-gnu/libnetsnmp.so.30.0.2
  1960:20160915:063034.767 7f38474ca000-7f38474cc000 rw-p 000a2000 fc:00 3015309                    /usr/lib/x86_64-linux-gnu/libnetsnmp.so.30.0.2
  1960:20160915:063034.767 7f38474cc000-7f3847502000 rw-p 00000000 00:00 0 
  1960:20160915:063034.767 7f3847502000-7f3847561000 r-xp 00000000 fc:00 3020775                    /usr/lib/x86_64-linux-gnu/libodbc.so.1.0.0
  1960:20160915:063034.767 7f3847561000-7f3847760000 ---p 0005f000 fc:00 3020775                    /usr/lib/x86_64-linux-gnu/libodbc.so.1.0.0
  1960:20160915:063034.767 7f3847760000-7f3847761000 r--p 0005e000 fc:00 3020775                    /usr/lib/x86_64-linux-gnu/libodbc.so.1.0.0
  1960:20160915:063034.767 7f3847761000-7f3847768000 rw-p 0005f000 fc:00 3020775                    /usr/lib/x86_64-linux-gnu/libodbc.so.1.0.0
  1960:20160915:063034.767 7f3847768000-7f3847769000 rw-p 00000000 00:00 0 
  1960:20160915:063034.767 7f3847769000-7f38478c5000 r-xp 00000000 fc:00 3020770                    /usr/lib/x86_64-linux-gnu/libxml2.so.2.9.1
  1960:20160915:063034.767 7f38478c5000-7f3847ac5000 ---p 0015c000 fc:00 3020770                    /usr/lib/x86_64-linux-gnu/libxml2.so.2.9.1
  1960:20160915:063034.767 7f3847ac5000-7f3847acd000 r--p 0015c000 fc:00 3020770                    /usr/lib/x86_64-linux-gnu/libxml2.so.2.9.1
  1960:20160915:063034.768 7f3847acd000-7f3847acf000 rw-p 00164000 fc:00 3020770                    /usr/lib/x86_64-linux-gnu/libxml2.so.2.9.1
  1960:20160915:063034.768 7f3847acf000-7f3847ad0000 rw-p 00000000 00:00 0 
  1960:20160915:063034.768 7f3847ad0000-7f3847adc000 r-xp 00000000 fc:00 3020780                    /usr/lib/libiksemel.so.3.0.0
  1960:20160915:063034.768 7f3847adc000-7f3847cdc000 ---p 0000c000 fc:00 3020780                    /usr/lib/libiksemel.so.3.0.0
  1960:20160915:063034.768 7f3847cdc000-7f3847cdd000 r--p 0000c000 fc:00 3020780                    /usr/lib/libiksemel.so.3.0.0
  1960:20160915:063034.768 7f3847cdd000-7f3847cde000 rw-p 0000d000 fc:00 3020780                    /usr/lib/libiksemel.so.3.0.0
  1960:20160915:063034.768 7f3847cde000-7f3847f8e000 r-xp 00000000 fc:00 3015236                    /usr/lib/x86_64-linux-gnu/libmysqlclient.so.18.0.0
  1960:20160915:063034.768 7f3847f8e000-7f384818e000 ---p 002b0000 fc:00 3015236                    /usr/lib/x86_64-linux-gnu/libmysqlclient.so.18.0.0
  1960:20160915:063034.768 7f384818e000-7f3848193000 r--p 002b0000 fc:00 3015236                    /usr/lib/x86_64-linux-gnu/libmysqlclient.so.18.0.0
  1960:20160915:063034.768 7f3848193000-7f3848211000 rw-p 002b5000 fc:00 3015236                    /usr/lib/x86_64-linux-gnu/libmysqlclient.so.18.0.0
  1960:20160915:063034.768 7f3848211000-7f3848216000 rw-p 00000000 00:00 0 
  1960:20160915:063034.768 7f3848216000-7f3848239000 r-xp 00000000 fc:00 5640204                    /lib/x86_64-linux-gnu/ld-2.19.so
  1960:20160915:063034.768 7f38483fe000-7f384841a000 rw-s 00000000 00:05 4653062                    /SYSV53000018 (deleted)
  1960:20160915:063034.768 7f384841a000-7f3848431000 rw-p 00000000 00:00 0 
  1960:20160915:063034.768 7f3848434000-7f3848435000 rw-p 00000000 00:00 0 
  1960:20160915:063034.768 7f3848435000-7f3848436000 rw-p 00000000 00:00 0 
  1960:20160915:063034.768 7f3848436000-7f3848438000 rw-p 00000000 00:00 0 
  1960:20160915:063034.768 7f3848438000-7f3848439000 r--p 00022000 fc:00 5640204                    /lib/x86_64-linux-gnu/ld-2.19.so
  1960:20160915:063034.768 7f3848439000-7f384843a000 rw-p 00023000 fc:00 5640204                    /lib/x86_64-linux-gnu/ld-2.19.so
  1960:20160915:063034.768 7f384843a000-7f384843b000 rw-p 00000000 00:00 0 
  1960:20160915:063034.768 7ffee074d000-7ffee09f6000 rw-p 00000000 00:00 0                          [stack]
  1960:20160915:063034.768 7ffee09f9000-7ffee09fb000 r--p 00000000 00:00 0                          [vvar]
  1960:20160915:063034.768 7ffee09fb000-7ffee09fd000 r-xp 00000000 00:00 0                          [vdso]
  1960:20160915:063034.768 ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0                  [vsyscall]
  1960:20160915:063034.768 ================================
  1960:20160915:063034.768 Please consider attaching a disassembly listing to your bug report.
  1960:20160915:063034.768 This listing can be produced with, e.g., objdump -DSswx zabbix_server.
  1960:20160915:063034.768 ================================

So it crashes in delete_inactive_ipmi_hosts() and close_inactive_host().

Comment by Aleksandrs Saveljevs [ 2016 Sep 20 ]

The root cause seems to be the same as ZBX-10940.

Comment by Andris Mednis [ 2016 Nov 11 ]
  1960:20160915:063034.681 In delete_inactive_ipmi_hosts()
  1960:20160915:063034.681 In close_inactive_host(): 192.168.1.137
  1960:20160915:063034.682 EINF: 1(f.f)(m,0) sdr.c(handle_sdr_info): IPMI Error getting SDR info: ff
  1960:20160915:063034.682 In sensor_change() phost:0xcc6210 host:'[192.168.1.250]:623'
  1960:20160915:063034.683 In delete_ipmi_sensor() phost:0xcc6210 psensor:0xc94730
  1960:20160915:063034.684 sensor 'System Temp@[192.168.1.250]:623' deleted
....
  1960:20160915:063034.721 In delete_ipmi_control() phost:0xcc6210 pcontrol:0xcc05c0
  1960:20160915:063034.721 control 'reset@[192.168.1.250]:623' deleted
  1960:20160915:063034.721 End of delete_ipmi_control()
  1960:20160915:063034.722 End of control_change()
  1960:20160915:063034.722 In entity_change() phost:0xcc6210 host:'[192.168.1.250]:623'
  1960:20160915:063034.722 End of entity_change()
  1960:20160915:063034.722 In domain_closed() phost:0xc85460 host:'[192.168.1.137]:623'
  1960:20160915:063034.722 End of domain_closed()
  1960:20160915:063034.723 Got signal [signal:11(SIGSEGV),reason:2,refaddr:0x

Strange thing is that close_inactive_host() is called for host 192.168.1.137, but sensors and controls are deleted for 192.168.1.250. Seems a severe error.

Edit: Investigated and solution proposed in development branch svn://svn.zabbix.com/branches/dev/ZBX-10983 r64208.

Comment by Mohannad [ 2016 Nov 11 ]

Maybe part of the log is missing ? because both of these hosts should be monitored through IPMI, and is there any progress in solving this issue ?

Comment by Andris Mednis [ 2016 Nov 14 ]

Thanks for your logs - they are good and helpful ! I'm investigating this together with ZBX-10940, but cannot yet find what causes it. It seems a rare corner-case. Not every communication timeout causes crash. You have a lot of IPMI pollers and each of them scans and polls IPMI hosts independently. There are IPMI hosts which cannot handle multiple parallel sessions and it is better to use StartIPMIPollers=1 with them if possible.

Comment by Andris Mednis [ 2016 Dec 09 ]

Linked as a duplicate of ZBX-10983.

ZBX-10940 is a different crash.

Comment by Andris Mednis [ 2016 Dec 16 ]

I propose to close this bug as a duplicate of ZBX-10983. The ZBX-10983 fixes a crash in IPMI poller and adds more checks and debug logging.

Comment by VOD Pty Ltd [ 2017 Jan 12 ]

Thanks I have set StartIPMIPollers to 1 and upgraded zabbix server to 3.2 and it seems to be more stable.





[ZBX-10940] *** buffer overflow detected ***: /usr/sbin/zabbix_server: unreachable poller #1 Created: 2016 Jun 24  Updated: 2017 Aug 28  Resolved: 2017 Aug 28

Status: Closed
Project: ZABBIX BUGS AND ISSUES
Component/s: Server (S)
Affects Version/s: 3.0.3
Fix Version/s: 3.4 (plan)

Type: Problem report Priority: Major
Reporter: Danilenko Evgeniy Assignee: Unassigned
Resolution: Incomplete Votes: 0
Labels: crash, ipmi
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

Linux Ubuntu, kernel 3.13.0-79-generic


Attachments: Zip Archive objdump.zip     Zip Archive zabbix_server.log.zip     Zip Archive zabbix_server_with_openipmi_2.0.22.zip    
Issue Links:
Duplicate
Sub-task
part of ZBX-11697 zabbix server randomly crashes Closed
Team: Team A
Sprint: Sprint 5, Sprint 6, Sprint 7, Sprint 8, Sprint 9, Sprint 10, Sprint 11, Sprint 12, Sprint 13, Sprint 14, Sprint 15
Story Points: 4

 Description   

application caught an error and stopped work:

*** buffer overflow detected ***: /usr/sbin/zabbix_server: unreachable poller #1 [got 0 values in 0.000003 sec, getting values] terminated
======= Backtrace: =========
/lib/x86_64-linux-gnu/libc.so.6(+0x7338f)[0x7f8d4e02838f]
/lib/x86_64-linux-gnu/libc.so.6(__fortify_fail+0x5c)[0x7f8d4e0bfc9c]
/lib/x86_64-linux-gnu/libc.so.6(+0x109b60)[0x7f8d4e0beb60]
/lib/x86_64-linux-gnu/libc.so.6(+0x10abe7)[0x7f8d4e0bfbe7]
/usr/lib/libOpenIPMIposix.so.0(sel_set_fd_read_handler+0x7d)[0x7f8d4f7a358d]
/usr/lib/libOpenIPMIposix.so.0(+0x15f2)[0x7f8d4f7a25f2]
/usr/lib/libOpenIPMI.so.0(+0x947c1)[0x7f8d4fa3b7c1]
/usr/lib/libOpenIPMI.so.0(+0x97e86)[0x7f8d4fa3ee86]
/usr/lib/libOpenIPMI.so.0(+0x9a1fb)[0x7f8d4fa411fb]
/usr/lib/libOpenIPMIposix.so.0(+0x20e1)[0x7f8d4f7a30e1]
/usr/lib/libOpenIPMIposix.so.0(sel_select+0x39)[0x7f8d4f7a4499]
/usr/lib/libOpenIPMIposix.so.0(+0x171c)[0x7f8d4f7a271c]
/usr/sbin/zabbix_server: unreachable poller #1 [got 0 values in 0.000003 sec, getting values](get_value_ipmi+0x253)[0x426923]
/usr/sbin/zabbix_server: unreachable poller #1 [got 0 values in 0.000003 sec, getting values][0x429819]
/usr/sbin/zabbix_server: unreachable poller #1 [got 0 values in 0.000003 sec, getting values](poller_thread+0xf0)[0x429a00]
/usr/sbin/zabbix_server: unreachable poller #1 [got 0 values in 0.000003 sec, getting values](zbx_thread_start+0x45)[0x47ebb5]
/usr/sbin/zabbix_server: unreachable poller #1 [got 0 values in 0.000003 sec, getting values](MAIN_ZABBIX_ENTRY+0x605)[0x41cfc5]
/usr/sbin/zabbix_server: unreachable poller #1 [got 0 values in 0.000003 sec, getting values](daemon_start+0x1ad)[0x47d8bd]
/usr/sbin/zabbix_server: unreachable poller #1 [got 0 values in 0.000003 sec, getting values](main+0x311)[0x417c71]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf5)[0x7f8d4dfd6ec5]
/usr/sbin/zabbix_server: unreachable poller #1 [got 0 values in 0.000003 sec, getting values][0x417efe]
======= Memory map: ========
00400000-0056b000 r-xp 00000000 09:01 23724739                           /usr/sbin/zabbix_server
0076b000-0076c000 r--p 0016b000 09:01 23724739                           /usr/sbin/zabbix_server
0076c000-00773000 rw-p 0016c000 09:01 23724739                           /usr/sbin/zabbix_server
00773000-00779000 rw-p 00000000 00:00 0 
00a78000-00a99000 rw-p 00000000 00:00 0                                  [heap]
00a99000-00ad3000 rw-p 00000000 00:00 0                                  [heap]
00ad3000-00b95000 rw-p 00000000 00:00 0                                  [heap]
7f8d40000000-7f8d40021000 rw-p 00000000 00:00 0 
7f8d40021000-7f8d44000000 ---p 00000000 00:00 0 
7f8d45cd3000-7f8d45cd8000 r-xp 00000000 09:01 13113357                   /lib/x86_64-linux-gnu/libnss_dns-2.19.so
7f8d45cd8000-7f8d45ed7000 ---p 00005000 09:01 13113357                   /lib/x86_64-linux-gnu/libnss_dns-2.19.so
7f8d45ed7000-7f8d45ed8000 r--p 00004000 09:01 13113357                   /lib/x86_64-linux-gnu/libnss_dns-2.19.so
7f8d45ed8000-7f8d45ed9000 rw-p 00005000 09:01 13113357                   /lib/x86_64-linux-gnu/libnss_dns-2.19.so
7f8d45ed9000-7f8d461a2000 r--p 00000000 09:01 26614076                   /usr/lib/locale/locale-archive
7f8d461a2000-7f8d461b8000 r-xp 00000000 09:01 13107264                   /lib/x86_64-linux-gnu/libgcc_s.so.1
7f8d461b8000-7f8d463b7000 ---p 00016000 09:01 13107264                   /lib/x86_64-linux-gnu/libgcc_s.so.1
7f8d463b7000-7f8d463b8000 rw-p 00015000 09:01 13107264                   /lib/x86_64-linux-gnu/libgcc_s.so.1
7f8d463b8000-7f8d463b9000 ---p 00000000 00:00 0 
7f8d463b9000-7f8d46bb9000 rw-p 00000000 00:00 0 
7f8d46bb9000-7f8d473b9000 rw-s 00000000 00:04 1900550                    /SYSV7601001c (deleted)
7f8d473b9000-7f8d47a86000 rw-s 00000000 00:04 1802243                    /SYSV6701001c (deleted)
7f8d47a86000-7f8d47e86000 rw-s 00000000 00:04 1769474                    /SYSV7401001c (deleted)
7f8d47e86000-7f8d48286000 rw-s 00000000 00:04 1736705                    /SYSV4801001c (deleted)
7f8d48286000-7f8d49286000 rw-s 00000000 00:04 1703936                    /SYSV6801001c (deleted)
7f8d49286000-7f8d49291000 r-xp 00000000 09:01 13113363                   /lib/x86_64-linux-gnu/libnss_files-2.19.so
7f8d49291000-7f8d49490000 ---p 0000b000 09:01 13113363                   /lib/x86_64-linux-gnu/libnss_files-2.19.so
7f8d49490000-7f8d49491000 r--p 0000a000 09:01 13113363                   /lib/x86_64-linux-gnu/libnss_files-2.19.so
7f8d49491000-7f8d49492000 rw-p 0000b000 09:01 13113363                   /lib/x86_64-linux-gnu/libnss_files-2.19.so
7f8d49492000-7f8d4949d000 r-xp 00000000 09:01 13113355                   /lib/x86_64-linux-gnu/libnss_nis-2.19.so
7f8d4949d000-7f8d4969c000 ---p 0000b000 09:01 13113355                   /lib/x86_64-linux-gnu/libnss_nis-2.19.so
7f8d4969c000-7f8d4969d000 r--p 0000a000 09:01 13113355                   /lib/x86_64-linux-gnu/libnss_nis-2.19.so
7f8d4969d000-7f8d4969e000 rw-p 0000b000 09:01 13113355                   /lib/x86_64-linux-gnu/libnss_nis-2.19.so
7f8d4969e000-7f8d496b5000 r-xp 00000000 09:01 13113351                   /lib/x86_64-linux-gnu/libnsl-2.19.so
7f8d496b5000-7f8d498b4000 ---p 00017000 09:01 13113351                   /lib/x86_64-linux-gnu/libnsl-2.19.so
7f8d498b4000-7f8d498b5000 r--p 00016000 09:01 13113351                   /lib/x86_64-linux-gnu/libnsl-2.19.so
7f8d498b5000-7f8d498b6000 rw-p 00017000 09:01 13113351                   /lib/x86_64-linux-gnu/libnsl-2.19.so
7f8d498b6000-7f8d498b8000 rw-p 00000000 00:00 0 
7f8d498b8000-7f8d498c1000 r-xp 00000000 09:01 13113350                   /lib/x86_64-linux-gnu/libnss_compat-2.19.so
7f8d498c1000-7f8d49ac0000 ---p 00009000 09:01 13113350                   /lib/x86_64-linux-gnu/libnss_compat-2.19.so
7f8d49ac0000-7f8d49ac1000 r--p 00008000 09:01 13113350                   /lib/x86_64-linux-gnu/libnss_compat-2.19.so
7f8d49ac1000-7f8d49ac2000 rw-p 00009000 09:01 13113350                   /lib/x86_64-linux-gnu/libnss_compat-2.19.so
7f8d49ac2000-7f8d49ac4000 r-xp 00000000 09:01 13107381                   /lib/x86_64-linux-gnu/libkeyutils.so.1.4
7f8d49ac4000-7f8d49cc4000 ---p 00002000 09:01 13107381                   /lib/x86_64-linux-gnu/libkeyutils.so.1.4
7f8d49cc4000-7f8d49cc5000 r--p 00002000 09:01 13107381                   /lib/x86_64-linux-gnu/libkeyutils.so.1.4
7f8d49cc5000-7f8d49cc6000 rw-p 00003000 09:01 13107381                   /lib/x86_64-linux-gnu/libkeyutils.so.1.4
7f8d49cc6000-7f8d49ccf000 r-xp 00000000 09:01 13113348                   /lib/x86_64-linux-gnu/libcrypt-2.19.so
7f8d49ccf000-7f8d49ecf000 ---p 00009000 09:01 13113348                   /lib/x86_64-linux-gnu/libcrypt-2.19.so
7f8d49ecf000-7f8d49ed0000 r--p 00009000 09:01 13113348                   /lib/x86_64-linux-gnu/libcrypt-2.19.so
7f8d49ed0000-7f8d49ed1000 rw-p 0000a000 09:01 13113348                   /lib/x86_64-linux-gnu/libcrypt-2.19.so
7f8d49ed1000-7f8d49eff000 rw-p 00000000 00:00 0 
7f8d49eff000-7f8d49fb3000 r-xp 00000000 09:01 24906769                   /usr/lib/x86_64-linux-gnu/libsqlite3.so.0.8.6
7f8d49fb3000-7f8d4a1b3000 ---p 000b4000 09:01 24906769                   /usr/lib/x86_64-linux-gnu/libsqlite3.so.0.8.6
7f8d4a1b3000-7f8d4a1b5000 r--p 000b4000 09:01 24906769                   /usr/lib/x86_64-linux-gnu/libsqlite3.so.0.8.6
7f8d4a1b5000-7f8d4a1b7000 rw-p 000b6000 09:01 24906769                   /usr/lib/x86_64-linux-gnu/libsqlite3.so.0.8.6
7f8d4a1b7000-7f8d4a1b8000 rw-p 00000000 00:00 0 
7f8d4a1b8000-7f8d4a1fd000 r-xp 00000000 09:01 24906929                   /usr/lib/x86_64-linux-gnu/libhx509.so.5.0.0
7f8d4a1fd000-7f8d4a3fc000 ---p 00045000 09:01 24906929                   /usr/lib/x86_64-linux-gnu/libhx509.so.5.0.0
7f8d4a3fc000-7f8d4a3fe000 r--p 00044000 09:01 24906929                   /usr/lib/x86_64-linux-gnu/libhx509.so.5.0.0
7f8d4a3fe000-7f8d4a400000 rw-p 00046000 09:01 24906929                   /usr/lib/x86_64-linux-gnu/libhx509.so.5.0.0
7f8d4a400000-7f8d4a401000 rw-p 00000000 00:00 0 
7f8d4a401000-7f8d4a40e000 r-xp 00000000 09:01 24906951                   /usr/lib/x86_64-linux-gnu/libheimbase.so.1.0.0
7f8d4a40e000-7f8d4a60d000 ---p 0000d000 09:01 24906951                   /usr/lib/x86_64-linux-gnu/libheimbase.so.1.0.0
7f8d4a60d000-7f8d4a60e000 r--p 0000c000 09:01 24906951                   /usr/lib/x86_64-linux-gnu/libheimbase.so.1.0.0
7f8d4a60e000-7f8d4a60f000 rw-p 0000d000 09:01 24906951                   /usr/lib/x86_64-linux-gnu/libheimbase.so.1.0.0
7f8d4a60f000-7f8d4a636000 r-xp 00000000 09:01 24906976                   /usr/lib/x86_64-linux-gnu/libwind.so.0.0.0
7f8d4a636000-7f8d4a836000 ---p 00027000 09:01 24906976                   /usr/lib/x86_64-linux-gnu/libwind.so.0.0.0
7f8d4a836000-7f8d4a837000 r--p 00027000 09:01 24906976                   /usr/lib/x86_64-linux-gnu/libwind.so.0.0.0
7f8d4a837000-7f8d4a838000 rw-p 00028000 09:01 24906976                   /usr/lib/x86_64-linux-gnu/libwind.so.0.0.0
7f8d4a838000-7f8d4a83f000 r-xp 00000000 09:01 24906747                   /usr/lib/x86_64-linux-gnu/libffi.so.6.0.1
7f8d4a83f000-7f8d4aa3e000 ---p 00007000 09:01 24906747                   /usr/lib/x86_64-linux-gnu/libffi.so.6.0.1
7f8d4aa3e000-7f8d4aa3f000 r--p 00006000 09:01 24906747                   /usr/lib/x86_64-linux-gnu/libffi.so.6.0.1
7f8d4aa3f000-7f8d4aa40000 rw-p 00007000 09:01 24906747                   /usr/lib/x86_64-linux-gnu/libffi.so.6.0.1
7f8d4aa40000-7f8d4aa4a000 r-xp 00000000 09:01 24906470                   /usr/lib/x86_64-linux-gnu/libkrb5support.so.0.1
7f8d4aa4a000-7f8d4ac49000 ---p 0000a000 09:01 24906470                   /usr/lib/x86_64-linux-gnu/libkrb5support.so.0.1
7f8d4ac49000-7f8d4ac4a000 r--p 00009000 09:01 24906470                   /usr/lib/x86_64-linux-gnu/libkrb5support.so.0.1
7f8d4ac4a000-7f8d4ac4b000 rw-p 0000a000 09:01 24906470                   /usr/lib/x86_64-linux-gnu/libkrb5support.so.0.1
7f8d4ac4b000-7f8d4ac77000 r-xp 00000000 09:01 24906557                   /usr/lib/x86_64-linux-gnu/libk5crypto.so.3.1
7f8d4ac77000-7f8d4ae76000 ---p 0002c000 09:01 24906557                   /usr/lib/x86_64-linux-gnu/libk5crypto.so.3.1
7f8d4ae76000-7f8d4ae78000 r--p 0002b000 09:01 24906557                   /usr/lib/x86_64-linux-gnu/libk5crypto.so.3.1
7f8d4ae78000-7f8d4ae79000 rw-p 0002d000 09:01 24906557                   /usr/lib/x86_64-linux-gnu/libk5crypto.so.3.1
7f8d4ae79000-7f8d4ae7a000 rw-p 00000000 00:00 0 
7f8d4ae7a000-7f8d4af36000 r-xp 00000000 09:01 24906746                   /usr/lib/x86_64-linux-gnu/libkrb5.so.3.3
7f8d4af36000-7f8d4b136000 ---p 000bc000 09:01 24906746                   /usr/lib/x86_64-linux-gnu/libkrb5.so.3.3
7f8d4b136000-7f8d4b143000 r--p 000bc000 09:01 24906746                   /usr/lib/x86_64-linux-gnu/libkrb5.so.3.3
7f8d4b143000-7f8d4b145000 rw-p 000c9000 09:01 24906746                   /usr/lib/x86_64-linux-gnu/libkrb5.so.3.3
7f8d4b145000-7f8d4b159000 r-xp 00000000 09:01 24906522                   /usr/lib/x86_64-linux-gnu/libroken.so.18.1.0
7f8d4b159000-7f8d4b358000 ---p 00014000 09:01 24906522                   /usr/lib/x86_64-linux-gnu/libroken.so.18.1.0
7f8d4b358000-7f8d4b359000 r--p 00013000 09:01 24906522                   /usr/lib/x86_64-linux-gnu/libroken.so.18.1.0
7f8d4b359000-7f8d4b35a000 rw-p 00014000 09:01 24906522                   /usr/lib/x86_64-linux-gnu/libroken.so.18.1.0
7f8d4b35a000-7f8d4b38a000 r-xp 00000000 09:01 24906672                   /usr/lib/x86_64-linux-gnu/libhcrypto.so.4.1.0
7f8d4b38a000-7f8d4b58a000 ---p 00030000 09:01 24906672                   /usr/lib/x86_64-linux-gnu/libhcrypto.so.4.1.0
7f8d4b58a000-7f8d4b58b000 r--p 00030000 09:01 24906672                   /usr/lib/x86_64-linux-gnu/libhcrypto.so.4.1.0
7f8d4b58b000-7f8d4b58c000 rw-p 00031000 09:01 24906672                   /usr/lib/x86_64-linux-gnu/libhcrypto.so.4.1.0
7f8d4b58c000-7f8d4b58d000 rw-p 00000000 00:00 0 
7f8d4b58d000-7f8d4b590000 r-xp 00000000 09:01 13107282                   /lib/x86_64-linux-gnu/libcom_err.so.2.1
7f8d4b590000-7f8d4b78f000 ---p 00003000 09:01 13107282                   /lib/x86_64-linux-gnu/libcom_err.so.2.1
7f8d4b78f000-7f8d4b790000 r--p 00002000 09:01 13107282                   /lib/x86_64-linux-gnu/libcom_err.so.2.1
7f8d4b790000-7f8d4b791000 rw-p 00003000 09:01 13107282                   /lib/x86_64-linux-gnu/libcom_err.so.2.1
7f8d4b791000-7f8d4b82e000 r-xp 00000000 09:01 24906552                   /usr/lib/x86_64-linux-gnu/libasn1.so.8.0.0
7f8d4b82e000-7f8d4ba2e000 ---p 0009d000 09:01 24906552                   /usr/lib/x86_64-linux-gnu/libasn1.so.8.0.0
7f8d4ba2e000-7f8d4ba2f000 r--p 0009d000 09:01 24906552                   /usr/lib/x86_64-linux-gnu/libasn1.so.8.0.0
7f8d4ba2f000-7f8d4ba32000 rw-p 0009e000 09:01 24906552                   /usr/lib/x86_64-linux-gnu/libasn1.so.8.0.0
7f8d4ba32000-7f8d4bab4000 r-xp 00000000 09:01 24906623                   /usr/lib/x86_64-linux-gnu/libkrb5.so.26.0.0
7f8d4bab4000-7f8d4bcb3000 ---p 00082000 09:01 24906623                   /usr/lib/x86_64-linux-gnu/libkrb5.so.26.0.0
7f8d4bcb3000-7f8d4bcb6000 r--p 00081000 09:01 24906623                   /usr/lib/x86_64-linux-gnu/libkrb5.so.26.0.0
7f8d4bcb6000-7f8d4bcb9000 rw-p 00084000 09:01 24906623                   /usr/lib/x86_64-linux-gnu/libkrb5.so.26.0.0
7f8d4bcb9000-7f8d4bcba000 rw-p 00000000 00:00 0 
7f8d4bcba000-7f8d4bcc2000 r-xp 00000000 09:01 24906630                   /usr/lib/x86_64-linux-gnu/libheimntlm.so.0.1.0
7f8d4bcc2000-7f8d4bec1000 ---p 00008000 09:01 24906630                   /usr/lib/x86_64-linux-gnu/libheimntlm.so.0.1.0
7f8d4bec1000-7f8d4bec2000 r--p 00007000 09:01 24906630                   /usr/lib/x86_64-linux-gnu/libheimntlm.so.0.1.0
7f8d4bec2000-7f8d4bec3000 rw-p 00008000 09:01 24906630                   /usr/lib/x86_64-linux-gnu/libheimntlm.so.0.1.0
7f8d4bec3000-7f8d4bec7000 r-xp 00000000 09:01 13107476                   /lib/x86_64-linux-gnu/libgpg-error.so.0.10.0
7f8d4bec7000-7f8d4c0c6000 ---p 00004000 09:01 13107476                   /lib/x86_64-linux-gnu/libgpg-error.so.0.10.0
7f8d4c0c6000-7f8d4c0c7000 r--p 00003000 09:01 13107476                   /lib/x86_64-linux-gnu/libgpg-error.so.0.10.0
7f8d4c0c7000-7f8d4c0c8000 rw-p 00004000 09:01 13107476                   /lib/x86_64-linux-gnu/libgpg-error.so.0.10.0
7f8d4c0c8000-7f8d4c103000 r-xp 00000000 09:01 24906895                   /usr/lib/x86_64-linux-gnu/libp11-kit.so.0.0.0
7f8d4c103000-7f8d4c302000 ---p 0003b000 09:01 24906895                   /usr/lib/x86_64-linux-gnu/libp11-kit.so.0.0.0
7f8d4c302000-7f8d4c308000 r--p 0003a000 09:01 24906895                   /usr/lib/x86_64-linux-gnu/libp11-kit.so.0.0.0
7f8d4c308000-7f8d4c30a000 rw-p 00040000 09:01 24906895                   /usr/lib/x86_64-linux-gnu/libp11-kit.so.0.0.0
7f8d4c30a000-7f8d4c31c000 r-xp 00000000 09:01 24906487                   /usr/lib/x86_64-linux-gnu/libtasn1.so.6.2.0
7f8d4c31c000-7f8d4c51c000 ---p 00012000 09:01 24906487                   /usr/lib/x86_64-linux-gnu/libtasn1.so.6.2.0
7f8d4c51c000-7f8d4c51d000 r--p 00012000 09:01 24906487                   /usr/lib/x86_64-linux-gnu/libtasn1.so.6.2.0
7f8d4c51d000-7f8d4c51e000 rw-p 00013000 09:01 24906487                   /usr/lib/x86_64-linux-gnu/libtasn1.so.6.2.0
7f8d4c51e000-7f8d4c562000 r-xp 00000000 09:01 24906684                   /usr/lib/x86_64-linux-gnu/libgssapi_krb5.so.2.2
7f8d4c562000-7f8d4c762000 ---p 00044000 09:01 24906684                   /usr/lib/x86_64-linux-gnu/libgssapi_krb5.so.2.2
7f8d4c762000-7f8d4c763000 r--p 00044000 09:01 24906684                   /usr/lib/x86_64-linux-gnu/libgssapi_krb5.so.2.2
7f8d4c763000-7f8d4c765000 rw-p 00045000 09:01 24906684                   /usr/lib/x86_64-linux-gnu/libgssapi_krb5.so.2.2
7f8d4c765000-7f8d4c77e000 r-xp 00000000 09:01 24906937                   /usr/lib/x86_64-linux-gnu/librtmp.so.0
7f8d4c77e000-7f8d4c97d000 ---p 00019000 09:01 24906937                   /usr/lib/x86_64-linux-gnu/librtmp.so.0
7f8d4c97d000-7f8d4c97e000 r--p 00018000 09:01 24906937                   /usr/lib/x86_64-linux-gnu/librtmp.so.0
7f8d4c97e000-7f8d4c97f000 rw-p 00019000 09:01 24906937                   /usr/lib/x86_64-linux-gnu/librtmp.so.0
7f8d4c97f000-7f8d4c9b0000 r-xp 00000000 09:01 24906826                   /usr/lib/x86_64-linux-gnu/libidn.so.11.6.11
7f8d4c9b0000-7f8d4cbb0000 ---p 00031000 09:01 24906826                   /usr/lib/x86_64-linux-gnu/libidn.so.11.6.11
7f8d4cbb0000-7f8d4cbb1000 r--p 00031000 09:01 24906826                   /usr/lib/x86_64-linux-gnu/libidn.so.11.6.11
7f8d4cbb1000-7f8d4cbb2000 rw-p 00032000 09:01 24906826                   /usr/lib/x86_64-linux-gnu/libidn.so.11.6.11
7f8d4cbb2000-7f8d4cbec000 r-xp 00000000 09:01 24906637                   /usr/lib/x86_64-linux-gnu/libgssapi.so.3.0.0
7f8d4cbec000-7f8d4cdec000 ---p 0003a000 09:01 24906637                   /usr/lib/x86_64-linux-gnu/libgssapi.so.3.0.0
7f8d4cdec000-7f8d4cded000 r--p 0003a000 09:01 24906637                   /usr/lib/x86_64-linux-gnu/libgssapi.so.3.0.0
7f8d4cded000-7f8d4cdef000 rw-p 0003b000 09:01 24906637                   /usr/lib/x86_64-linux-gnu/libgssapi.so.3.0.0
7f8d4cdef000-7f8d4cdf0000 rw-p 00000000 00:00 0 
7f8d4cdf0000-7f8d4ce09000 r-xp 00000000 09:01 24906688                   /usr/lib/x86_64-linux-gnu/libsasl2.so.2.0.25
7f8d4ce09000-7f8d4d009000 ---p 00019000 09:01 24906688                   /usr/lib/x86_64-linux-gnu/libsasl2.so.2.0.25
7f8d4d009000-7f8d4d00a000 r--p 00019000 09:01 24906688                   /usr/lib/x86_64-linux-gnu/libsasl2.so.2.0.25
7f8d4d00a000-7f8d4d00b000 rw-p 0001a000 09:01 24906688                   /usr/lib/x86_64-linux-gnu/libsasl2.so.2.0.25
7f8d4d00b000-7f8d4d012000 r-xp 00000000 09:01 23728619                   /usr/lib/libOpenIPMIutils.so.0.0.1
7f8d4d012000-7f8d4d212000 ---p 00007000 09:01 23728619                   /usr/lib/libOpenIPMIutils.so.0.0.1
7f8d4d212000-7f8d4d213000 r--p 00007000 09:01 23728619                   /usr/lib/libOpenIPMIutils.so.0.0.1
7f8d4d213000-7f8d4d214000 rw-p 00008000 09:01 23728619                   /usr/lib/libOpenIPMIutils.so.0.0.1
7f8d4d214000-7f8d4d290000 r-xp 00000000 09:01 13107291                   /lib/x86_64-linux-gnu/libgcrypt.so.11.8.2
7f8d4d290000-7f8d4d490000 ---p 0007c000 09:01 13107291                   /lib/x86_64-linux-gnu/libgcrypt.so.11.8.2
7f8d4d490000-7f8d4d491000 r--p 0007c000 09:01 13107291                   /lib/x86_64-linux-gnu/libgcrypt.so.11.8.2
7f8d4d491000-7f8d4d494000 rw-p 0007d000 09:01 13107291                   /lib/x86_64-linux-gnu/libgcrypt.so.11.8.2
7f8d4d494000-7f8d4d49d000 r-xp 00000000 09:01 24906431                   /usr/lib/x86_64-linux-gnu/libltdl.so.7.3.0
7f8d4d49d000-7f8d4d69c000 ---p 00009000 09:01 24906431                   /usr/lib/x86_64-linux-gnu/libltdl.so.7.3.0
7f8d4d69c000-7f8d4d69d000 r--p 00008000 09:01 24906431                   /usr/lib/x86_64-linux-gnu/libltdl.so.7.3.0
7f8d4d69d000-7f8d4d69e000 rw-p 00009000 09:01 24906431                   /usr/lib/x86_64-linux-gnu/libltdl.so.7.3.0
7f8d4d69e000-7f8d4d6bf000 r-xp 00000000 09:01 13107459                   /lib/x86_64-linux-gnu/liblzma.so.5.0.0
7f8d4d6bf000-7f8d4d8be000 ---p 00021000 09:01 13107459                   /lib/x86_64-linux-gnu/liblzma.so.5.0.0
7f8d4d8be000-7f8d4d8bf000 r--p 00020000 09:01 13107459                   /lib/x86_64-linux-gnu/liblzma.so.5.0.0
7f8d4d8bf000-7f8d4d8c0000 rw-p 00021000 09:01 13107459                   /lib/x86_64-linux-gnu/liblzma.so.5.0.0
7f8d4d8c0000-7f8d4d977000 r-xp 00000000 09:01 24906566                   /usr/lib/x86_64-linux-gnu/libgnutls.so.26.22.6
7f8d4d977000-7f8d4db76000 ---p 000b7000 09:01 24906566                   /usr/lib/x86_64-linux-gnu/libgnutls.so.26.22.6
7f8d4db76000-7f8d4db7c000 r--p 000b6000 09:01 24906566                   /usr/lib/x86_64-linux-gnu/libgnutls.so.26.22.6
7f8d4db7c000-7f8d4db7d000 rw-p 000bc000 09:01 24906566                   /usr/lib/x86_64-linux-gnu/libgnutls.so.26.22.6
7f8d4db7d000-7f8d4db7e000 rw-p 00000000 00:00 0 
7f8d4db7e000-7f8d4db97000 r-xp 00000000 09:01 13113352                   /lib/x86_64-linux-gnu/libpthread-2.19.so
7f8d4db97000-7f8d4dd96000 ---p 00019000 09:01 13113352                   /lib/x86_64-linux-gnu/libpthread-2.19.so
7f8d4dd96000-7f8d4dd97000 r--p 00018000 09:01 13113352                   /lib/x86_64-linux-gnu/libpthread-2.19.so
7f8d4dd97000-7f8d4dd98000 rw-p 00019000 09:01 13113352                   /lib/x86_64-linux-gnu/libpthread-2.19.so
7f8d4dd98000-7f8d4dd9c000 rw-p 00000000 00:00 0 
7f8d4dd9c000-7f8d4ddb4000 r-xp 00000000 09:01 13107267                   /lib/x86_64-linux-gnu/libz.so.1.2.8
7f8d4ddb4000-7f8d4dfb3000 ---p 00018000 09:01 13107267                   /lib/x86_64-linux-gnu/libz.so.1.2.8
7f8d4dfb3000-7f8d4dfb4000 r--p 00017000 09:01 13107267                   /lib/x86_64-linux-gnu/libz.so.1.2.8
7f8d4dfb4000-7f8d4dfb5000 rw-p 00018000 09:01 13107267                   /lib/x86_64-linux-gnu/libz.so.1.2.8
7f8d4dfb5000-7f8d4e170000 r-xp 00000000 09:01 13113360                   /lib/x86_64-linux-gnu/libc-2.19.so
7f8d4e170000-7f8d4e36f000 ---p 001bb000 09:01 13113360                   /lib/x86_64-linux-gnu/libc-2.19.so
7f8d4e36f000-7f8d4e373000 r--p 001ba000 09:01 13113360                   /lib/x86_64-linux-gnu/libc-2.19.so
7f8d4e373000-7f8d4e375000 rw-p 001be000 09:01 13113360                   /lib/x86_64-linux-gnu/libc-2.19.so
7f8d4e375000-7f8d4e37a000 rw-p 00000000 00:00 0 
7f8d4e37a000-7f8d4e391000 r-xp 00000000 09:01 13113343                   /lib/x86_64-linux-gnu/libresolv-2.19.so
7f8d4e391000-7f8d4e591000 ---p 00017000 09:01 13113343                   /lib/x86_64-linux-gnu/libresolv-2.19.so
7f8d4e591000-7f8d4e592000 r--p 00017000 09:01 13113343                   /lib/x86_64-linux-gnu/libresolv-2.19.so
7f8d4e592000-7f8d4e593000 rw-p 00018000 09:01 13113343                   /lib/x86_64-linux-gnu/libresolv-2.19.so
7f8d4e593000-7f8d4e595000 rw-p 00000000 00:00 0 
7f8d4e595000-7f8d4e598000 r-xp 00000000 09:01 13113347                   /lib/x86_64-linux-gnu/libdl-2.19.so
7f8d4e598000-7f8d4e797000 ---p 00003000 09:01 13113347                   /lib/x86_64-linux-gnu/libdl-2.19.so
7f8d4e797000-7f8d4e798000 r--p 00002000 09:01 13113347                   /lib/x86_64-linux-gnu/libdl-2.19.so
7f8d4e798000-7f8d4e799000 rw-p 00003000 09:01 13113347                   /lib/x86_64-linux-gnu/libdl-2.19.so
7f8d4e799000-7f8d4e89e000 r-xp 00000000 09:01 13113345                   /lib/x86_64-linux-gnu/libm-2.19.so
7f8d4e89e000-7f8d4ea9d000 ---p 00105000 09:01 13113345                   /lib/x86_64-linux-gnu/libm-2.19.so
7f8d4ea9d000-7f8d4ea9e000 r--p 00104000 09:01 13113345                   /lib/x86_64-linux-gnu/libm-2.19.so
7f8d4ea9e000-7f8d4ea9f000 rw-p 00105000 09:01 13113345                   /lib/x86_64-linux-gnu/libm-2.19.so
7f8d4ea9f000-7f8d4eb03000 r-xp 00000000 09:01 24906110                   /usr/lib/x86_64-linux-gnu/libcurl.so.4.3.0
7f8d4eb03000-7f8d4ed03000 ---p 00064000 09:01 24906110                   /usr/lib/x86_64-linux-gnu/libcurl.so.4.3.0
7f8d4ed03000-7f8d4ed05000 r--p 00064000 09:01 24906110                   /usr/lib/x86_64-linux-gnu/libcurl.so.4.3.0
7f8d4ed05000-7f8d4ed06000 rw-p 00066000 09:01 24906110                   /usr/lib/x86_64-linux-gnu/libcurl.so.4.3.0
7f8d4ed06000-7f8d4ed13000 r-xp 00000000 09:01 24907046                   /usr/lib/x86_64-linux-gnu/liblber-2.4.so.2.8.3
7f8d4ed13000-7f8d4ef13000 ---p 0000d000 09:01 24907046                   /usr/lib/x86_64-linux-gnu/liblber-2.4.so.2.8.3
7f8d4ef13000-7f8d4ef14000 r--p 0000d000 09:01 24907046                   /usr/lib/x86_64-linux-gnu/liblber-2.4.so.2.8.3
7f8d4ef14000-7f8d4ef15000 rw-p 0000e000 09:01 24907046                   /usr/lib/x86_64-linux-gnu/liblber-2.4.so.2.8.3
7f8d4ef15000-7f8d4ef62000 r-xp 00000000 09:01 24906718                   /usr/lib/x86_64-linux-gnu/libldap_r-2.4.so.2.8.3
7f8d4ef62000-7f8d4f161000 ---p 0004d000 09:01 24906718                   /usr/lib/x86_64-linux-gnu/libldap_r-2.4.so.2.8.3
7f8d4f161000-7f8d4f163000 r--p 0004c000 09:01 24906718                   /usr/lib/x86_64-linux-gnu/libldap_r-2.4.so.2.8.3
7f8d4f163000-7f8d4f164000 rw-p 0004e000 09:01 24906718                   /usr/lib/x86_64-linux-gnu/libldap_r-2.4.so.2.8.3
7f8d4f164000-7f8d4f166000 rw-p 00000000 00:00 0 
7f8d4f166000-7f8d4f319000 r-xp 00000000 09:01 13107388                   /lib/x86_64-linux-gnu/libcrypto.so.1.0.0
7f8d4f319000-7f8d4f518000 ---p 001b3000 09:01 13107388                   /lib/x86_64-linux-gnu/libcrypto.so.1.0.0
7f8d4f518000-7f8d4f533000 r--p 001b2000 09:01 13107388                   /lib/x86_64-linux-gnu/libcrypto.so.1.0.0
7f8d4f533000-7f8d4f53e000 rw-p 001cd000 09:01 13107388                   /lib/x86_64-linux-gnu/libcrypto.so.1.0.0
7f8d4f53e000-7f8d4f542000 rw-p 00000000 00:00 0 
7f8d4f542000-7f8d4f597000 r-xp 00000000 09:01 13107390                   /lib/x86_64-linux-gnu/libssl.so.1.0.0
7f8d4f597000-7f8d4f797000 ---p 00055000 09:01 13107390                   /lib/x86_64-linux-gnu/libssl.so.1.0.0
7f8d4f797000-7f8d4f79a000 r--p 00055000 09:01 13107390                   /lib/x86_64-linux-gnu/libssl.so.1.0.0
7f8d4f79a000-7f8d4f7a1000 rw-p 00058000 09:01 13107390                   /lib/x86_64-linux-gnu/libssl.so.1.0.0
7f8d4f7a1000-7f8d4f7a6000 r-xp 00000000 09:01 23728680                   /usr/lib/libOpenIPMIposix.so.0.0.1
7f8d4f7a6000-7f8d4f9a5000 ---p 00005000 09:01 23728680                   /usr/lib/libOpenIPMIposix.so.0.0.1
7f8d4f9a5000-7f8d4f9a6000 r--p 00004000 09:01 23728680                   /usr/lib/libOpenIPMIposix.so.0.0.1
7f8d4f9a6000-7f8d4f9a7000 rw-p 00005000 09:01 23728680                   /usr/lib/libOpenIPMIposix.so.0.0.1
7f8d4f9a7000-7f8d4fa8f000 r-xp 00000000 09:01 23728495                   /usr/lib/libOpenIPMI.so.0.0.5
7f8d4fa8f000-7f8d4fc8e000 ---p 000e8000 09:01 23728495                   /usr/lib/libOpenIPMI.so.0.0.5
7f8d4fc8e000-7f8d4fca4000 r--p 000e7000 09:01 23728495                   /usr/lib/libOpenIPMI.so.0.0.5
7f8d4fca4000-7f8d4fcaa000 rw-p 000fd000 09:01 23728495                   /usr/lib/libOpenIPMI.so.0.0.5
7f8d4fcaa000-7f8d4fcae000 rw-p 00000000 00:00 0 
7f8d4fcae000-7f8d4fcd5000 r-xp 00000000 09:01 24906645                   /usr/lib/x86_64-linux-gnu/libssh2.so.1.0.1
7f8d4fcd5000-7f8d4fed4000 ---p 00027000 09:01 24906645                   /usr/lib/x86_64-linux-gnu/libssh2.so.1.0.1
7f8d4fed4000-7f8d4fed5000 r--p 00026000 09:01 24906645                   /usr/lib/x86_64-linux-gnu/libssh2.so.1.0.1
7f8d4fed5000-7f8d4fed6000 rw-p 00027000 09:01 24906645                   /usr/lib/x86_64-linux-gnu/libssh2.so.1.0.1
7f8d4fed6000-7f8d4ff77000 r-xp 00000000 09:01 24906243                   /usr/lib/x86_64-linux-gnu/libnetsnmp.so.30.0.2
7f8d4ff77000-7f8d50177000 ---p 000a1000 09:01 24906243                   /usr/lib/x86_64-linux-gnu/libnetsnmp.so.30.0.2
7f8d50177000-7f8d50178000 r--p 000a1000 09:01 24906243                   /usr/lib/x86_64-linux-gnu/libnetsnmp.so.30.0.2
7f8d50178000-7f8d5017a000 rw-p 000a2000 09:01 24906243                   /usr/lib/x86_64-linux-gnu/libnetsnmp.so.30.0.2
7f8d5017a000-7f8d501b0000 rw-p 00000000 00:00 0 
7f8d501b0000-7f8d5020f000 r-xp 00000000 09:01 24906669                   /usr/lib/x86_64-linux-gnu/libodbc.so.1.0.0
7f8d5020f000-7f8d5040e000 ---p 0005f000 09:01 24906669                   /usr/lib/x86_64-linux-gnu/libodbc.so.1.0.0
7f8d5040e000-7f8d5040f000 r--p 0005e000 09:01 24906669                   /usr/lib/x86_64-linux-gnu/libodbc.so.1.0.0
7f8d5040f000-7f8d50416000 rw-p 0005f000 09:01 24906669                   /usr/lib/x86_64-linux-gnu/libodbc.so.1.0.0
7f8d50416000-7f8d50417000 rw-p 00000000 00:00 0 
7f8d50417000-7f8d50573000 r-xp 00000000 09:01 24906561                   /usr/lib/x86_64-linux-gnu/libxml2.so.2.9.1
7f8d50573000-7f8d50773000 ---p 0015c000 09:01 24906561                   /usr/lib/x86_64-linux-gnu/libxml2.so.2.9.1
7f8d50773000-7f8d5077b000 r--p 0015c000 09:01 24906561                   /usr/lib/x86_64-linux-gnu/libxml2.so.2.9.1
7f8d5077b000-7f8d5077d000 rw-p 00164000 09:01 24906561                   /usr/lib/x86_64-linux-gnu/libxml2.so.2.9.1
7f8d5077d000-7f8d5077e000 rw-p 00000000 00:00 0 
7f8d5077e000-7f8d5078a000 r-xp 00000000 09:01 23728627                   /usr/lib/libiksemel.so.3.0.0
7f8d5078a000-7f8d5098a000 ---p 0000c000 09:01 23728627                   /usr/lib/libiksemel.so.3.0.0
7f8d5098a000-7f8d5098b000 r--p 0000c000 09:01 23728627                   /usr/lib/libiksemel.so.3.0.0
7f8d5098b000-7f8d5098c000 rw-p 0000d000 09:01 23728627                   /usr/lib/libiksemel.so.3.0.0
7f8d5098c000-7f8d50c3c000 r-xp 00000000 09:01 24906163                   /usr/lib/x86_64-linux-gnu/libmysqlclient.so.18.0.0
7f8d50c3c000-7f8d50e3c000 ---p 002b0000 09:01 24906163                   /usr/lib/x86_64-linux-gnu/libmysqlclient.so.18.0.0
7f8d50e3c000-7f8d50e41000 r--p 002b0000 09:01 24906163                   /usr/lib/x86_64-linux-gnu/libmysqlclient.so.18.0.0
7f8d50e41000-7f8d50ebf000 rw-p 002b5000 09:01 24906163                   /usr/lib/x86_64-linux-gnu/libmysqlclient.so.18.0.0
7f8d50ebf000-7f8d50ec4000 rw-p 00000000 00:00 0 
7f8d50ec4000-7f8d50ee7000 r-xp 00000000 09:01 13113353                   /lib/x86_64-linux-gnu/ld-2.19.so
7f8d50f8b000-7f8d510bf000 rw-s 00000000 00:04 1835012                    /SYSV7301001c (deleted)
7f8d510bf000-7f8d510d6000 rw-p 00000000 00:00 0 
7f8d510df000-7f8d510e0000 rw-p 00000000 00:00 0 
7f8d510e0000-7f8d510e3000 rw-s 00000000 00:04 1867781                    /SYSV5301001c (deleted)
7f8d510e3000-7f8d510e4000 rw-p 00000000 00:00 0 
7f8d510e4000-7f8d510e6000 rw-p 00000000 00:00 0 
7f8d510e6000-7f8d510e7000 r--p 00022000 09:01 13113353                   /lib/x86_64-linux-gnu/ld-2.19.so
7f8d510e7000-7f8d510e8000 rw-p 00023000 09:01 13113353                   /lib/x86_64-linux-gnu/ld-2.19.so
7f8d510e8000-7f8d510e9000 rw-p 00000000 00:00 0 
7ffedb5bb000-7ffedb87d000 rw-p 00000000 00:00 0                          [stack]
7ffedb8a5000-7ffedb8a7000 r-xp 00000000 00:00 0                          [vdso]
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0                  [vsyscall]
 17207:20160624:180404.848 One child process died (PID:17228,exitcode/signal:6). Exiting ...
 17207:20160624:180406.884 syncing history data...
 17207:20160624:180407.041 syncing history data done
 17207:20160624:180407.042 syncing trends data...
 17207:20160624:180407.232 syncing trends data done
 17207:20160624:180407.232 Zabbix Server stopped. Zabbix 3.0.3 (revision 60173).

What can be the cause of this issue?



 Comments   
Comment by Aleksandrs Saveljevs [ 2016 Jun 27 ]

Does the issue reproduce reliably? Did it start crashing after an upgrade? Would it be possible to attach a DebugLevel=4 log that shows what items Zabbix was trying to process before the crash? What OpenIPMI library version are you using?

Comment by Danilenko Evgeniy [ 2016 Jun 27 ]

This issue has been reproduced several times since zabbix upgraded to 3.0.3 from 2.2.2 version. unfortunately, DebugLevel=4 was disabled, so i can not attach this type of log, but i will enable it. If this issue happen again, i will attach log file.
About openipmi:

# dpkg -l | grep ipmi
ii  ipmitool                                        1.8.13-1ubuntu0.6                       amd64        utility for IPMI control with kernel driver or LAN interface
ii  libopenipmi0                                    2.0.18-0ubuntu7.1                       amd64        Intelligent Platform Management Interface - runtime
ii  openipmi                                        2.0.18-0ubuntu7.1                       amd64        Intelligent Platform Management Interface (for servers)
Comment by Danilenko Evgeniy [ 2016 Jun 29 ]

logs with DebugLevel=4

Comment by Danilenko Evgeniy [ 2016 Jun 29 ]

Hi, Aleksandr.

We had this crash again, but now i have logs with DebugLevel=4. i`ve just attached it!

Comment by Aleksandrs Saveljevs [ 2016 Jul 04 ]

Evgeniy, thank you for the log!

Since Zabbix server crashes in an unreachable poller, it means that the monitored IPMI device was unreachable at that time.

Does it crash every time an IPMI device becomes unreachable or just occasionally? Does it crash only on this particular device (192.168.16.7) or any other? Did you disable and then enable this IPMI host prior to the crash? If you upgrade the OpenIPM library, does it solve the problem?

In any case, if the problem is on Zabbix side, then it might have been introduced in ZBX-7847, because that seems to be the biggest difference between 2.2 and 3.0.

Comment by Danilenko Evgeniy [ 2016 Jul 04 ]

Hi, Alexander!
It crash occasionally, not every time an IPMI device becomes unreachable. Unfortunately, we monitor only one IPMI device (SuperMicro server). We experienced strange behavior of IPMI monitoring, it becomes unreachable from time to time in zabbix, but host is pingable and mtr to 623 port looks fine. To fix it i am changing Authentication algorithm form md5 to straight or vise versa and it becomes available. We have been experiencing this issue since 2.2 version, so i don`t know is it problem with zabbix or not, but with zabbix 2.2.2 we hadn`t this issue. I`ve updated OpenIPMI library to the highest version in ubuntu repository:

ii  ipmitool                                        1.8.13-1ubuntu0.6                       amd64        utility for IPMI control with kernel driver or LAN interface
ii  libopenipmi0                                    2.0.18-0ubuntu7.2                       amd64        Intelligent Platform Management Interface - runtime
ii  openipmi                                        2.0.18-0ubuntu7.2                       amd64        Intelligent Platform Management Interface (for servers)

Lets see what happen next...

Comment by Aleksandrs Saveljevs [ 2016 Jul 04 ]

How did you install Zabbix? Did you compile from sources or install from the repository? If repository, then which one: Ubuntu or Zabbix official?

Was OpenIPMI upgraded at the same time as Zabbix? Version 2.0.18 was released about 6 years ago. The latest version is 2.0.22, see https://sourceforge.net/projects/openipmi/files/OpenIPMI%202.0%20Library/ .

Comment by Danilenko Evgeniy [ 2016 Jul 04 ]

zabbix installed from zabbix repository:

#cat /etc/apt/sources.list.d/zabbix.list

deb http://repo.zabbix.com/zabbix/3.0/ubuntu trusty main
deb-src http://repo.zabbix.com/zabbix/3.0/ubuntu trusty main

OpenIPMI wasn`t upgraded with Zabbix. I used repository to upgrade OpenIPMI, and 2.0.18 is the latest one in Ubuntu repository. I will build it from sourceforgre.net if it is necessary to have latest version.

Comment by Danilenko Evgeniy [ 2016 Jul 05 ]

Compiled and installed to the latest version:

ii  libopenipmi0                                    2.0.22-1                                amd64        Intelligent Platform Management Interface - runtime
ii  openipmi                                        2.0.22-1                                amd64        Intelligent Platform Management Interface (for servers)
Comment by Aleksandrs Saveljevs [ 2016 Jul 11 ]

Potentially related issue: ZBX-10983.

Comment by Danilenko Evgeniy [ 2016 Jul 11 ]

new crash with openipmi library 2.0.22

Comment by Danilenko Evgeniy [ 2016 Jul 11 ]

I have just added log file with new one crash from zabbix server with up-to-date openipmi library. If I'm not mistaken, nothing new in it.

Comment by Aleksandrs Saveljevs [ 2016 Jul 11 ]

Thank you! Could you please also attach the disassembly listing of Zabbix server produced by objdump -DSswx zabbix_server?

Comment by Danilenko Evgeniy [ 2016 Jul 11 ]

object dump with keys: -DSswx

Comment by Aleksandrs Saveljevs [ 2016 Jul 12 ]

Thank you again! Here is a small annotated part of the disassembly. It belongs in function read_ipmi_sensor():

  4268ca:	83 f8 01             	cmp    $0x1,%eax                                case IPMI_EVENT_READING_TYPE_THRESHOLD:
  4268cd:	0f 85 7d 01 00 00    	jne    426a50 <get_value_ipmi+0x380>
  4268d3:	48 8b 7d 00          	mov    0x0(%rbp),%rdi
  4268d7:	48 89 da             	mov    %rbx,%rdx
  4268da:	be 50 5a 42 00       	mov    $0x425a50,%esi
  4268df:	e8 fc ef fe ff       	callq  4158e0 <ipmi_sensor_get_reading@plt>
  4268e4:	85 c0                	test   %eax,%eax
  4268e6:	41 89 c4             	mov    %eax,%r12d
  4268e9:	0f 85 f1 02 00 00    	jne    426be0 <get_value_ipmi+0x510>            if (0 != (ret = ...))
  4268ef:	44 8b 43 4c          	mov    0x4c(%rbx),%r8d
  4268f3:	48 c7 44 24 10 0a 00 00 00 	movq   $0xa,0x10(%rsp)                  tv.tv_sec = 10;
  4268fc:	4c 8d 64 24 10       	lea    0x10(%rsp),%r12
  426901:	48 c7 44 24 18 00 00 00 00 	movq   $0x0,0x18(%rsp)                  tv.tv_usec = 0;
  42690a:	45 85 c0             	test   %r8d,%r8d
  42690d:	75 1b                	jne    42692a <get_value_ipmi+0x25a>            while (0 == h->done)
  42690f:	90                   	nop
  426910:	48 8b 05 a9 be 34 00 	mov    0x34bea9(%rip),%rax        # 7727c0 <progname+0x98>
  426917:	4c 89 e6             	mov    %r12,%rsi
  42691a:	48 89 c7             	mov    %rax,%rdi
  42691d:	ff 90 d0 00 00 00    	callq  *0xd0(%rax)                              os_hnd->perform_one_op()

According to the last log, it crashes on the instruction just before 0x426923, which is a call to os_hnd->perform_one_op(), as per the disassembly above.

Comment by Danilenko Evgeniy [ 2016 Jul 12 ]

what does it mean? How can we fix it?

Comment by Aleksandrs Saveljevs [ 2016 Jul 12 ]

It means that it crashes inside OpenIPMI, but we are not sure yet where exactly is the problem and what a potential fix can be. This is still to be investigated.

Comment by Danilenko Evgeniy [ 2016 Jul 12 ]

ok, thanks! i am looking forward to solution for this issue....

Comment by Danilenko Evgeniy [ 2016 Sep 15 ]

Hello, Guys!
It`s been two month! Do you have any updates about this issue?

Comment by Andris Mednis [ 2016 Nov 03 ]
Zabbix server poller_thread
 \
  get_values()
   \
    get_value()
     \
      get_value_ipmi()
       \
        read_ipmi_sensor()
         \
          while (0 == h->done)
            os_hnd->perform_one_op(os_hnd, &tv);
----------------------------------------------------- libOpenIPMIposix.so
             \
              sel_select()
               \
                process_timers()
                 \
                  ?????
                   \
                    rsp_timeout_handler()
                     \
                      lan_put()
                       \
                        lan_cleanup()
                         \
                          cleanup_con()
                           \
                            release_lan_fd()
                             \
                              lan_os_hnd->remove_fd_to_wait_for(lan_os_hnd, item->fd_wait_id);
                               \
                                remove_fd()
                                 \
                                  sel_set_fd_read_handler(posix_sel, fd_data->fd, SEL_FD_HANDLER_DISABLED);
----------------------------------------------------- libc.so
                                   \
                                    FD_SET() or FD_CLR() with bad 'fd' (negative or >1023)
                                     \
                                      __fdelt_chk(): check "if (d < 0 || d >= FD_SETSIZE)" fails
                                       \
                                        __chk_fail()
                                         \
                                          __fortify_fail ("buffer overflow detected");
Comment by Andris Mednis [ 2016 Nov 11 ]

Hi, Evgeniy !

Can you tell what are your 'StartIPMIPollers' and 'StartPollersUnreachable' values ir zabbix server.conf ?

Comment by Danilenko Evgeniy [ 2016 Nov 11 ]

Hello, Andris!

From config file:

# Number of pre-forked instances of IPMI pollers
# Default value is 0
# This parameter must be between 0 and 255
StartIPMIPollers=1

# Number of pre-forked instances of pollers for unreachable hosts
# Default value is 1
# This parameter must be between 0 and 255
StartPollersUnreachable=1
Comment by Andris Mednis [ 2016 Nov 14 ]

Thanks! StartIPMIPollers=1 and StartPollersUnreachable=1 are best possible values currently. I'm investigating this together with ZBX-11169, but cannot yet find what causes it. It seems a rare corner-case. Not every communication timeout causes crash - so far it never crashed on my test machine, but definitely there is a bug hidden somewhere.

Comment by Andris Mednis [ 2016 Nov 15 ]

Note that ZBX-10940 reports crash in 'unreachable poller' process, where deleting inactive IPMI hosts (added in ZBX-7847) is not used. ZBX-10983 and ZBX-11169 report crashes in 'ipmi poller'.

Comment by Andris Mednis [ 2016 Dec 08 ]

Hi, danilenko !
Have you seen more crashes since July ?
I have solved crash ZBX-10983 in 'ipmi poller', fixed number of other potential bugs, but still cannot reproduce this crash (even after decoding its backtrace).

Comment by Danilenko Evgeniy [ 2016 Dec 09 ]

Hello Andris!
I disabled it in July and haven't enabled it yet!
I can enabled it and look what I will get?!

Comment by Andris Mednis [ 2016 Dec 12 ]

Hi, Evgeniy !
I think there is no point to crash your server again running the same code you have now. Your log files were helpful to discover other potential bugs. The fixes and improvements are in development branch svn://svn.zabbix.com/branches/dev/ZBX-10983. When it will be tested and merged then it would be good to test it in your environment.

Comment by Danilenko Evgeniy [ 2016 Dec 12 ]

Hello Andris!
so i am looking forward to your merged fixes!
thank you!

Comment by Andris Mednis [ 2016 Dec 16 ]

Today a fix for ZBX-10983 was merged into 3.0.7rc1, 3.2.3rc1 with many smaller fixes. When 3.0.7 comes out then it would be interesting to test IPMI again. I don't think this ZBX-10940 is fixed (I could not reproduce it) but at least we'll get better debug info.

Comment by Andris Mednis [ 2017 Jan 23 ]

ZBX-11697 seems the same crash but with ZBX-10983 fix and much better debug logs.

Comment by Andris Mednis [ 2017 Feb 02 ]

So far no luck - cannot reproduce this crash in unreachable poller.
Testing with Zabbix Appliance 3.2.3 (as in ZBX-11697) in virtual machine.





[ZBX-10008] incoming float values trimmed to integer for ipmi items Created: 2015 Oct 27  Updated: 2017 May 30  Resolved: 2015 Oct 27

Status: Closed
Project: ZABBIX BUGS AND ISSUES
Component/s: Server (S)
Affects Version/s: 3.0.0alpha3
Fix Version/s: None

Type: Incident report Priority: Minor
Reporter: richlv Assignee: Unassigned
Resolution: Duplicate Votes: 0
Labels: ipmi
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Duplicate
duplicates ZBX-4210 Simple check item returning float doe... Closed

 Description   

zabbix logic for incoming values and integer target items is as follows :

  • if an incoming value is an integer but we end up with a decimal part as a result of the internal calculations (multiplier, store value), we trim the decimal part and store the result;
  • if an incoming value is a decimal, the item turns unsupported

that's not the case with ipmi items - if an ipmi item has an incoming decimal value, it is silently trimmed and the result is stored.
instead, the item should turn unsupported






[ZBX-9590] Please increase size of ipmi_username Created: 2015 May 26  Updated: 2017 May 30

Status: Open
Project: ZABBIX BUGS AND ISSUES
Component/s: Installation (I)
Affects Version/s: 2.4.5
Fix Version/s: None

Type: Incident report Priority: Trivial
Reporter: Peteris Krisjanis Assignee: Unassigned
Resolution: Unresolved Votes: 1
Labels: dbpatches, ipmi, schema
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Description   

Current length of ipmi_username is 16, but for example sometimes username can be email address (in case of IPMI of loaned servers which are provided by ISP). It needs some considerable increase (50 chars could be enough).






[ZBX-9581] zabbix_server crashed with SIG11 while using IPv6 for IPMI Created: 2015 May 21  Updated: 2017 May 30  Resolved: 2016 Sep 20

Status: Closed
Project: ZABBIX BUGS AND ISSUES
Component/s: Server (S)
Affects Version/s: 2.4.5
Fix Version/s: None

Type: Incident report Priority: Minor
Reporter: Uwe Kiewel Assignee: Unassigned
Resolution: Cannot Reproduce Votes: 0
Labels: crash, ipmi
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

Centos 6.6
Linux 2.6.32-504.16.2.el6.x86_64 #1 SMP Wed Apr 22 06:48:29 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

Zabbix is installed from yum repo at repo.zabbix.com



 Description   

zabbix_server crashed with SIG11 while using IPv6 for IPMI:

4002:20150521:190217.780 Got signal [signal:11(SIGSEGV),reason:1,refaddr:0x7f43fc686bf8]. Crashing ...
  4002:20150521:190217.780 ====== Fatal information: ======
  4002:20150521:190217.780 Program counter: 0x7f2bfc41ec2e
  4002:20150521:190217.780 === Registers: ===
  4002:20150521:190217.780 r8      =         ffffffff =           4294967295 =           4294967295
  4002:20150521:190217.780 r9      =                1 =                    1 =                    1
  4002:20150521:190217.780 r10     =     7ffff44293b0 =      140737291391920 =      140737291391920
  4002:20150521:190217.780 r11     =     7f2bfaf5a610 =      139826870724112 =      139826870724112
  4002:20150521:190217.780 r12     =                1 =                    1 =                    1
  4002:20150521:190217.780 r13     =     7ffff44296c0 =      140737291392704 =      140737291392704
  4002:20150521:190217.780 r14     =     7ffff44296e0 =      140737291392736 =      140737291392736
  4002:20150521:190217.780 r15     =     7ffff4429930 =      140737291393328 =      140737291393328
  4002:20150521:190217.780 rdi     =           dd2c80 =             14494848 =             14494848
  4002:20150521:190217.780 rsi     =                0 =                    0 =                    0
  4002:20150521:190217.780 rbp     =         ffffff99 =           4294967193 =           4294967193
  4002:20150521:190217.780 rbx     =                0 =                    0 =                    0
  4002:20150521:190217.780 rdx     =     7f2bfc6875a0 =      139826895025568 =      139826895025568
  4002:20150521:190217.780 rax     =       17fffff658 =         103079212632 =         103079212632
  4002:20150521:190217.780 rcx     =                0 =                    0 =                    0
  4002:20150521:190217.780 rsp     =     7ffff4429630 =      140737291392560 =      140737291392560
  4002:20150521:190217.780 rip     =     7f2bfc41ec2e =      139826892500014 =      139826892500014
  4002:20150521:190217.780 efl     =            10246 =                66118 =                66118
  4002:20150521:190217.780 csgsfs  =               33 =                   51 =                   51
  4002:20150521:190217.780 err     =                4 =                    4 =                    4
  4002:20150521:190217.780 trapno  =                e =                   14 =                   14
  4002:20150521:190217.780 oldmask =                0 =                    0 =                    0
  4002:20150521:190217.780 cr2     =     7f43fc686bf8 =      139929974238200 =      139929974238200
  4002:20150521:190217.780 === Backtrace: ===
  4002:20150521:190217.781 14: zabbix_server: ipmi poller #1 [got 0 values in 0.000003 sec, getting values](print_fatal_info+0x280) [0x469fa0]
  4002:20150521:190217.781 13: zabbix_server: ipmi poller #1 [got 0 values in 0.000003 sec, getting values]() [0x46a6bc]
  4002:20150521:190217.781 12: /lib64/libc.so.6(+0x326a0) [0x7f2bfaebd6a0]
  4002:20150521:190217.781 11: /usr/lib64/libOpenIPMI.so.0(ipmi_lanp_setup_con+0x59e) [0x7f2bfc41ec2e]
  4002:20150521:190217.781 10: /usr/lib64/libOpenIPMI.so.0(ipmi_ip_setup_con+0xa4) [0x7f2bfc41f844]
  4002:20150521:190217.781 9: zabbix_server: ipmi poller #1 [got 0 values in 0.000003 sec, getting values]() [0x42113b]
  4002:20150521:190217.781 8: zabbix_server: ipmi poller #1 [got 0 values in 0.000003 sec, getting values](get_value_ipmi+0x8a) [0x422b6a]
  4002:20150521:190217.781 7: zabbix_server: ipmi poller #1 [got 0 values in 0.000003 sec, getting values]() [0x425ba1]
  4002:20150521:190217.781 6: zabbix_server: ipmi poller #1 [got 0 values in 0.000003 sec, getting values](poller_thread+0xf3) [0x426053]
  4002:20150521:190217.781 5: zabbix_server: ipmi poller #1 [got 0 values in 0.000003 sec, getting values](zbx_thread_start+0x64) [0x46ab64]
  4002:20150521:190217.781 4: zabbix_server: ipmi poller #1 [got 0 values in 0.000003 sec, getting values](MAIN_ZABBIX_ENTRY+0x3de) [0x418c3e]
  4002:20150521:190217.781 3: zabbix_server: ipmi poller #1 [got 0 values in 0.000003 sec, getting values](daemon_start+0x1a1) [0x468f91]
  4002:20150521:190217.781 2: zabbix_server: ipmi poller #1 [got 0 values in 0.000003 sec, getting values](main+0x2d3) [0x4192a3]
  4002:20150521:190217.781 1: /lib64/libc.so.6(__libc_start_main+0xfd) [0x7f2bfaea9d5d]
  4002:20150521:190217.781 0: zabbix_server: ipmi poller #1 [got 0 values in 0.000003 sec, getting values]() [0x4139d9]
  4002:20150521:190217.781 === Memory map: ===
  4002:20150521:190217.781 00400000-004fb000 r-xp 00000000 08:03 33163382                           /usr/sbin/zabbix_server_mysql
  4002:20150521:190217.781 006fb000-0074e000 rw-p 000fb000 08:03 33163382                           /usr/sbin/zabbix_server_mysql
  4002:20150521:190217.781 0074e000-00754000 rw-p 00000000 00:00 0 
  4002:20150521:190217.781 00dc7000-00de8000 rw-p 00000000 00:00 0 
  4002:20150521:190217.781 00de8000-00e21000 rw-p 00000000 00:00 0 
  4002:20150521:190217.781 7f2bf161b000-7f2bf1631000 r-xp 00000000 08:03 10485907                   /lib64/libgcc_s-4.4.7-20120601.so.1
  4002:20150521:190217.781 7f2bf1631000-7f2bf1830000 ---p 00016000 08:03 10485907                   /lib64/libgcc_s-4.4.7-20120601.so.1
  4002:20150521:190217.781 7f2bf1830000-7f2bf1831000 rw-p 00015000 08:03 10485907                   /lib64/libgcc_s-4.4.7-20120601.so.1
  4002:20150521:190217.781 7f2bf1831000-7f2bf183d000 r-xp 00000000 08:03 10485841                   /lib64/libnss_files-2.12.so
  4002:20150521:190217.781 7f2bf183d000-7f2bf1a3d000 ---p 0000c000 08:03 10485841                   /lib64/libnss_files-2.12.so
  4002:20150521:190217.781 7f2bf1a3d000-7f2bf1a3e000 r--p 0000c000 08:03 10485841                   /lib64/libnss_files-2.12.so
  4002:20150521:190217.781 7f2bf1a3e000-7f2bf1a3f000 rw-p 0000d000 08:03 10485841                   /lib64/libnss_files-2.12.so
  4002:20150521:190217.781 7f2bf1a3f000-7f2bf223f000 rw-s 00000000 00:04 2424839                    /SYSV76030518 (deleted)
  4002:20150521:190217.781 7f2bf223f000-7f2bf2a3f000 rw-s 00000000 00:04 2392070                    /SYSV77030518 (deleted)
  4002:20150521:190217.781 7f2bf2a3f000-7f2bf2f0c000 rw-s 00000000 00:04 2326532                    /SYSV73030518 (deleted)
  4002:20150521:190217.781 7f2bf2f0c000-7f2bf4a40000 rw-s 00000000 00:04 2293763                    /SYSV67030518 (deleted)
  4002:20150521:190217.781 7f2bf4a40000-7f2bf4e40000 rw-s 00000000 00:04 2260994                    /SYSV74030518 (deleted)
  4002:20150521:190217.781 7f2bf4e40000-7f2bf5e40000 rw-s 00000000 00:04 2228225                    /SYSV78030518 (deleted)
  4002:20150521:190217.781 7f2bf5e40000-7f2bf6640000 rw-s 00000000 00:04 2195456                    /SYSV68030518 (deleted)
  4002:20150521:190217.781 7f2bf6640000-7f2bf665d000 r-xp 00000000 08:03 10485904                   /lib64/libselinux.so.1
  4002:20150521:190217.781 7f2bf665d000-7f2bf685c000 ---p 0001d000 08:03 10485904                   /lib64/libselinux.so.1
  4002:20150521:190217.781 7f2bf685c000-7f2bf685d000 r--p 0001c000 08:03 10485904                   /lib64/libselinux.so.1
  4002:20150521:190217.781 7f2bf685d000-7f2bf685e000 rw-p 0001d000 08:03 10485904                   /lib64/libselinux.so.1
  4002:20150521:190217.781 7f2bf685e000-7f2bf685f000 rw-p 00000000 00:00 0 
  4002:20150521:190217.781 7f2bf685f000-7f2bf6861000 r-xp 00000000 08:03 10485908                   /lib64/libkeyutils.so.1.3
  4002:20150521:190217.782 7f2bf6861000-7f2bf6a60000 ---p 00002000 08:03 10485908                   /lib64/libkeyutils.so.1.3
  4002:20150521:190217.782 7f2bf6a60000-7f2bf6a61000 r--p 00001000 08:03 10485908                   /lib64/libkeyutils.so.1.3
  4002:20150521:190217.782 7f2bf6a61000-7f2bf6a62000 rw-p 00002000 08:03 10485908                   /lib64/libkeyutils.so.1.3
  4002:20150521:190217.782 7f2bf6a62000-7f2bf6a6c000 r-xp 00000000 08:03 10485945                   /lib64/libkrb5support.so.0.1
  4002:20150521:190217.782 7f2bf6a6c000-7f2bf6c6b000 ---p 0000a000 08:03 10485945                   /lib64/libkrb5support.so.0.1
  4002:20150521:190217.782 7f2bf6c6b000-7f2bf6c6c000 r--p 00009000 08:03 10485945                   /lib64/libkrb5support.so.0.1
  4002:20150521:190217.782 7f2bf6c6c000-7f2bf6c6d000 rw-p 0000a000 08:03 10485945                   /lib64/libkrb5support.so.0.1
  4002:20150521:190217.782 7f2bf6c6d000-7f2bf6c8a000 r-xp 00000000 08:03 10485776                   /lib64/libtinfo.so.5.7
  4002:20150521:190217.782 7f2bf6c8a000-7f2bf6e8a000 ---p 0001d000 08:03 10485776                   /lib64/libtinfo.so.5.7
  4002:20150521:190217.782 7f2bf6e8a000-7f2bf6e8e000 rw-p 0001d000 08:03 10485776                   /lib64/libtinfo.so.5.7
  4002:20150521:190217.782 7f2bf6e8e000-7f2bf6e9e000 r-xp 00000000 08:03 33163163                   /usr/lib64/libtasn1.so.3.1.6
  4002:20150521:190217.782 7f2bf6e9e000-7f2bf709d000 ---p 00010000 08:03 33163163                   /usr/lib64/libtasn1.so.3.1.6
  4002:20150521:190217.782 7f2bf709d000-7f2bf709e000 rw-p 0000f000 08:03 33163163                   /usr/lib64/libtasn1.so.3.1.6
  4002:20150521:190217.782 7f2bf709e000-7f2bf70a0000 r-xp 00000000 08:03 10485947                   /lib64/libfreebl3.so
  4002:20150521:190217.782 7f2bf70a0000-7f2bf729f000 ---p 00002000 08:03 10485947                   /lib64/libfreebl3.so
  4002:20150521:190217.782 7f2bf729f000-7f2bf72a0000 r--p 00001000 08:03 10485947                   /lib64/libfreebl3.so
  4002:20150521:190217.782 7f2bf72a0000-7f2bf72a1000 rw-p 00002000 08:03 10485947                   /lib64/libfreebl3.so
  4002:20150521:190217.782 7f2bf72a1000-7f2bf72a4000 r-xp 00000000 08:03 10485783                   /lib64/libcom_err.so.2.1
  4002:20150521:190217.782 7f2bf72a4000-7f2bf74a3000 ---p 00003000 08:03 10485783                   /lib64/libcom_err.so.2.1
  4002:20150521:190217.782 7f2bf74a3000-7f2bf74a4000 r--p 00002000 08:03 10485783                   /lib64/libcom_err.so.2.1
  4002:20150521:190217.782 7f2bf74a4000-7f2bf74a5000 rw-p 00003000 08:03 10485783                   /lib64/libcom_err.so.2.1
  4002:20150521:190217.782 7f2bf74a5000-7f2bf74ce000 r-xp 00000000 08:03 10485788                   /lib64/libk5crypto.so.3.1
  4002:20150521:190217.782 7f2bf74ce000-7f2bf76ce000 ---p 00029000 08:03 10485788                   /lib64/libk5crypto.so.3.1
  4002:20150521:190217.782 7f2bf76ce000-7f2bf76cf000 r--p 00029000 08:03 10485788                   /lib64/libk5crypto.so.3.1
  4002:20150521:190217.782 7f2bf76cf000-7f2bf76d0000 rw-p 0002a000 08:03 10485788                   /lib64/libk5crypto.so.3.1
  4002:20150521:190217.782 7f2bf76d0000-7f2bf76d1000 rw-p 00000000 00:00 0 
  4002:20150521:190217.782 7f2bf76d1000-7f2bf77ac000 r-xp 00000000 08:03 10485914                   /lib64/libkrb5.so.3.3
  4002:20150521:190217.782 7f2bf77ac000-7f2bf79ab000 ---p 000db000 08:03 10485914                   /lib64/libkrb5.so.3.3
  4002:20150521:190217.782 7f2bf79ab000-7f2bf79b5000 r--p 000da000 08:03 10485914                   /lib64/libkrb5.so.3.3
  4002:20150521:190217.782 7f2bf79b5000-7f2bf79b7000 rw-p 000e4000 08:03 10485914                   /lib64/libkrb5.so.3.3
  4002:20150521:190217.782 7f2bf79b7000-7f2bf79f8000 r-xp 00000000 08:03 10485874                   /lib64/libgssapi_krb5.so.2.2
  4002:20150521:190217.782 7f2bf79f8000-7f2bf7bf8000 ---p 00041000 08:03 10485874                   /lib64/libgssapi_krb5.so.2.2
  4002:20150521:190217.782 7f2bf7bf8000-7f2bf7bf9000 r--p 00041000 08:03 10485874                   /lib64/libgssapi_krb5.so.2.2
  4002:20150521:190217.782 7f2bf7bf9000-7f2bf7bfb000 rw-p 00042000 08:03 10485874                   /lib64/libgssapi_krb5.so.2.2
  4002:20150521:190217.782 7f2bf7bfb000-7f2bf7c2d000 r-xp 00000000 08:03 10485781                   /lib64/libidn.so.11.6.1
  4002:20150521:190217.782 7f2bf7c2d000-7f2bf7e2c000 ---p 00032000 08:03 10485781                   /lib64/libidn.so.11.6.1
  4002:20150521:190217.782 7f2bf7e2c000-7f2bf7e2d000 rw-p 00031000 08:03 10485781                   /lib64/libidn.so.11.6.1
  4002:20150521:190217.782 7f2bf7e2d000-7f2bf7e66000 r-xp 00000000 08:03 10485925                   /lib64/libnspr4.so
  4002:20150521:190217.782 7f2bf7e66000-7f2bf8066000 ---p 00039000 08:03 10485925                   /lib64/libnspr4.so
  4002:20150521:190217.782 7f2bf8066000-7f2bf8067000 r--p 00039000 08:03 10485925                   /lib64/libnspr4.so
  4002:20150521:190217.782 7f2bf8067000-7f2bf8069000 rw-p 0003a000 08:03 10485925                   /lib64/libnspr4.so
  4002:20150521:190217.782 7f2bf8069000-7f2bf806b000 rw-p 00000000 00:00 0 
  4002:20150521:190217.782 7f2bf806b000-7f2bf806f000 r-xp 00000000 08:03 10486231                   /lib64/libplc4.so
  4002:20150521:190217.782 7f2bf806f000-7f2bf826e000 ---p 00004000 08:03 10486231                   /lib64/libplc4.so
  4002:20150521:190217.782 7f2bf826e000-7f2bf826f000 r--p 00003000 08:03 10486231                   /lib64/libplc4.so
  4002:20150521:190217.782 7f2bf826f000-7f2bf8270000 rw-p 00004000 08:03 10486231                   /lib64/libplc4.so
  4002:20150521:190217.782 7f2bf8270000-7f2bf8273000 r-xp 00000000 08:03 10486232                   /lib64/libplds4.so
  4002:20150521:190217.782 7f2bf8273000-7f2bf8472000 ---p 00003000 08:03 10486232                   /lib64/libplds4.so
  4002:20150521:190217.782 7f2bf8472000-7f2bf8473000 r--p 00002000 08:03 10486232                   /lib64/libplds4.so
  4002:20150521:190217.782 7f2bf8473000-7f2bf8474000 rw-p 00003000 08:03 10486232                   /lib64/libplds4.so
  4002:20150521:190217.782 7f2bf8474000-7f2bf8499000 r-xp 00000000 08:03 33163287                   /usr/lib64/libnssutil3.so
  4002:20150521:190217.782 7f2bf8499000-7f2bf8699000 ---p 00025000 08:03 33163287                   /usr/lib64/libnssutil3.so
  4002:20150521:190217.783 7f2bf8699000-7f2bf869f000 r--p 00025000 08:03 33163287                   /usr/lib64/libnssutil3.so
  4002:20150521:190217.783 7f2bf869f000-7f2bf86a0000 rw-p 0002b000 08:03 33163287                   /usr/lib64/libnssutil3.so
  4002:20150521:190217.783 7f2bf86a0000-7f2bf87d7000 r-xp 00000000 08:03 33163201                   /usr/lib64/libnss3.so
  4002:20150521:190217.783 7f2bf87d7000-7f2bf89d6000 ---p 00137000 08:03 33163201                   /usr/lib64/libnss3.so
  4002:20150521:190217.783 7f2bf89d6000-7f2bf89db000 r--p 00136000 08:03 33163201                   /usr/lib64/libnss3.so
  4002:20150521:190217.783 7f2bf89db000-7f2bf89dd000 rw-p 0013b000 08:03 33163201                   /usr/lib64/libnss3.so
  4002:20150521:190217.783 7f2bf89dd000-7f2bf89df000 rw-p 00000000 00:00 0 
  4002:20150521:190217.783 7f2bf89df000-7f2bf8a07000 r-xp 00000000 08:03 33163214                   /usr/lib64/libsmime3.so
  4002:20150521:190217.783 7f2bf8a07000-7f2bf8c06000 ---p 00028000 08:03 33163214                   /usr/lib64/libsmime3.so
  4002:20150521:190217.783 7f2bf8c06000-7f2bf8c0a000 r--p 00027000 08:03 33163214                   /usr/lib64/libsmime3.so
  4002:20150521:190217.783 7f2bf8c0a000-7f2bf8c0b000 rw-p 0002b000 08:03 33163214                   /usr/lib64/libsmime3.so
  4002:20150521:190217.783 7f2bf8c0b000-7f2bf8c46000 r-xp 00000000 08:03 33163290                   /usr/lib64/libssl3.so
  4002:20150521:190217.783 7f2bf8c46000-7f2bf8e45000 ---p 0003b000 08:03 33163290                   /usr/lib64/libssl3.so
  4002:20150521:190217.783 7f2bf8e45000-7f2bf8e48000 r--p 0003a000 08:03 33163290                   /usr/lib64/libssl3.so
  4002:20150521:190217.783 7f2bf8e48000-7f2bf8e49000 rw-p 0003d000 08:03 33163290                   /usr/lib64/libssl3.so
  4002:20150521:190217.783 7f2bf8e49000-7f2bf8e4a000 rw-p 00000000 00:00 0 
  4002:20150521:190217.783 7f2bf8e4a000-7f2bf8e63000 r-xp 00000000 08:03 33163052                   /usr/lib64/libsasl2.so.2.0.23
  4002:20150521:190217.783 7f2bf8e63000-7f2bf9062000 ---p 00019000 08:03 33163052                   /usr/lib64/libsasl2.so.2.0.23
  4002:20150521:190217.783 7f2bf9062000-7f2bf9063000 r--p 00018000 08:03 33163052                   /usr/lib64/libsasl2.so.2.0.23
  4002:20150521:190217.783 7f2bf9063000-7f2bf9064000 rw-p 00019000 08:03 33163052                   /usr/lib64/libsasl2.so.2.0.23
  4002:20150521:190217.783 7f2bf9064000-7f2bf906a000 r-xp 00000000 08:03 33163141                   /usr/lib64/libgdbm.so.2.0.0
  4002:20150521:190217.783 7f2bf906a000-7f2bf9269000 ---p 00006000 08:03 33163141                   /usr/lib64/libgdbm.so.2.0.0
  4002:20150521:190217.783 7f2bf9269000-7f2bf926a000 rw-p 00005000 08:03 33163141                   /usr/lib64/libgdbm.so.2.0.0
  4002:20150521:190217.783 7f2bf926a000-7f2bf928c000 r-xp 00000000 08:03 10485765                   /lib64/libncurses.so.5.7
  4002:20150521:190217.783 7f2bf928c000-7f2bf948b000 ---p 00022000 08:03 10485765                   /lib64/libncurses.so.5.7
  4002:20150521:190217.783 7f2bf948b000-7f2bf948c000 rw-p 00021000 08:03 10485765                   /lib64/libncurses.so.5.7
  4002:20150521:190217.783 7f2bf948c000-7f2bf9493000 r-xp 00000000 08:03 33163302                   /usr/lib64/libOpenIPMIutils.so.0.0.1
  4002:20150521:190217.783 7f2bf9493000-7f2bf9693000 ---p 00007000 08:03 33163302                   /usr/lib64/libOpenIPMIutils.so.0.0.1
  4002:20150521:190217.783 7f2bf9693000-7f2bf9694000 rw-p 00007000 08:03 33163302                   /usr/lib64/libOpenIPMIutils.so.0.0.1
  4002:20150521:190217.783 7f2bf9694000-7f2bf96ab000 r-xp 00000000 08:03 10485832                   /lib64/libpthread-2.12.so
  4002:20150521:190217.783 7f2bf96ab000-7f2bf98ab000 ---p 00017000 08:03 10485832                   /lib64/libpthread-2.12.so
  4002:20150521:190217.783 7f2bf98ab000-7f2bf98ac000 r--p 00017000 08:03 10485832                   /lib64/libpthread-2.12.so
  4002:20150521:190217.783 7f2bf98ac000-7f2bf98ad000 rw-p 00018000 08:03 10485832                   /lib64/libpthread-2.12.so
  4002:20150521:190217.783 7f2bf98ad000-7f2bf98b1000 rw-p 00000000 00:00 0 
  4002:20150521:190217.783 7f2bf98b1000-7f2bf98ba000 r-xp 00000000 08:03 33163322                   /usr/lib64/libltdl.so.7.2.1
  4002:20150521:190217.783 7f2bf98ba000-7f2bf9ab9000 ---p 00009000 08:03 33163322                   /usr/lib64/libltdl.so.7.2.1
  4002:20150521:190217.783 7f2bf9ab9000-7f2bf9aba000 rw-p 00008000 08:03 33163322                   /usr/lib64/libltdl.so.7.2.1
  4002:20150521:190217.783 7f2bf9aba000-7f2bf9abd000 r-xp 00000000 08:03 10485937                   /lib64/libgpg-error.so.0.5.0
  4002:20150521:190217.783 7f2bf9abd000-7f2bf9cbc000 ---p 00003000 08:03 10485937                   /lib64/libgpg-error.so.0.5.0
  4002:20150521:190217.783 7f2bf9cbc000-7f2bf9cbd000 r--p 00002000 08:03 10485937                   /lib64/libgpg-error.so.0.5.0
  4002:20150521:190217.783 7f2bf9cbd000-7f2bf9cbe000 rw-p 00003000 08:03 10485937                   /lib64/libgpg-error.so.0.5.0
  4002:20150521:190217.783 7f2bf9cbe000-7f2bf9d30000 r-xp 00000000 08:03 10485814                   /lib64/libgcrypt.so.11.5.3
  4002:20150521:190217.783 7f2bf9d30000-7f2bf9f2f000 ---p 00072000 08:03 10485814                   /lib64/libgcrypt.so.11.5.3
  4002:20150521:190217.783 7f2bf9f2f000-7f2bf9f30000 r--p 00071000 08:03 10485814                   /lib64/libgcrypt.so.11.5.3
  4002:20150521:190217.783 7f2bf9f30000-7f2bf9f33000 rw-p 00072000 08:03 10485814                   /lib64/libgcrypt.so.11.5.3
  4002:20150521:190217.783 7f2bf9f33000-7f2bf9fcf000 r-xp 00000000 08:03 33163377                   /usr/lib64/libgnutls.so.26.14.12
  4002:20150521:190217.783 7f2bf9fcf000-7f2bfa1cf000 ---p 0009c000 08:03 33163377                   /usr/lib64/libgnutls.so.26.14.12
  4002:20150521:190217.783 7f2bfa1cf000-7f2bfa1d6000 rw-p 0009c000 08:03 33163377                   /usr/lib64/libgnutls.so.26.14.12
  4002:20150521:190217.783 7f2bfa1d6000-7f2bfa1eb000 r-xp 00000000 08:03 10485934                   /lib64/libz.so.1.2.3
  4002:20150521:190217.783 7f2bfa1eb000-7f2bfa3ea000 ---p 00015000 08:03 10485934                   /lib64/libz.so.1.2.3
  4002:20150521:190217.784 7f2bfa3ea000-7f2bfa3eb000 r--p 00014000 08:03 10485934                   /lib64/libz.so.1.2.3
  4002:20150521:190217.784 7f2bfa3eb000-7f2bfa3ec000 rw-p 00015000 08:03 10485934                   /lib64/libz.so.1.2.3
  4002:20150521:190217.784 7f2bfa3ec000-7f2bfa5a5000 r-xp 00000000 08:03 33163085                   /usr/lib64/libcrypto.so.1.0.1e
  4002:20150521:190217.784 7f2bfa5a5000-7f2bfa7a4000 ---p 001b9000 08:03 33163085                   /usr/lib64/libcrypto.so.1.0.1e
  4002:20150521:190217.784 7f2bfa7a4000-7f2bfa7bf000 r--p 001b8000 08:03 33163085                   /usr/lib64/libcrypto.so.1.0.1e
  4002:20150521:190217.784 7f2bfa7bf000-7f2bfa7cb000 rw-p 001d3000 08:03 33163085                   /usr/lib64/libcrypto.so.1.0.1e
  4002:20150521:190217.784 7f2bfa7cb000-7f2bfa7cf000 rw-p 00000000 00:00 0 
  4002:20150521:190217.784 7f2bfa7cf000-7f2bfa831000 r-xp 00000000 08:03 33163025                   /usr/lib64/libssl.so.1.0.1e
  4002:20150521:190217.784 7f2bfa831000-7f2bfaa30000 ---p 00062000 08:03 33163025                   /usr/lib64/libssl.so.1.0.1e
  4002:20150521:190217.784 7f2bfaa30000-7f2bfaa34000 r--p 00061000 08:03 33163025                   /usr/lib64/libssl.so.1.0.1e
  4002:20150521:190217.784 7f2bfaa34000-7f2bfaa3b000 rw-p 00065000 08:03 33163025                   /usr/lib64/libssl.so.1.0.1e
  4002:20150521:190217.784 7f2bfaa3b000-7f2bfaa51000 r-xp 00000000 08:03 10485822                   /lib64/libnsl-2.12.so
  4002:20150521:190217.784 7f2bfaa51000-7f2bfac50000 ---p 00016000 08:03 10485822                   /lib64/libnsl-2.12.so
  4002:20150521:190217.784 7f2bfac50000-7f2bfac51000 r--p 00015000 08:03 10485822                   /lib64/libnsl-2.12.so
  4002:20150521:190217.784 7f2bfac51000-7f2bfac52000 rw-p 00016000 08:03 10485822                   /lib64/libnsl-2.12.so
  4002:20150521:190217.784 7f2bfac52000-7f2bfac54000 rw-p 00000000 00:00 0 
  4002:20150521:190217.784 7f2bfac54000-7f2bfac5b000 r-xp 00000000 08:03 10485892                   /lib64/libcrypt-2.12.so
  4002:20150521:190217.784 7f2bfac5b000-7f2bfae5b000 ---p 00007000 08:03 10485892                   /lib64/libcrypt-2.12.so
  4002:20150521:190217.784 7f2bfae5b000-7f2bfae5c000 r--p 00007000 08:03 10485892                   /lib64/libcrypt-2.12.so
  4002:20150521:190217.784 7f2bfae5c000-7f2bfae5d000 rw-p 00008000 08:03 10485892                   /lib64/libcrypt-2.12.so
  4002:20150521:190217.784 7f2bfae5d000-7f2bfae8b000 rw-p 00000000 00:00 0 
  4002:20150521:190217.784 7f2bfae8b000-7f2bfb015000 r-xp 00000000 08:03 10485899                   /lib64/libc-2.12.so
  4002:20150521:190217.784 7f2bfb015000-7f2bfb215000 ---p 0018a000 08:03 10485899                   /lib64/libc-2.12.so
  4002:20150521:190217.784 7f2bfb215000-7f2bfb219000 r--p 0018a000 08:03 10485899                   /lib64/libc-2.12.so
  4002:20150521:190217.784 7f2bfb219000-7f2bfb21a000 rw-p 0018e000 08:03 10485899                   /lib64/libc-2.12.so
  4002:20150521:190217.784 7f2bfb21a000-7f2bfb21f000 rw-p 00000000 00:00 0 
  4002:20150521:190217.784 7f2bfb21f000-7f2bfb235000 r-xp 00000000 08:03 10485797                   /lib64/libresolv-2.12.so
  4002:20150521:190217.784 7f2bfb235000-7f2bfb435000 ---p 00016000 08:03 10485797                   /lib64/libresolv-2.12.so
  4002:20150521:190217.784 7f2bfb435000-7f2bfb436000 r--p 00016000 08:03 10485797                   /lib64/libresolv-2.12.so
  4002:20150521:190217.784 7f2bfb436000-7f2bfb437000 rw-p 00017000 08:03 10485797                   /lib64/libresolv-2.12.so
  4002:20150521:190217.784 7f2bfb437000-7f2bfb439000 rw-p 00000000 00:00 0 
  4002:20150521:190217.784 7f2bfb439000-7f2bfb440000 r-xp 00000000 08:03 10485932                   /lib64/librt-2.12.so
  4002:20150521:190217.784 7f2bfb440000-7f2bfb63f000 ---p 00007000 08:03 10485932                   /lib64/librt-2.12.so
  4002:20150521:190217.784 7f2bfb63f000-7f2bfb640000 r--p 00006000 08:03 10485932                   /lib64/librt-2.12.so
  4002:20150521:190217.784 7f2bfb640000-7f2bfb641000 rw-p 00007000 08:03 10485932                   /lib64/librt-2.12.so
  4002:20150521:190217.784 7f2bfb641000-7f2bfb643000 r-xp 00000000 08:03 10485793                   /lib64/libdl-2.12.so
  4002:20150521:190217.784 7f2bfb643000-7f2bfb843000 ---p 00002000 08:03 10485793                   /lib64/libdl-2.12.so
  4002:20150521:190217.784 7f2bfb843000-7f2bfb844000 r--p 00002000 08:03 10485793                   /lib64/libdl-2.12.so
  4002:20150521:190217.784 7f2bfb844000-7f2bfb845000 rw-p 00003000 08:03 10485793                   /lib64/libdl-2.12.so
  4002:20150521:190217.784 7f2bfb845000-7f2bfb8c8000 r-xp 00000000 08:03 10485930                   /lib64/libm-2.12.so
  4002:20150521:190217.784 7f2bfb8c8000-7f2bfbac7000 ---p 00083000 08:03 10485930                   /lib64/libm-2.12.so
  4002:20150521:190217.784 7f2bfbac7000-7f2bfbac8000 r--p 00082000 08:03 10485930                   /lib64/libm-2.12.so
  4002:20150521:190217.784 7f2bfbac8000-7f2bfbac9000 rw-p 00083000 08:03 10485930                   /lib64/libm-2.12.so
  4002:20150521:190217.784 7f2bfbac9000-7f2bfbb1b000 r-xp 00000000 08:03 33163149                   /usr/lib64/libcurl.so.4.1.1
  4002:20150521:190217.784 7f2bfbb1b000-7f2bfbd1a000 ---p 00052000 08:03 33163149                   /usr/lib64/libcurl.so.4.1.1
  4002:20150521:190217.784 7f2bfbd1a000-7f2bfbd1d000 rw-p 00051000 08:03 33163149                   /usr/lib64/libcurl.so.4.1.1
  4002:20150521:190217.784 7f2bfbd1d000-7f2bfbd2b000 r-xp 00000000 08:03 10485768                   /lib64/liblber-2.4.so.2.10.2
  4002:20150521:190217.784 7f2bfbd2b000-7f2bfbf2a000 ---p 0000e000 08:03 10485768                   /lib64/liblber-2.4.so.2.10.2
  4002:20150521:190217.784 7f2bfbf2a000-7f2bfbf2b000 r--p 0000d000 08:03 10485768                   /lib64/liblber-2.4.so.2.10.2
  4002:20150521:190217.784 7f2bfbf2b000-7f2bfbf2c000 rw-p 0000e000 08:03 10485768                   /lib64/liblber-2.4.so.2.10.2
  4002:20150521:190217.784 7f2bfbf2c000-7f2bfbf79000 r-xp 00000000 08:03 10485933                   /lib64/libldap-2.4.so.2.10.2
  4002:20150521:190217.784 7f2bfbf79000-7f2bfc178000 ---p 0004d000 08:03 10485933                   /lib64/libldap-2.4.so.2.10.2
  4002:20150521:190217.784 7f2bfc178000-7f2bfc17a000 r--p 0004c000 08:03 10485933                   /lib64/libldap-2.4.so.2.10.2
  4002:20150521:190217.784 7f2bfc17a000-7f2bfc17c000 rw-p 0004e000 08:03 10485933                   /lib64/libldap-2.4.so.2.10.2
  4002:20150521:190217.784 7f2bfc17c000-7f2bfc181000 r-xp 00000000 08:03 33163296                   /usr/lib64/libOpenIPMIposix.so.0.0.1
  4002:20150521:190217.784 7f2bfc181000-7f2bfc380000 ---p 00005000 08:03 33163296                   /usr/lib64/libOpenIPMIposix.so.0.0.1
  4002:20150521:190217.785 7f2bfc380000-7f2bfc381000 rw-p 00004000 08:03 33163296                   /usr/lib64/libOpenIPMIposix.so.0.0.1
  4002:20150521:190217.785 7f2bfc381000-7f2bfc469000 r-xp 00000000 08:03 33161880                   /usr/lib64/libOpenIPMI.so.0.0.5
  4002:20150521:190217.785 7f2bfc469000-7f2bfc669000 ---p 000e8000 08:03 33161880                   /usr/lib64/libOpenIPMI.so.0.0.5
  4002:20150521:190217.785 7f2bfc669000-7f2bfc685000 rw-p 000e8000 08:03 33161880                   /usr/lib64/libOpenIPMI.so.0.0.5
  4002:20150521:190217.785 7f2bfc685000-7f2bfc689000 rw-p 00000000 00:00 0 
  4002:20150521:190217.785 7f2bfc689000-7f2bfc6b0000 r-xp 00000000 08:03 33163064                   /usr/lib64/libssh2.so.1.0.1
  4002:20150521:190217.785 7f2bfc6b0000-7f2bfc8af000 ---p 00027000 08:03 33163064                   /usr/lib64/libssh2.so.1.0.1
  4002:20150521:190217.785 7f2bfc8af000-7f2bfc8b1000 rw-p 00026000 08:03 33163064                   /usr/lib64/libssh2.so.1.0.1
  4002:20150521:190217.785 7f2bfc8b1000-7f2bfc951000 r-xp 00000000 08:03 33163306                   /usr/lib64/libnetsnmp.so.20.0.0
  4002:20150521:190217.785 7f2bfc951000-7f2bfcb50000 ---p 000a0000 08:03 33163306                   /usr/lib64/libnetsnmp.so.20.0.0
  4002:20150521:190217.785 7f2bfcb50000-7f2bfcb53000 r--p 0009f000 08:03 33163306                   /usr/lib64/libnetsnmp.so.20.0.0
  4002:20150521:190217.785 7f2bfcb53000-7f2bfcb55000 rw-p 000a2000 08:03 33163306                   /usr/lib64/libnetsnmp.so.20.0.0
  4002:20150521:190217.785 7f2bfcb55000-7f2bfcb8b000 rw-p 00000000 00:00 0 
  4002:20150521:190217.785 7f2bfcb8b000-7f2bfcbea000 r-xp 00000000 08:03 33163340                   /usr/lib64/libodbc.so.2.0.0
  4002:20150521:190217.785 7f2bfcbea000-7f2bfcde9000 ---p 0005f000 08:03 33163340                   /usr/lib64/libodbc.so.2.0.0
  4002:20150521:190217.785 7f2bfcde9000-7f2bfcdf1000 rw-p 0005e000 08:03 33163340                   /usr/lib64/libodbc.so.2.0.0
  4002:20150521:190217.785 7f2bfcdf1000-7f2bfcdf2000 rw-p 00000000 00:00 0 
  4002:20150521:190217.785 7f2bfcdf2000-7f2bfcf3b000 r-xp 00000000 08:03 33163137                   /usr/lib64/libxml2.so.2.7.6
  4002:20150521:190217.785 7f2bfcf3b000-7f2bfd13a000 ---p 00149000 08:03 33163137                   /usr/lib64/libxml2.so.2.7.6
  4002:20150521:190217.785 7f2bfd13a000-7f2bfd143000 rw-p 00148000 08:03 33163137                   /usr/lib64/libxml2.so.2.7.6
  4002:20150521:190217.785 7f2bfd143000-7f2bfd145000 rw-p 00000000 00:00 0 
  4002:20150521:190217.785 7f2bfd145000-7f2bfd152000 r-xp 00000000 08:03 33163381                   /usr/lib64/libiksemel.so.3.1.1
  4002:20150521:190217.785 7f2bfd152000-7f2bfd351000 ---p 0000d000 08:03 33163381                   /usr/lib64/libiksemel.so.3.1.1
  4002:20150521:190217.785 7f2bfd351000-7f2bfd352000 rw-p 0000c000 08:03 33163381                   /usr/lib64/libiksemel.so.3.1.1
  4002:20150521:190217.785 7f2bfd352000-7f2bfd488000 r-xp 00000000 08:03 33687875                   /usr/lib64/mysql/libmysqlclient.so.16.0.0
  4002:20150521:190217.785 7f2bfd488000-7f2bfd687000 ---p 00136000 08:03 33687875                   /usr/lib64/mysql/libmysqlclient.so.16.0.0
  4002:20150521:190217.785 7f2bfd687000-7f2bfd6d5000 rw-p 00135000 08:03 33687875                   /usr/lib64/mysql/libmysqlclient.so.16.0.0
  4002:20150521:190217.785 7f2bfd6d5000-7f2bfd6d6000 rw-p 00000000 00:00 0 
  4002:20150521:190217.785 7f2bfd6d6000-7f2bfd6f6000 r-xp 00000000 08:03 10485785                   /lib64/ld-2.12.so
  4002:20150521:190217.785 7f2bfd8d7000-7f2bfd8ee000 rw-p 00000000 00:00 0 
  4002:20150521:190217.785 7f2bfd8f0000-7f2bfd8f1000 rw-p 00000000 00:00 0 
  4002:20150521:190217.785 7f2bfd8f1000-7f2bfd8f3000 rw-s 00000000 00:04 2359301                    /SYSV53030518 (deleted)
  4002:20150521:190217.785 7f2bfd8f3000-7f2bfd8f4000 rw-p 00000000 00:00 0 
  4002:20150521:190217.785 7f2bfd8f4000-7f2bfd8f5000 rw-p 00000000 00:00 0 
  4002:20150521:190217.785 7f2bfd8f5000-7f2bfd8f6000 r--p 0001f000 08:03 10485785                   /lib64/ld-2.12.so
  4002:20150521:190217.785 7f2bfd8f6000-7f2bfd8f7000 rw-p 00020000 08:03 10485785                   /lib64/ld-2.12.so
  4002:20150521:190217.785 7f2bfd8f7000-7f2bfd8f8000 rw-p 00000000 00:00 0 
  4002:20150521:190217.785 7ffff4417000-7ffff449f000 rw-p 00000000 00:00 0                          [stack]
  4002:20150521:190217.785 7ffff44d0000-7ffff44d1000 r-xp 00000000 00:00 0                          [vdso]
  4002:20150521:190217.785 ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0                  [vsyscall]
  4002:20150521:190217.785 ================================
  4002:20150521:190217.785 Please consider attaching a disassembly listing to your bug report.
  4002:20150521:190217.785 This listing can be produced with, e.g., objdump -DSswx zabbix_server.
  4002:20150521:190217.785 ================================
  3984:20150521:190217.786 One child process died (PID:4002,exitcode/signal:1). Exiting ...
  3984:20150521:190219.787 syncing history data...
  3984:20150521:190219.788 syncing history data done
  3984:20150521:190219.788 syncing trends data...
  3984:20150521:190220.244 syncing trends data done
  3984:20150521:190220.244 Zabbix Server stopped. Zabbix 2.4.5 (revision 53282).


 Comments   
Comment by Aleksandrs Saveljevs [ 2015 May 22 ]

Which OpenIPMI library version are you using?

Comment by Uwe Kiewel [ 2015 May 22 ]

OpenIPMI-libs-2.0.16-14.el6.x86_64

Comment by Aleksandrs Saveljevs [ 2015 May 22 ]

In our experience, there were many problems with 2.0.16 that were solved simply by upgrading to a newer version. If you upgrade to a newer version, does that solve the problem?

Comment by Uwe Kiewel [ 2015 May 23 ]

Same with 2.0.18. Later versions of OpenIPMI I'm unable to build due to build errors or linker errors

Comment by Igors Homjakovs (Inactive) [ 2015 May 27 ]

Uwe,

Could you please do the following:

  • attach Zabbix server log file with DebugLevel=4 after the crash happens;
  • attach libOpenIPMI.so.0;
  • run the following command with IPv6 address specified and check if returns something: ipmitool -I lanplus -H <IPADDR> -U <user> -P <password> sdr elist full.

That would be very helpful for us to pin-point the problem.

Comment by Uwe Kiewel [ 2015 Jun 23 ]

hm... I'm not sure was happened. I am unable to reproduce that behaviour since a couple of days...

Comment by Aleksandrs Saveljevs [ 2016 Sep 20 ]

Closing as "Cannot Reproduce" then. If the issue is still relevant, please reopen.

Comment by Jaroslav Skrivan [ 2017 May 02 ]

For other people who find this bug report. We faced the same situation on Debian Jessie with openipmi and libopenipmi 2.0.16 installed from the debian repository.

We solved the problem by upgrading libopenipmi to 2.0.22-1.1 (package from stretch) and openipmi to 2.0.18-0ubuntu7 (package from ubuntu trusty repo).





[ZBX-9167] IPMI sensor retrieval fails on non-present sensors Created: 2014 Dec 24  Updated: 2019 Dec 10

Status: Open
Project: ZABBIX BUGS AND ISSUES
Component/s: Proxy (P), Server (S)
Affects Version/s: 2.2.8
Fix Version/s: None

Type: Incident report Priority: Trivial
Reporter: Aristarkh Zagorodnikov Assignee: Unassigned
Resolution: Unresolved Votes: 0
Labels: ipmi, patch
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: PNG File chart2.png     Text File ipmi.patch    

 Description   

Certain IPMI modules (SuperMicro X10-based) return non-present sensors, but error out on data retrieval. Because got_thresh_reading(...) in src/zabbix_server/poller/checks_ipmi.c handles any errors as network failures this leads to other sensors from the same machine to silently fail as if real network failure occurred. I'm attaching patch that treats the "non-present sensor" error as "not supported". The difference before/after the patch is clearly seen in the attached chart.






[ZBX-8992] IPMI problem with zabbix 2.4.1 Created: 2014 Nov 04  Updated: 2017 May 30  Resolved: 2014 Dec 05

Status: Closed
Project: ZABBIX BUGS AND ISSUES
Component/s: Proxy (P), Server (S)
Affects Version/s: 2.4.1
Fix Version/s: 2.2.8rc1, 2.4.3rc1, 2.5.0

Type: Incident report Priority: Major
Reporter: ufocek Assignee: Unassigned
Resolution: Fixed Votes: 0
Labels: ipmi, sensors
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

uname -a
Linux zabbix2 3.12.4-031204-generic #201312081207 SMP Sun Dec 8 17:08:59 UTC 2013 x86_64 GNU/Linux
cat /etc/debian_version
7.7
dpkg -l |grep ipmi
ii ipmitool 1.8.11-5 amd64 utility for IPMI control with kernel driver or LAN interface
openipmi-2.0.21 from source..


Attachments: File ipmi2.log     File preferring-threshold.patch     HTML File zabbix2013_ipmi_values     HTML File zabbix2013_ipmi_values     PNG File zabbix2013_ipmi_values.png     HTML File zabbix241_ipmi_values     PNG File zabbix241_ipmi_values.png     Text File zabbix_log_ipmi_fan2.txt    

 Description   

Hi,

I have problem with zabbix-2.4.1 and ipmi, because I received wrong information about sensor:

$ ipmitool -I lanplus -H 192.168.249.34 -L USER -A MD5 -U  -P  sdr, show
Ambient          | 21 degrees C      | ok
Systemboard      | 29 degrees C      | ok
CPU1             | 48 degrees C      | ok
CPU2             | 44 degrees C      | ok

This is a data from ipmitool command line, and show good value for FAN2 SYS:

$ ipmitool -I lanplus -H 192.168.249.34 -L USER -A MD5 -U -P sensor get "FAN2 SYS"
Sensor ID              : FAN2 SYS (0x27)
 Entity ID             : 29.1
 Sensor Type (Analog)  : Fan
 Sensor Reading        : 6240 (+/- 60) RPM
 Status                : ok
 Lower Non-Recoverable : na
 Lower Critical        : 1920.000
 Lower Non-Critical    : na
 Upper Non-Critical    : na
 Upper Critical        : na
 Upper Non-Recoverable : na
 Assertion Events      : 
 Assertions Enabled    : 

In zabbix gui i check data, and fan2 sys have value: 1 RPM... or is not supported...

For test I try configure fan2 sys and configure in zabbix like this:

Now I add for test in zabbix key:

Name: FAN2 SYS
Type: ipmi agent
Key: ipmi.fan2.sys.rpm
Host interface: ip:623
IPMI sensor: FAN2 SYS (I put space between FAN1 and SYS)
Type of information: Numeric
Units: RPM

Zabbix show me information about item: not_supported, sensor data is not available or show value: 1 RPM
What I set wrong in zabbix, that isn't work? This is a problem zabbix or ipmi?



 Comments   
Comment by Aleksandrs Saveljevs [ 2014 Nov 04 ]

This might be similar to ZBX-8395, where ipmitool sees an analog sensor, but Zabbix sees a discrete sensor. What kind of hardware are you monitoring?

Could you please also post the output of "grep FAN2 ..." of DebugLevel=4 log? Note that since Zabbix 2.4.0 log level can be increased at runtime and only for a specific type of processes: see http://blog.zabbix.com/zabbix-2-4-features-part-6-runtime-loglevel-changing/ .

Comment by ufocek [ 2014 Nov 04 ]

Aleksandrs - I try monitor servers RX200S6/7 from Fujitsu Technology Solutions.. Ok, I increase a log and check this when I have access to this machine..

Comment by ufocek [ 2014 Nov 05 ]

Log file for FAN2

Comment by Aleksandrs Saveljevs [ 2014 Nov 05 ]

The attached log shows a case where that item becomes not supported. However, you mentioned in the issue description that sometimes it has a value of 1. Could you please show a longer part of the log where the latter case is observable, too?

Comment by ufocek [ 2014 Nov 06 ]

Hi Aleksandrs,

In attachment I put log file from ipmi debug, maybe this log help yoy and I put screen from zabbix what I see when I turn on ipmi, what values I get from ipmi for: FANs, CPUs,Power....
I have a weird data.. because from command line e.g Total Power show me 178W, but in zabbix gui only 1W...

Comment by Aleksandrs Saveljevs [ 2014 Nov 06 ]

Attached "zabbix241_ipmi_values.png" with the correct extension:

Comment by Aleksandrs Saveljevs [ 2014 Nov 06 ]

Attached log contains lines like the following, which shows that Zabbix indeed sees this sensor as a discrete sensor:

 15987:20141106:073522.823 In read_ipmi_sensor() sensor:'FAN2 SYS@[192.168.249.34]:623'
 15987:20141106:073522.825 State [FAN2 SYS | fan_cooling | invalid | sensor specific | state 0 value is 1]
 15987:20141106:073522.825 State [FAN2 SYS | fan_cooling | invalid | sensor specific | state 1 value is 0]
 15987:20141106:073522.825 State [FAN2 SYS | fan_cooling | invalid | sensor specific | state 2 value is 0]
 15987:20141106:073522.826 State [FAN2 SYS | fan_cooling | invalid | sensor specific | state 3 value is 0]
 15987:20141106:073522.826 State [FAN2 SYS | fan_cooling | invalid | sensor specific | state 4 value is 0]
 15987:20141106:073522.826 State [FAN2 SYS | fan_cooling | invalid | sensor specific | state 5 value is 0]
 15987:20141106:073522.826 State [FAN2 SYS | fan_cooling | invalid | sensor specific | state 7 value is 0]

So this issue is the same as ZBX-8395. Not sure whether there is anything that can be done about it.

Comment by ufocek [ 2014 Nov 06 ]

I wonder what happened to that previously worked....

Comment by Aleksandrs Saveljevs [ 2014 Nov 06 ]

Do you mean that it used to work in a previous version of Zabbix? If so, which version that is?

Comment by ufocek [ 2014 Nov 06 ]

If I good memory it's work with zabbix 2.0.x version.

Comment by ufocek [ 2014 Nov 06 ]

Good data values from ipmi source with zabbix 2.0.13, the same configuration on zabbix-2.4.2 return bad values.

Comment by ufocek [ 2014 Nov 06 ]

Aleksandrs - I installed for test zabbix 2.0.13 and add the same source ipmi. Zabbix 2.0.13 show me a good values, in attachment I put screen with zabbix data.
So for me something change in zabbix-2.2 or 2.4.x version... It's possible to solve this problem?

Comment by Aleksandrs Saveljevs [ 2014 Nov 06 ]

Reattaching "zabbix2013_ipmi_values.png" with a proper extension:

Comment by Aleksandrs Saveljevs [ 2014 Nov 06 ]

ufocek, thank you, the fact that it used to work before is valuable information. We shall investigate.

Comment by ufocek [ 2014 Nov 06 ]

Ok. It's possible to get a patch before you public a new version, I can test in my environment..?

Comment by Aleksandrs Saveljevs [ 2014 Nov 07 ]

Yes, when the development is complete (we will let you know here), we would appreciate if you would test the new version. Zabbix IPMI works well with the hardware we have locally.

Comment by Aleksandrs Saveljevs [ 2014 Dec 02 ]

In one of the attached logs we can see the following lines:

$ grep FAN2.SYS ipmi2.log
 ...
 15983:20141106:080104.052 In allocate_ipmi_sensor() sensor:'FAN2 SYS@[192.168.249.34]:623'
 15983:20141106:080104.052 Added sensor: host:'192.168.249.34:623' id_type:0 id_sz:9 id:'FAN2 SYS' reading_type:0x6f ('sensor specific') type:0xe6 ('invalid') full_name:'(29.1).FAN2 SYS'
 15983:20141106:080104.071 In allocate_ipmi_sensor() sensor:'FAN2 SYS@[192.168.249.34]:623'
 15983:20141106:080104.071 Added sensor: host:'192.168.249.34:623' id_type:0 id_sz:9 id:'FAN2 SYS' reading_type:0x1 ('threshold') type:0x4 ('fan') full_name:'(29.1).FAN2 SYS'
 ...

Here, the same sensor is seen by Zabbix as both a threshold sensor and a discrete sensor. This is dubious, but we can work around this by preferring the threshold sensor, if both are available.

This solution has been implemented in development branch svn://svn.zabbix.com/branches/dev/ZBX-8992 for 2.2.

Comment by Aleksandrs Saveljevs [ 2014 Dec 02 ]

ufocek, please try attached "preferring-threshold.patch" and let us know your results. It was created for Zabbix 2.4.1.

Comment by ufocek [ 2014 Dec 02 ]

Aleksandrs, I download your patch and apply for zabbix 2.4.2, becuase I have that version. Then I add my templates for ipmi and now everything looks correct, I have a good values from ipmi for all my sensor.

Comment by Alexander Vladishev [ 2014 Dec 04 ]

(1) Made some code simplification in r51001. Please review.

asaveljevs Looks good. CLOSED.

Comment by Aleksandrs Saveljevs [ 2014 Dec 04 ]

Fixed in pre-2.2.8 r51007, pre-2.4.3 r51008, and pre-2.5.0 (trunk) r51009.

Comment by Aleksandrs Saveljevs [ 2014 Dec 05 ]

(2) Documented at the following locations:

sasha it should be mentioned in:

REOPENED

asaveljevs Added notes at the bottom. RESOLVED.

sasha Thanks a lot! CLOSED





[ZBX-8734] Zabbix ipmi poller processes more than 75% busy Created: 2014 Sep 10  Updated: 2017 May 30  Resolved: 2014 Sep 10

Status: Closed
Project: ZABBIX BUGS AND ISSUES
Component/s: Proxy (P), Server (S)
Affects Version/s: 2.4.0rc2
Fix Version/s: 2.4.0rc3

Type: Incident report Priority: Blocker
Reporter: Areg Vrtanesyan Assignee: Unassigned
Resolution: Fixed Votes: 0
Labels: ipmi
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: PNG File Busy processes - ipmi - v2.4.0rc2.PNG    

 Description   

Hi

In recent 2.4.0rc2 and couple of previous versions I have noticed that the IPMI Poller busy data is not correct.

It always show that the pollers are 100% busy even there are no IPMI items configured in the system and all the hosts monitoring is disabled.

https://www.zabbix.com/forum/showthread.php?p=155241

Regards,
Areg



 Comments   
Comment by Aleksandrs Saveljevs [ 2014 Sep 10 ]

This problem is only present in trunk (2.4.0rc1) and is not present in 2.2 branch. It got broken when ZBXNEXT-101 was merged, which introduced the following code in get_process_info_by_thread() in server.c:

...
else if (server_num <= (server_count += CONFIG_IPMIPOLLER_FORKS))
{
	*process_type = ZBX_PROCESS_TYPE_SNMPTRAPPER;
	*process_num = server_num - server_count + CONFIG_IPMIPOLLER_FORKS;
}
...

Note the SNMPTRAPPER typo instead of IPMIPOLLER. I fixed that for proxy.c in 48519, but missed the same case in server.c.

Comment by Aleksandrs Saveljevs [ 2014 Sep 10 ]

Fixed in development branch svn://svn.zabbix.com/branches/dev/ZBX-8734 .

Comment by Andris Zeila [ 2014 Sep 10 ]

Successfully tested

Comment by Aleksandrs Saveljevs [ 2014 Sep 10 ]

Fixed in pre-2.4.0rc3 (trunk) r48910.





[ZBX-8661] IPMI return 1 instead real value Created: 2014 Aug 25  Updated: 2017 May 30  Resolved: 2014 Dec 04

Status: Closed
Project: ZABBIX BUGS AND ISSUES
Component/s: Proxy (P), Server (S)
Affects Version/s: 2.2.5
Fix Version/s: None

Type: Incident report Priority: Major
Reporter: Sergey A. Volkov Assignee: Unassigned
Resolution: Duplicate Votes: 0
Labels: ipmi
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

HP Proliant DL380e Gen8
iLO 4 Firmware 1.5
libopenipmi 2.0.16-1.3 and 2.0.19
ipmitool 1.8.11-5


Attachments: PNG File ilo.png     PNG File ipmi_ui.png     PNG File zabbix.png     PNG File zbx_element_cfg.png    
Issue Links:
Duplicate
duplicates ZBX-8395 Zabbix display wrong values with IPMI... Closed

 Description   

Zabbix server usage libopenipmi0. This lib can't see value some sensors.
When usage ipmitool:

root@zabbix:~# ipmitool -I lanplus -H X.X.X.X -U user -P password sensor | fgrep 'Fan 1'
Fan 1            | 51.352     | unspecified | nc    | na        | na        | na        | na        | na        | na

When usage ipmi_ui (libopenipmi0) i see 1 and zabbix server see 1 too.



 Comments   
Comment by richlv [ 2014 Aug 25 ]

could this be same as ZBX-8395 ?

Comment by Alexander Vladishev [ 2014 Aug 26 ]

Please attach a item configuration screenshot.
Thank you.

Comment by Aleksandrs Saveljevs [ 2014 Nov 05 ]

Could you please post the output of "grep Fan.1 ..." of DebugLevel=4 log, so that we know whether this issue is the same as ZBX-8395 and whether Zabbix treats that sensor as a discrete sensor?

Comment by Aleksandrs Saveljevs [ 2014 Dec 04 ]

No response, closing as a duplicate of ZBX-8395. Please reopen if more information is available.





[ZBX-8310] IPMI dont work, and strange error message Created: 2014 Jun 06  Updated: 2019 Dec 02  Resolved: 2014 Jun 06

Status: Closed
Project: ZABBIX BUGS AND ISSUES
Component/s: Server (S)
Affects Version/s: 2.2.3
Fix Version/s: None

Type: Incident report Priority: Major
Reporter: Stefan Assignee: Unassigned
Resolution: Duplicate Votes: 0
Labels: ipmi
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

Debian Wheezy, MySQL and Apache


Attachments: JPEG File zbx_ipmi.JPG    
Issue Links:
Duplicate
duplicates ZBX-6077 Can not fetch IPMI values if privileg... Closed

 Description   

Hello,
in the zabbix frontend i have the following error:
cannot connect to IPMI host: [22] Invalid argument
Why he said host 22 ?! i dont has a 22?!

for host1(HP Dl380 Gen7)
6019:20140605:162258.496 In get_values()
6019:20140605:162258.496 In DCconfig_get_poller_items() poller_type:1
6019:20140605:162258.496 End of DCconfig_get_poller_items():1
6019:20140605:162258.496 In substitute_key_macros() data:'Temp5'
6019:20140605:162258.496 End of substitute_key_macros():SUCCEED data:'Temp5'
6019:20140605:162258.496 In substitute_simple_macros() data:'623'
6019:20140605:162258.496 In get_value() key:'Temp5'
6019:20140605:162258.496 In get_value_ipmi() key:'34333336-3530-5a43-3231-343130434a38:Temp5'
6019:20140605:162258.496 In init_ipmi_host() host:'[192.168.200.64]:623'
6019:20140605:162258.496 In get_ipmi_host() host:'[192.168.200.64]:623'
6019:20140605:162258.496 End of get_ipmi_host():0xff9c00
6019:20140605:162258.500 EINF: 0 ipmi_lan.c(got_rmcpp_open_session_rsp): Expected privilege 2, got 4
6019:20140605:162258.500 In setup_done() phost:0xff9c00 host:'[192.168.200.64]:623'
6019:20140605:162258.500 setup_done() fail: [22] Invalid argument
6019:20140605:162258.500 End of setup_done():NETWORK_ERROR
6019:20140605:162258.500 In domain_closed() phost:0xff9c00 host:'[192.168.200.64]:623'
6019:20140605:162258.500 End of domain_closed()
6019:20140605:162258.500 End of init_ipmi_host():0xff9c00
6019:20140605:162258.500 Item [34333336-3530-5a43-3231-343130434a38:Temp5] error: cannot connect to IPMI host: [22] Invalid argument
6019:20140605:162258.500 End of get_value():NETWORK_ERROR
6019:20140605:162258.500 In deactivate_host() hostid:10167 itemid:26780 type:12
6019:20140605:162258.500 query [txnlev:1] [begin;]
6019:20140605:162258.500 query [txnlev:1] [update hosts set ipmi_disable_until=1401978238,ipmi_error='cannot connect to IPMI host: [22] Invalid argument' where hostid=10167]
6019:20140605:162258.501 query [txnlev:1] [commit;]
6019:20140605:162258.502 deactivate_host() errors_from:1401975807 available:2
6019:20140605:162258.503 End of deactivate_host()
6019:20140605:162258.503 End of get_values():1

when i try this in the console it works fine (note, i must use -I lanplus, otherwise it dont work):
ipmitool -I lanplus -H 192.168.200.64 -U zabbix -P <PW>-L User sdr
UID Light | 0 unspecified | ok
Sys. Health LED | 0 unspecified | ok
Power Supply 1 | 40 Watts | nc
Power Supply 2 | 60 Watts | nc
Power Supplies | 0 unspecified | nc
Fan 1 | 13.72 unspecifi | nc
Fan 2 | 13.72 unspecifi | nc
Fan 3 | 13.72 unspecifi | nc
Fan 4 | 13.72 unspecifi | nc
Fan 5 | disabled | ns
Fan 6 | disabled | ns
Fans | 0 unspecified | nc
Temp 1 | 17 degrees C | ok
Temp 2 | 40 degrees C | ok
Temp 3 | disabled | ns
Temp 4 | 33 degrees C | ok
Temp 5 | 32 degrees C | ok
Temp 6 | disabled | ns
Temp 7 | disabled | ns
Temp 8 | 38 degrees C | ok
Temp 9 | 33 degrees C | ok
Temp 10 | 42 degrees C | ok
Temp 11 | 36 degrees C | ok
Temp 12 | 46 degrees C | ok
Temp 13 | disabled | ns
Temp 14 | disabled | ns
Temp 15 | disabled | ns
Temp 16 | disabled | ns
Temp 17 | disabled | ns
Temp 18 | disabled | ns
Temp 19 | 23 degrees C | ok
Temp 20 | 30 degrees C | ok
Temp 21 | 34 degrees C | ok
Temp 22 | 31 degrees C | ok
Temp 23 | 41 degrees C | ok
Temp 24 | 37 degrees C | ok
Temp 25 | 29 degrees C | ok
Temp 26 | 30 degrees C | ok
Temp 27 | disabled | ns
Temp 28 | disabled | ns
Temp 29 | 35 degrees C | ok
Temp 30 | 65 degrees C | ok
Memory | 0 error | ok
Power Meter | 144 Watts | cr



 Comments   
Comment by Aleksandrs Saveljevs [ 2014 Jun 06 ]

Answering your first question about "cannot connect to IPMI host: [22] Invalid argument", "22" is the value of "errno" variable, not the host name.

Comment by Aleksandrs Saveljevs [ 2014 Jun 06 ]

The key line in the log is probably this:

  6019:20140605:162258.500 EINF: 0 ipmi_lan.c(got_rmcpp_open_session_rsp): Expected privilege 2, got 4 

In Zabbix frontend, you probably have privilege level for that host configured to "Admin", whereas in the command line you use "-L User".

Comment by Aleksandrs Saveljevs [ 2014 Jun 06 ]

In general, since this is not a bug report, in the future you might wish to investigate configuration issues together with the community at https://www.zabbix.org/wiki/Getting_help .

Comment by Stefan [ 2014 Jun 06 ]

pleas see the screenshot, i think i configure it correct..

my other Servers (Dell, Supermicro, etc) works correct with IPMI

Comment by Aleksandrs Saveljevs [ 2014 Jun 06 ]

ZBX-3389 seems to be the same problem. Perhaps you could check the privileges and play with them a bit (see the last comments in that issue)?

Comment by Stefan [ 2014 Jun 06 ]

i think its a bug..

it works only if i give the zabbix-user Administrative right in the iLOs and use Admin-Privileg-Level, but i would like only use User-rights..

Comment by richlv [ 2014 Jun 06 ]

it's a combo of hp ilo and openipmi bug, see ZBX-6077 for more info





[ZBX-8219] Broken support of ON/OFF values in IPMI commands Created: 2014 May 16  Updated: 2017 May 30  Resolved: 2014 May 19

Status: Closed
Project: ZABBIX BUGS AND ISSUES
Component/s: Server (S)
Affects Version/s: 2.2.3
Fix Version/s: 2.2.4rc1, 2.3.0

Type: Incident report Priority: Major
Reporter: Alexander Vladishev Assignee: Unassigned
Resolution: Fixed Votes: 0
Labels: ipmi
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Description   

From version 2.2.0 server doesn't support ON/OFF values in IPMI commands.

For example:
power off



 Comments   
Comment by Andris Mednis [ 2014 May 16 ]

Log file sample:
8488:20140516:101609.261 In zbx_execute_ipmi_command()
8488:20140516:101609.261 In parse_ipmi_command() command:'power off'
8488:20140516:101609.261 End of parse_ipmi_command():FAIL
8488:20140516:101609.261 End of zbx_execute_ipmi_command():FAIL

Comment by Juris Miščenko (Inactive) [ 2014 May 19 ]

Fix implemented in svn://svn.zabbix.com/branches/dev/ZBX-8219

Comment by Juris Miščenko (Inactive) [ 2014 May 20 ]

Fix merge in 2.2.4rc1 r45656, 2.3.0 (trunk) r45657





[ZBX-8188] Zabbix server continue to retrieve values from IPMI agent even if the machine is shutdown state Created: 2014 May 08  Updated: 2017 May 30  Resolved: 2014 May 12

Status: Closed
Project: ZABBIX BUGS AND ISSUES
Component/s: Proxy (P), Server (S)
Affects Version/s: 2.0.11, 2.2.3
Fix Version/s: 2.0.13rc1, 2.2.4, 2.3.2

Type: Incident report Priority: Minor
Reporter: Kodai Terashima Assignee: Unassigned
Resolution: Fixed Votes: 0
Labels: ipmi
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Duplicate

 Description   

Zabbix server and proxy continue to retrieve values from IPMI sensors even if the machine is shutdown state. ipmitools return "na" for such case.

IPMI device return values with sensor status but Zabbix doesn't check the status.
So Zabbix retrieve values from not working IPMI sensors (i.e. CPU temperature is 50 degree from stopped machine)

According to IPMI 2.0 specification, IPMI client should check sensor status before getting value. IPMI item should be not-supported state in that case.



 Comments   
Comment by Aleksandrs Saveljevs [ 2014 May 08 ]

ZBX-7847 might be related.

kodai I guess these are different issue. this is problem when monitored machine is stopped (shutdown), ZBX-7847 is about disabled host on Zabbix frontend.

Comment by Nikolajs Agafonovs (Inactive) [ 2014 May 12 ]

Fix available in svn://svn.zabbix.com/branches/dev/ZBX-8188

Comment by Aleksandrs Saveljevs [ 2014 May 16 ]

Kodai, could you please provide a reference to IPMI 2.0 specification where it says about checking sensor status?

kodai I checked ipmitool source (ipmitool return n/a for stopped device), haven't checked IPMI 2.0 specification. will try to find it from the specification.

Comment by Aleksandrs Saveljevs [ 2014 May 19 ]

Alternative solution is available in the same development branch: svn://svn.zabbix.com/branches/dev/ZBX-8188 .

It uses functions ipmi_is_sensor_scanning_enabled() and ipmi_is_initial_update_in_progress() as previously suggested by Kodai.

Comment by Kodai Terashima [ 2014 May 27 ]

According to page 464 on http://www.intel.com/content/dam/www/public/us/en/documents/product-briefs/second-gen-interface-spec-v2.pdf, Table 35-15, Get Sensor Reading Command 3rd bit of response data:

reading/state unavailable (formerly “initial update in progress”). This bit is set to indicate that a ‘re-arm’ or ‘Set Event Receiver’ command has been used to request an update of the sensor status, and that update has not occurred yet. Software should use this bit to avoid getting an incorrect status while the first sensor update is in progress. This bit is only required if it is possible for the controller to receive and process a ‘Get Sensor Reading’ or ‘Get Sensor Event Status’ command for the sensor before the update has completed. This is most likely to be the case for sensors, such as fan RPM sensors, that may require seconds to accumulate the first reading after a re-arm. The bit is also used to indicate when a reading/state is unavailable because the management controller cannot obtain a valid reading or state for the monitored entity, typically because the entity is not present. See Section 16.4, Event Status, Event Conditions, and Present State and Section 16.6, Re-arming for more information.

Comment by dimir [ 2014 Jun 02 ]

Another one from the same doc regarding "sensor scanning enabled":

An Entity is present if there is at least one active sensor for the Entity (and there is no explicit sensor saying
the Entity is 'absent'). A sensor is 'active' if scanning is enabled. For each of these sensors, check to see that
at least one of the sensors is scanning by checking the "sensor scanning disabled" bit via the Get Sensor
Reading command. Per section 11.5, software should ignore this bit if its set to 'disabled'. If there are no
active sensors for the entity, then it should be assumed that the Entity is absent.

Comment by dimir [ 2014 Jun 02 ]

(1) [S] As I understood from the above mentioned document, the "initial update in progress" and "sensor scanning disabled" means in general that the sensor data is not available (either not yet ready or entity does not exist). So I suggest to simplify the error message for the user like this:

        if (0 == ipmi_is_sensor_scanning_enabled(states))
        {
-               h->err = zbx_strdup(h->err, "sensor scanning is not enabled");
+               h->err = zbx_strdup(h->err, "sensor data is not available");
@@ -376,7 +376,7 @@
 
        if (0 != ipmi_is_initial_update_in_progress(states))
        {
-               h->err = zbx_strdup(h->err, "initial update is in progress");
+               h->err = zbx_strdup(h->err, "sensor data is not available");

Please see my suggestion in r46099.

asaveljevs Looks good, CLOSED.

Comment by dimir [ 2014 Jun 02 ]

Other than that the issue is tested.

Comment by Aleksandrs Saveljevs [ 2014 Jun 03 ]

(2) There was a conflict merging from 2.0 into 2.2 because the newer version has discrete sensor support.

Resolved conflicts are available in development branch svn://svn.zabbix.com/branches/dev/ZBX-8188-2.2 . Please take a look.

dimir Looks good. Please see my suggestions to simplify the code a bit in r46134. RESOLVED

asaveljevs Great! CLOSED.

Comment by Aleksandrs Saveljevs [ 2014 Jun 03 ]

Fixed in pre-2.0.13 r46114, pre-2.2.4 r46164, pre-2.3.2 (trunk) r46165.





[ZBX-7862] Zabbix does not use Timeout option for IPMI checks Created: 2014 Feb 24  Updated: 2022 Oct 08  Resolved: 2015 Dec 14

Status: Closed
Project: ZABBIX BUGS AND ISSUES
Component/s: Proxy (P), Server (S)
Affects Version/s: 2.2.2
Fix Version/s: None

Type: Incident report Priority: Blocker
Reporter: Alexey Pustovalov Assignee: Unassigned
Resolution: Fixed Votes: 0
Labels: ipmi, timeout
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Duplicate

 Description   

Zabbix must use Timeout configuration option for IPMI checks.



 Comments   
Comment by Oleksii Zagorskyi [ 2015 Jul 08 ]

ZBXNEXT-1096 would resolve current request in a more general way.

Comment by Sandis Neilands (Inactive) [ 2015 Nov 05 ]

This is a feature request rather than bug report. Both documentation and comment in configuration file state that Timeout is only applicable to Zabbix agent, SNMP, external checks: "Specifies how long we wait for agent, SNMP device or external check (in seconds)."

Low-level IPMI timeouts

IPMI over LAN uses unreliable transport protocol (UDP). Multiple RMCP or RMCP+ request/response exchanges are needed to get a single value even if there are no IP/UDP level errors. We are using OpenIPMI libray for communications with BMC over LAN. The library manages all low-level protocol details including timeouts, number of retries. As of the current version (2.0.21) these parameters are hardcoded in internal OpenIPMI C file.

lib/ipmi_lan.c

* Timeout to wait for IPMI responses, in microseconds.  For commands
   with side effects, we wait 5 seconds, not one. */
#define LAN_RSP_TIMEOUT 1000000 
#define LAN_RSP_TIMEOUT_SIDEEFF 5000000

/* # of times to try a message before we fail it. */
#define LAN_RSP_RETRIES 6

Errors

Low-level

If OpenIPMI invokes Zabbix callback functions with error then Zabbix sets NETWORK_ERROR and closes the connection. Generally this happens when BMC responds with unsuccessful CC or RCMP+ error code, or if there was an internal error in the OpenIPMI library (OS error).

In case of IPMI over LAN the OpenIPMI library sets IPMI_TIMEOUT_CC (0xC3) if BMC has not responded to six consecutive requests in time. This CC can also be included in the response from the BMC. The library sets IPMI_UNKNOWN_ERR_CC (0xFF) for various (and many) reasons. Unfortunately these reasons are not logged or otherwise reported back to the library's user.

High level

For other, high-level IPMI item errors Zabbix sets the item to NOTSUPPORTED and provides relevant error message.

Troubleshooting

Troubleshooting OpenIPMI is rather difficult due to most of the IPMI over LAN traffic being encrypted. In development environment one can use DEBUG_RAWMSG_ENABLE(), DEBUG_MSG_ERR_ENABLE() and other OpenIPMI macros to enable extra logging from the library.

As for the following code pattern in read_ipmi_sensor(), read_ipmi_control(), set_ipmi_control() and init_ipmi_host()...

src/zabbix_server/poller/checks_ipmi.c

        tv.tv_sec = 10;
        tv.tv_usec = 0;

        while (0 == h->done)
                os_hnd->perform_one_op(os_hnd, &tv);

As far as I can tell the timeout tv is used as maximum timeout for select() call inside OpenIPMI library. The actual timeout for select() will be the time remaining to the nearest timer expiration. If there is no activity on the read or write sockets, or timers expiring during this time then perform_one_op() returns and is invoked again with maximum timeout of 10 seconds.

The value 10 is arbitrary, it can be changed with no effect to the overall functionality.

Summary

The low-level IPMI protocol timeouts and retry counts are defined in OpenIPMI library. The library doesn't provide interface to change the values of these constants so there is nothing that we can do on the Zabbix side.

High-level, per-item check IPMI timeouts

Since multiple requests/responses are needed to get value for a single item it might be desirable from the users PoV to have a total timeout for the whole interaction.

The options for what to do when such timeout is reached are limited by the design of the IPMI and OpenIPMI library. Currently the only straightforward and safe option is to destroy the domain and start over during the next check (this will cause a full scan of the BMC - can take at least a minute).

In summary: implementing a per-item check IPMI timeout currently has no justification.

IPMI session inactivity timeout

The IPMI specification (section 6.12.15 Session Inactivity Timeouts) requires session inactivity timeout to be implemented. For LAN the default timeout 60 +/- 3 seconds. In order to keep inactive session open the system monitoring software can use Activate Session command.

I doesn't seem to be possible to enable periodic sending of Activate Session command with OpenIPMI. If there are no IPMI item checks form Zabbix to a particular BMC for more than the session timeout configured in BMC then for the next IPMI check after the timeout is expired will time out due to low-level timeouts, retires or receive error CC. After that a new session will be opened and full rescan of the BMC will be initiated.

In summary: if the users want to avoid unnecessary rescans of the BMC it is advised to set the IPMI item polling interval below IPMI's session inactivity timeout configured in BMC.

Comment by Oleksii Zagorskyi [ 2015 Nov 05 ]

What about a feature request creation on a OpenIPMI tracker ?
https://sourceforge.net/p/openipmi/feature-requests/

Could be this report (reported as bug) related ?
https://sourceforge.net/p/openipmi/bugs/52/

sandis.neilands Yes, that is the plan. In any case it will take time until they have a new release and it will take even longer time until that release lands in distros. Until then - if you are building from source you might as well change these lines to suit your use case.

zalex_ua Lines in the patch mentions a bit different variables than you mentioned above. Could you check that ?

Also, looking to recent version of sources OpenIPMI-2.0.21.tar.gz with date 2014-01-28 AND taking into account that the existing patch reported 2010-04-19 AND that, for example, in Debian SID packaged version is still 2.0.16 - we can only hope that our children will get the feature in libopenipmi and then in Zabbix
https://packages.debian.org/sid/openipmi

sandis.neilands Yes, the issue that you mentioned deals with this exact problem but only in another type of interface.

sandis.neilands Opened a feature request in OpenIPMI tracker to make the low-level LAN interface timeouts tunable.

Comment by Sandis Neilands (Inactive) [ 2015 Nov 06 ]

We should document the following OpenIPMI behaviours:

  • low-level timeouts;
  • session inactivity timeout handling (e.g. that there are no periodic Activate Session commands towards BMC).

martins-v I tried to add some of the details mentioned here to:

Please review.

Reviewed by sandis.neilands.

Copied to:

CLOSED.





[ZBX-7847] zabbix server continues polling disabled ipmi hosts Created: 2014 Feb 20  Updated: 2017 May 30  Resolved: 2015 Sep 08

Status: Closed
Project: ZABBIX BUGS AND ISSUES
Component/s: Proxy (P), Server (S)
Affects Version/s: 2.2.2
Fix Version/s: 3.0.0alpha2

Type: Incident report Priority: Blocker
Reporter: richlv Assignee: Unassigned
Resolution: Fixed Votes: 0
Labels: ipmi
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

openipmi 2.0.16 and 2.0.21


Issue Links:
Duplicate

 Description   

test scenario :
have two hosts, each with one ipmi item. verify that both work. disable item on one of those hosts.
observe how server (openipmi lib, probably) still connects to this host.

probably same happens if host is disabled, is in nodata maintenance and maybe some other scenario (like is deleted ?).

it does not seem to be reproducible with one ipmi host only - there must be another, even though they are not related.



 Comments   
Comment by Aleksandrs Saveljevs [ 2014 May 08 ]

ZBX-8188 might be related.

Comment by Igors Homjakovs (Inactive) [ 2015 Jun 30 ]

Fixed in svn://svn.zabbix.com/branches/dev/ZBX-7847

Comment by dimir [ 2015 Jul 29 ]

Yes, zabbix still connects to ipmi host even after the host is deleted. In result, if you delete ipmi host and then add it again you end up with 2 entities each connecting to ipmi host. And if you do that again you get 3 and so on.

Comment by dimir [ 2015 Jul 29 ]

(1) Question, should we delete ipmi connection when host is deleted? With the fix it will be deleted after one day (we check for inactive ipmi connection every hour and delete it if it's one day old).

dimir After discussion we decided to keep it that way but this has to be documented (see sub-issue 3 below).

CLOSED

Comment by dimir [ 2015 Jul 29 ]

(2) In current implementation we use integer to store auto-incremented ID of ipmi host. Should we worry about overflow if we add lots of ipmi hosts and server constantly running without restart?

dimir After discussion we decided that 4 billion should be enough for creating all ipmi connections during the working time of zabbix server or proxy.

CLOSED

Comment by dimir [ 2015 Jul 30 ]

(3) [D] If ipmi checks are not performed (by any reason: all host ipmi items disabled/notsupported, host disabled/deleted, host in maintenance etc.) the ipmi connection will be terminated from Zabbix server or proxy in one day (24 hours + 0..60 minutes). This needs to be documented.

igorsh Documented in https://www.zabbix.com/documentation/3.0/manual/config/items/itemtypes/ipmi

RESOLVED.

<richlv> 0..60 notation is not used often. also, where does the 60 second interval come ? if that's config cache update, that could be changed, and we should actually say so here.

what about older versions, do they keep on making connections for deleted/disabled hosts, too ? if so, we should document that.

<dimir> The 60 minutes comes from the fact that we check for inactive hosts once an hour. So 0..60 depends on the time when Zabbix server was started. Documentation updated.

RESOLVED

<richlv> oh, i was sure it's in seconds
how about simplifying it to "in 3 to 4 hours, depending on the time when Zabbix server was started" ?
also, let's add the information it to all versions instead - users of 2.2 are not that likely to check for this behaviour in 3.0 docs

<dimir>

https://www.zabbix.com/documentation/2.2/manual/config/items/itemtypes/ipmi
https://www.zabbix.com/documentation/2.4/manual/config/items/itemtypes/ipmi
https://www.zabbix.com/documentation/3.0/manual/config/items/itemtypes/ipmi

RESOLVED

<richlv> yay, that should help users a lot - thank you
CLOSED

Comment by dimir [ 2015 Jul 30 ]

Successfully tested. Please review my changes in r54613 and r54620.

Comment by richlv [ 2015 Aug 03 ]

(4) after some discussion on irc, i'd like to raise the fact that one day seems like a very long period. on one hand, we could imagine a user who sets up some initial ipmi monitoring that's too aggressive, disables or deletes that host when it starts killing the endpoint... but zabbix would keep on hammering that interface for 23 more hours.

another case could be a support call where zabbix is making ipmi connections, somebody could look at items - and no ipmi items would exist.

value cache was brought up as an example as that is checked for old items every 24 hours, but that's conceptually different as it is an internal thing and would not potentially impact other systems.

what are the potential drawbacks of lowering this time period ? what about ipmi items with interval longer than one day, how they be impacted ?

<dimir> I personally agree, but we already discussed it with sasha and he decided to keep it that way. I'm not sure it's worth making this "24 hours" value configurable, I'd think just lowering it to 3 hours would be much better. 3 as in often used 3 attempts to decide something is not responding.

sasha I agree to lowering this period to 3 hours

<dimir> RESOLVED in r55456

sasha Looks good! CLOSED

Comment by dimir [ 2015 Sep 08 ]

Fixed in pre-3.0.0alpha2 (r55473).





[ZBX-7190] IPMI interface: cannot connect to IPMI host: [33554436] Unknown error 33554436 Created: 2013 Oct 23  Updated: 2020 Sep 07  Resolved: 2015 Jul 28

Status: Closed
Project: ZABBIX BUGS AND ISSUES
Component/s: Proxy (P), Server (S)
Affects Version/s: None
Fix Version/s: None

Type: Incident report Priority: Minor
Reporter: Angelina Zainullina Assignee: Unassigned
Resolution: Duplicate Votes: 4
Labels: ipmi
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

zabbix server 2.0.8 on Debian 7, HP dl 365 g5


Attachments: Text File new 2.txt    
Issue Links:
Duplicate
duplicates ZBX-6139 IPMI checks not working when using Op... Closed

 Description   

I have zabbix server 2.0.8 on my debian 7. I monitor HP dl 365 g5 using IPMI agent. In zabbix frontend my host is red and zabbix get error: cannot connect to IPMI host: [33554436] Unknown error 33554436 or cannot connect to IPMI host: [33554436] Unknown error 33554433. Ipmitool (ipmitool I lanplus -L user -H xxx -U xxx ) return normal values all sensors are work and available. Traffic on UDP 623 come in and go out. In zabbix_server.conf StartIPMIPollers=1 (another values of this parameter I try on too). zabbix.log is attached
Thank you for attantion and a help!



 Comments   
Comment by Fabian Portmann [ 2014 Mar 04 ]

My Zabbix Version is 2.2.2 and I'm monitoring some HP DL380p Gen8 Servers with iLO 4.
The problem occurs with newer iLO firmware.
With firmware Version 1.13 everything works fine and I can monitor my host using IPMI.
With firmware Version 1.32 and 1.40 I get the error 33554436.
Try to up- or downgrade your iLO firmware.

Comment by Fabian Portmann [ 2014 Jul 10 ]

dimir added the following comment on https://support.zabbix.com/browse/ZBX-6139

"By default OpenIPMI is compiled with OpenSSL enabled which enables working with any privilege level: "callback" (1), "user" (2), "operator" (3), or "admin" (4).

Unfortunately Debian (and Ubuntu) OpenIPMI packages are compiled with ssl turned off which leads to only one available privilege level - “callback”. This is the explanation by Debian package maintainer Noèl Köthe:

<noel> http://www.openssl.org/support/faq.html#LEGAL2
<noel> https://en.wikipedia.org/wiki/OpenSSL#Licensing

The solution is recompiling OpenIPMI library with openssl enabled."

Another working solution is to install the openipmi package from: https://launchpad.net/~vvk/+archive/ubuntu/openipmi

Comment by dimir [ 2014 Jul 21 ]

Are the privilege levels set up the same way on different firmwares?

Comment by Stefan [ 2014 Sep 04 ]

Hello,

i reported Bug ZBX-8681.
i used diffenrent privileg levels but it was always the same error

Comment by Nicola Worthington [ 2015 Jun 24 ]

I am also seeing this.

 27783:20150624:152845.056 query [txnlev:1] [update hosts set ipmi_disable_until=1435156185,ipmi_error='cannot connect to IPMI host: [33554436] Unknown error 33554436' where hostid=10324]

Zabbix server 1:2.4.5-1+trusty is on Ubuntu Trusty 14.04 LTS trying to communicate with an HP c7000 BladeSystem iLO4 (firmware 2.10 Jan 15 2015).

Testing from the command line on the Zabbix server works okay. Experimenting with different authentication algorithms of privilege levels doesn't appear to help.

For example, the following works on the command line from the Zabbix server:

ipmitool -I lanplus -H remote_ilom_hostname -U zabbixuser -P zabbixuser_password -L USER sensor
Comment by Angelina Zainullina [ 2015 Jun 25 ]

We solved this problem installing
libopenipmi0 2.0.16-1.3
openipmi 2.0.16-1.3.
Another versions of soft do not work.

Comment by dimir [ 2015 Jul 28 ]

Closing as duplicate, see explanation in ZBX-6139.





[ZBX-6812] [IPMI agent] Error while getting value on HP ILO 3 Created: 2013 Jul 22  Updated: 2019 Dec 10

Status: Reopened
Project: ZABBIX BUGS AND ISSUES
Component/s: Server (S)
Affects Version/s: 2.0.6
Fix Version/s: None

Type: Incident report Priority: Trivial
Reporter: Philippe B. Assignee: Unassigned
Resolution: Unresolved Votes: 1
Labels: ilo, ipmi
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

Centos 6.4 on HP DL 360 G7


Issue Links:
Duplicate
duplicates ZBXNEXT-300 Add support for ipmi discrete sensor ... Closed

 Description   

Hi,

When zabbix request whit IPMI an HP ILO 3, some analogic sensors are view as discrete.

For exemple with the sensor Fan 1 :

$ ipmitool -H myilointerface -U user -P password -I lanplus sdr get "Fan 1"
Sensor ID : Fan 1 (0x6)
Entity ID : 7.1 (System Board)
Sensor Type (Analog) : Fan
Sensor Reading : 29.400 (+/- 0) unspecified
Status : Lower Non-Critical
Positive Hysteresis : Unspecified
Negative Hysteresis : Unspecified
Minimum sensor range : Unspecified
Maximum sensor range : Unspecified
Event Message Control : Entire Sensor Only
Readable Thresholds : No Thresholds

I get this result in the log file debug=3 :
30260:20130722:125556.710 item [myilointerface:ilo.fan.1] became not supported: discrete sensor is not supported

Discrete sensor like fans are correctly view as discrete by zabbix
$ ipmitool -H myilointerface -U user -P password -I lanplus sdr get Fans
Sensor ID : Fans (0xc)
Entity ID : 7.7 (System Board)
Sensor Type (Discrete): Fan
Sensor Reading : 0h
Event Message Control : Entire Sensor Only
States Asserted : Redundancy State
[Fully Redundant]



 Comments   
Comment by Alexei Vladishev [ 2013 Jul 23 ]

Zabbix 2.2 does not support discrete sensors. This functionality is coming in 2.2, you may consider to try alpha version, which is already available for download.

Comment by Oleksii Zagorskyi [ 2013 Jul 23 ]

Alexei probably meant that 2.0 does not support discrete sensors.
It has been implemented in ZBXNEXT-300

Comment by Philippe B. [ 2013 Jul 23 ]

If you look exactly the probleme is not the sames as the ticket 300 for supporting discrete sensor.

Zabbix consider that an analogic sensor on HP ILO that give standard numeric value (Fan 1 that give the speed rotation in % or Power 1 that give the power consumption in Watt) is a discrete sensor.

I give the detail in my first dialog about the direct difference between discrete and not discrete.

Comment by richlv [ 2013 Jul 24 ]

which libopenipmi version has zabbix been compiled against ?

Comment by Philippe B. [ 2013 Jul 24 ]

We use the package from zabbix.com that can be found here : http://repo.zabbix.com/zabbix/2.0/rhel/6/x86_64/

It need OpenIPMI-lib 2.0.14 as min version.

It seams to use the version 2.0.16 installed on our server and zabbix-server use this lib as dynamic.





[ZBX-6381] IPMI action fails at HP iLO Created: 2013 Mar 12  Updated: 2019 Dec 10

Status: Open
Project: ZABBIX BUGS AND ISSUES
Component/s: Server (S)
Affects Version/s: 2.0.4, 2.0.5
Fix Version/s: None

Type: Incident report Priority: Trivial
Reporter: Yuki Yamashita Assignee: Unassigned
Resolution: Unresolved Votes: 0
Labels: ipmi, remotecommands
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

OS: CentOS6.3 x86_64
Zabbix 2.0.5
OpenIPMI-libs-2.0.16-12.el6.x86_64
OpenIPMI-2.0.16-12.el6.x86_64
OpenIPMI-devel-2.0.16-12.el6.x86_64

HP DL360 G7



 Description   

Action -> Operations -> Remote Command (Type IPMI)
Command: power reset
It looks like power on command is not being passed correctly

Configure and run an error is logged to the following
log: /var/log/zabbix/zabbix_server.log

24797:20121203:112357.505 In substitute_simple_macros() data:'reset on'
24797:20121203:112357.505 In zbx_execute_script()
24797:20121203:112357.505 In zbx_execute_ipmi_command()
24797:20121203:112357.505 In substitute_simple_macros() data:'623'
24797:20121203:112357.505 In parse_ipmi_command() command:'reset on'
24797:20121203:112357.505 End of parse_ipmi_command():SUCCEED
24797:20121203:112357.505 In set_ipmi_control_value(control:reset, value:1)
24797:20121203:112357.505 In init_ipmi_host() host:'[192.168.113.250]:623'
24797:20121203:112357.505 In get_ipmi_host() host:'[192.168.113.250]:623'
24797:20121203:112357.505 End of get_ipmi_host():0xf94720
24797:20121203:112357.505 End of init_ipmi_host():0xf94720
24797:20121203:112357.505 In get_ipmi_control_by_name() reset@[192.168.113.250]:623
24797:20121203:112357.505 End of get_ipmi_control_by_name():(nil)
24797:20121203:112357.505 control reset@[192.168.113.250]:623 does not exist ?
24797:20121203:112357.505 End of zbx_execute_ipmi_command():FAIL
24797:20121203:112357.505 End of zbx_execute_script():FAIL
24797:20121203:112357.505 In add_command_alert()
24797:20121203:112357.505 In DCget_nextid() table:'alerts' num:1
24797:20121203:112357.505 End of DCget_nextid() table:'alerts' [30:30]
24797:20121203:112357.505 query [txnlev:1] [insert into alerts (alertid,actionid,eventid,clock,message,status,error,alerttype,esc_step) values (30,4,243,1354
501437,'admin1:reset on',2,'control reset@[192.168.113.250]:623 does not exist',1,1)]
24797:20121203:112357.505 End of add_command_alert()
24797:20121203:112357.505 End of execute_commands()
24797:20121203:112357.505 End of execute_operations()
24797:20121203:112357.505 End of execute_escalation()
24797:20121203:112357.505 query [txnlev:1] [update escalations set status=0,esc_step=1,nextcheck=1354505037 where escalationid=3;



 Comments   
Comment by richlv [ 2013 Mar 12 ]

hmm, comment says "power reset", but log shows "reset on"

why so ?

Comment by Yuki Yamashita [ 2013 Mar 12 ]

Sorry.
It was a mistake

Command: reset on

Comment by richlv [ 2019 Jan 24 ]

The relevant error message seems to be:

control reset@[192.168.113.250]:623 does not exist

The comment actually has two versions of the command listed: "power reset" and "power on".
The log shows that "reset on" is not available.

If this problem is still relevant, could you please clarify what exactly is done and what exactly happens?





[ZBX-6139] IPMI checks not working when using OpenIPMI library from Debian/Ubuntu package. Created: 2013 Jan 16  Updated: 2020 Sep 11  Resolved: 2015 Oct 29

Status: Closed
Project: ZABBIX BUGS AND ISSUES
Component/s: Documentation (D)
Affects Version/s: None
Fix Version/s: None

Type: Documentation task Priority: Trivial
Reporter: dimir Assignee: Unassigned
Resolution: Fixed Votes: 1
Labels: debian, ipmi, openssl, privilege
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

Debian/Ubuntu/Mint


Issue Links:
Duplicate
is duplicated by ZBX-8681 IPMI: cannot connect to IPMI host: [3... Closed
is duplicated by ZBX-7190 IPMI interface: cannot connect to IPM... Closed

 Description   

$ dpkg -l 'openipmi' | grep '^ii'
ii libopenipmi-dev 2.0.16-1.3 Intelligent Platform Management Interface - development
ii libopenipmi0 2.0.16-1.3 Intelligent Platform Management Interface - runtime
ii openipmi 2.0.16-1.3 Intelligent Platform Management Interface (for servers)

IPMI monitoring not available due to an error in the server log (with DebugLevel=4):

EINF: 0 ipmi_lan.c(got_rmcpp_open_session_rsp): Expected privilege 2, got 1

The error message "Expected privilege 2, got 1" means that we were requesting privilege level "user" (2) but got only "callback" (1) available. However with the same user and ipmitool the connection works:

ipmitool -I lan -H 192.168.3.133 -U foo -P bar -L user sensor

BB +1.8V SM | 1.793 | Volts | ok | na | 1.597 | 1.646 | 1.960 | 2.019 | na
BB +3.3V | 3.302 | Volts | ok | na | 2.876 | 2.970 | 3.618 | 3.729 | na



 Comments   
Comment by dimir [ 2013 Jan 16 ]

By default OpenIPMI is compiled with OpenSSL enabled which enables working with any privilege level: "callback" (1), "user" (2), "operator" (3), or "admin" (4).

Unfortunately Debian (and Ubuntu) OpenIPMI packages are compiled with ssl turned off which leads to only one available privilege level - “callback”. This is the explanation by Debian package maintainer Noèl Köthe:

<noel> http://www.openssl.org/support/faq.html#LEGAL2
<noel> https://en.wikipedia.org/wiki/OpenSSL#Licensing

The solution is recompiling OpenIPMI library with openssl enabled.

Comment by dimir [ 2015 Jul 28 ]

Steps to fix on Debian, Ubuntu, Mint:

  1. cd /tmp
  2. apt-get source openipmi
  3. cd openipmi-2*
  4. edit file debian/rules
  5. find ./configure in section "config.status:"
      CFLAGS="$(CFLAGS)" ./configure --host=$(DEB_HOST_GNU_TYPE) \
    add --with-openssl to ./configure:
      CFLAGS="$(CFLAGS)" ./configure --with-openssl --host=$(DEB_HOST_GNU_TYPE) \
    also remove --without-openssl if that exists
  6. Optional: if you want to debug openipmi library then comment out dh_strip, dh_compress lines.
  7. save file
  8. sudo dpkg-buildpackage
  9. cd ..
  10. sudo dpkg -i openipmi*.deb libopenipmi*.deb
  11. restart zabbix server
Comment by Nicola Worthington [ 2015 Jul 31 ]

There is a '--without-openssl' statement a couple of lines later. It needs to be noted that this line must be removed (or changed of course) as well.

<dimir> Thanks nicolaw, will update the instructions.

Comment by Oleksii Zagorskyi [ 2015 Sep 04 ]

We could document it in a similar way as we did it in ZBX-4850 for ssh checks.
Reopening to consider this suggestion by martins-v

martins-v Noted in IPMI check documentation for 3.0. Please review before copying to other doc branches.

zalex_ua looks good for me.

martins-v After discussing with sandis.neilands and richlv we decided to collect such third-party issues into the known issue section and link to them from affected sections (such as IPMI in this case).

martins-v Documented in known issues for 2.0, 2.2, 2.4 and 3.0. Linked to from IPMI pages.

Comment by Sandis Neilands (Inactive) [ 2015 Oct 14 ]

Just spent the better part of the day debugging the same issue on Mint and coming up with the same solution as dimir just to find this ZBX in Google when trying to find out why Debian broke OpenIPMI.

We definitely have to warn our users about this since neither Debian nor downstream projects felt necessary to warn their users about this.

Comment by Oleksii Zagorskyi [ 2015 Oct 14 ]

the same was with me when I wrote previous comment ...





[ZBX-6138] IPMI monitoring with zabbix requires admin privilege level with iLO otherwise connection to failed Created: 2013 Jan 16  Updated: 2017 May 30  Resolved: 2013 Jan 16

Status: Closed
Project: ZABBIX BUGS AND ISSUES
Component/s: Server (S)
Affects Version/s: 2.0.3
Fix Version/s: None

Type: Incident report Priority: Minor
Reporter: Alexey Pereklad Assignee: Unassigned
Resolution: Duplicate Votes: 0
Labels: freebsd, ipmi
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

Zabbix server 2.0.4 works on FreeBSD 8.1-RELEASE-p1, HP Proliant DL380 G5, iLO v.2.12.



 Description   

Can't connect to iLO IPMI with privilege level other than Administrator, while ipmitool works fine with any privilage level. But there is no problem in zabbix to connect with user privilege level to DRAC IPMI interface. There a few posts on forum with the same problem and iLO and no solution (except setting admin level). Looks like this problem specific only for zabbix and iLO.

If I set user level for iLO IPMI connection, I've got red IPMI icon in frontend and error: ""cannot connect to IPMI host: [22] Invalid argument". Also there record in logs like this: "Expected privilege 2, got 4"



 Comments   
Comment by richlv [ 2013 Jan 16 ]

duplicate of ZBX-6077 (a combination of a bug in hp ilo and a bug in openipmi)





[ZBX-6077] Can not fetch IPMI values if privilege is less than "Admin" Created: 2013 Jan 08  Updated: 2018 Nov 08  Resolved: 2013 Jan 15

Status: Closed
Project: ZABBIX BUGS AND ISSUES
Component/s: Server (S)
Affects Version/s: 2.0.3
Fix Version/s: None

Type: Incident report Priority: Minor
Reporter: Andrej Kacian Assignee: Unassigned
Resolution: Won't fix Votes: 1
Labels: ipmi
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

RHEL 6.3 amd64, HP ProLiant 685c G7 with iLO3 remote management interface


Issue Links:
Duplicate
is duplicated by ZBX-8310 IPMI dont work, and strange error mes... Closed

 Description   

I have a HP ProLiant 685c G7 with iLO3 that I need to monitor over IPMI.

If I configure Zabbix to use account with Administrator-level access, it works just fine.
If I create an iLO3 account with only User-level access and configure Zabbix to use this account instead, it stops working:
[IP address and hostname edited out]

26615:20130108:120809.007 In substitute_key_macros() data:'ipmi.proliant.Inlet_Ambient'
26615:20130108:120809.007 End of substitute_key_macros():SUCCEED data:'ipmi.proliant.Inlet_Ambient'
26615:20130108:120809.007 In get_value() key:'ipmi.proliant.Inlet_Ambient'
26615:20130108:120809.007 In get_value_ipmi() key:'myhost:ipmi.proliant.Inlet_Ambient'
26615:20130108:120809.007 In init_ipmi_host() host:'[x.x.x.x]:623'
26615:20130108:120809.007 In get_ipmi_host() host:'[x.x.x.x]:623'
26615:20130108:120809.007 End of get_ipmi_host():(nil)
26615:20130108:120809.007 In allocate_ipmi_host() host:'[x.x.x.x]:623'
26615:20130108:120809.007 End of allocate_ipmi_host():0x2059f30
26615:20130108:120809.012 EINF: 0 ipmi_lan.c(got_rmcpp_open_session_rsp): Expected privilege 2, got 4
26615:20130108:120809.013 End of init_ipmi_host():0x2059f30
26615:20130108:120809.013 Item [myhost:ipmi.proliant.Inlet_Ambient] error: cannot connect to IPMI host: [22] Invalid argument
26615:20130108:120809.013 query [txnlev:1] [update hosts set ipmi_error='cannot connect to IPMI host: [22] Invalid argument',ipmi_disable_until=1357646949 where hostid=10099]

Same thing happens if I use the account with Admin-level access, but select "user" privilege in Zabbix host configuration.

Access via Linux utility 'ipmitool' works just fine using the account with User-level access:
myhost/root)#ipmitool -H myhost -U zabbix -I lanplus -L USER sdr
Password:
UID Light | 0 unspecified | ok
Health LED | 0 unspecified | ok
VRM 1 | 0 unspecified | cr
VRM 2 | 0 unspecified | cr
[...]

I would prefer Zabbix to work with restricted accounts, following the common "least required privilege" security concept.



 Comments   
Comment by dimir [ 2013 Jan 09 ]

In our internal system when using OpenIPMI the connection to the same IPMI device works from some hosts and gets this error from other. Both have the same openipmi package versions, both are Debian squeeze. Using ipmitool it's possible to get sensor values from both hosts (ipmitool does not use OpenIPMI, Zabbix server does).

How to test it using OpenIPMI:

1) install package openipmi (Debian-based: aptitude install openipmi)
2) run: ipmi_ui lan <IP> <port> <auth> <level> <user> <pass> (e. g.: ipmi_ui lan 192.168.1.10 623 straight user myuser mypassword)
3) in the right window it will show the error:

│1357739333.331791: Starting setup, wait until complete before entering commands.
│1357739333.508663: EINF: first 0 ipmi_lan.c(got_rmcpp_open_session_rsp): Expected privilege 2, got 1
│1357739333.509021: IPMI connection to con.port 0.0 is down  due to error 0x16
│1357739333.509213: All IPMI connections down
[...]
Comment by dimir [ 2013 Jan 15 ]

This is not a Zabbix bug, but OpenIPMI:

http://www.mail-archive.com/[email protected]/msg01759.html

So in your case set the authentication level to "admin" and it should work. Alternatively you can patch your OpenIPMI library as they haven't fixed it yet (I have checked the latest version 2.0.20-rc1).

Closing as "Won't Fix" feel free to reopen if you disagree.

Comment by Eugene Varnavsky [ 2016 Feb 12 ]

Is it fixed already?

Comment by dimir [ 2016 Feb 25 ]

According to their repository, no, it is not fixed yet:

https://sourceforge.net/p/openipmi/code/ci/master/tree/lib/ipmi_lan.c

Comment by dimir [ 2016 Feb 25 ]

Related issue: ZBX-6139

Comment by Anton Samets [ 2018 Nov 08 ]

Finally this patch was accepted in OpenIPMI in 2.0.24 version
https://sourceforge.net/p/openipmi/code/ci/2b2b6741e178f245d795351ceee9f8c77a06f166/





[ZBX-4823] unreachable pollers may "hang" when are doing ipmi checks Created: 2012 Apr 02  Updated: 2017 Jul 14  Resolved: 2017 Jul 14

Status: Closed
Project: ZABBIX BUGS AND ISSUES
Component/s: Proxy (P), Server (S)
Affects Version/s: 1.8.11
Fix Version/s: None

Type: Incident report Priority: Major
Reporter: Alex Vorona Assignee: Unassigned
Resolution: Fixed Votes: 2
Labels: hang, ipmi, unreachable
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

CentOS 6.2 64bit


Attachments: Text File full-backtrace.txt     Text File strace-filtered.txt     Text File zabbix_server-filtered.log    
Issue Links:
Duplicate
Sub-task
part of ZBXNEXT-3386 IPMI connection to a single device is... Closed
part of ZBX-10983 zabbix server 3.0 crash - ipmi problem Closed

 Description   

I've got zabbix server hang with following backtrace (from gcore) and strace

Backtrace

Core was generated by `zabbix_server_pgsql'.
#0  0x00007fed9cd15893 in __select_nocancel () from /lib64/libc.so.6
Missing separate debuginfos, use: debuginfo-install zabbix-server-pgsql-1.8.11-0.x86_64
(gdb) bt full
#0  0x00007fed9cd15893 in __select_nocancel () from /lib64/libc.so.6
No symbol table info available.
#1  0x00007fed9d8c267d in ?? () from /usr/lib64/libOpenIPMIposix.so.0
No symbol table info available.
#2  0x00007fed9d8c2c5e in sel_select () from /usr/lib64/libOpenIPMIposix.so.0
No symbol table info available.
#3  0x00007fed9d8c057c in ?? () from /usr/lib64/libOpenIPMIposix.so.0
No symbol table info available.
#4  0x00000000004173e3 in init_ipmi_host ()
No symbol table info available.
#5  0x0000000000418412 in get_value_ipmi ()
No symbol table info available.
#6  0x000000000041a350 in get_values ()
No symbol table info available.
#7  0x000000000041ab4f in main_poller_loop ()
No symbol table info available.
#8  0x000000000041199d in MAIN_ZABBIX_ENTRY ()
No symbol table info available.
#9  0x0000000000440297 in daemon_start ()
No symbol table info available.
#10 0x00007fed9cc55cdd in __libc_start_main () from /lib64/libc.so.6
No symbol table info available.
#11 0x000000000040dd19 in _start ()
No symbol table info available.
(gdb) quit

strace

     0.000000 select(1, [0], [], [], {7, 858908}) = -1 EBADF (Bad file descriptor)
     0.000077 select(1, [0], [], [], {7, 858825}) = -1 EBADF (Bad file descriptor)
     0.000042 select(1, [0], [], [], {7, 858782}) = -1 EBADF (Bad file descriptor)
     0.000041 select(1, [0], [], [], {7, 858741}) = -1 EBADF (Bad file descriptor)
     0.000074 select(1, [0], [], [], {7, 858674}) = -1 EBADF (Bad file descriptor)
     0.000050 select(1, [0], [], [], {7, 858617}) = -1 EBADF (Bad file descriptor)
     0.000041 select(1, [0], [], [], {7, 858575}) = -1 EBADF (Bad file descriptor)
     0.000039 select(1, [0], [], [], {7, 858536}) = -1 EBADF (Bad file descriptor)
     0.000038 select(1, [0], [], [], {7, 858498}) = -1 EBADF (Bad file descriptor)
     0.000039 select(1, [0], [], [], {7, 858459}) = -1 EBADF (Bad file descriptor)
     0.000064 select(1, [0], [], [], {7, 858395}) = -1 EBADF (Bad file descriptor)
     0.000040 select(1, [0], [], [], {7, 858355}) = -1 EBADF (Bad file descriptor)
     0.000039 select(1, [0], [], [], {7, 858316}) = -1 EBADF (Bad file descriptor)
     0.000038 select(1, [0], [], [], {7, 858278}) = -1 EBADF (Bad file descriptor)
     0.000039 select(1, [0], [], [], {7, 858239}) = -1 EBADF (Bad file descriptor)
     0.000039 select(1, [0], [], [], {7, 858199}) = -1 EBADF (Bad file descriptor)
     0.000039 select(1, [0], [], [], {7, 858161}) = -1 EBADF (Bad file descriptor)
     0.000039 select(1, [0], [], [], {7, 858121}) = -1 EBADF (Bad file descriptor)
     0.000040 select(1, [0], [], [], {7, 858082}) = -1 EBADF (Bad file descriptor)

lsof tail

zabbix_se 18214 zabbix  DEL    REG                0,4            753667 /SYSV7801030c
zabbix_se 18214 zabbix  DEL    REG                0,4            884743 /SYSV5301030c
zabbix_se 18214 zabbix    1w   REG                9,2 93959114   262695 /var/zabbix/zabbix_server.log
zabbix_se 18214 zabbix    2w   REG                9,2 93959114   262695 /var/zabbix/zabbix_server.log
zabbix_se 18214 zabbix    3w   REG                9,2        5   524521 /var/run/zabbix/zabbix.pid
zabbix_se 18214 zabbix    4u  IPv4           57254714      0t0      TCP *:zabbix-trapper (LISTEN)
zabbix_se 18214 zabbix    5u  unix 0xffff88010e921c80      0t0 57254932 socket

Here is log tail for this pid

 18214:20120327:075815.715 server #33 started [unreachable poller #1]

 18214:20120402:145629.524 temporarily disabling Zabbix agent checks on host [xxx]: host unavailable
 18214:20120402:145936.829 IPMI item [Analog_Fan_RPM[FAN 1]] on host [xxx] failed: another network error, wait for 15 seconds
 18214:20120402:145939.831 IPMI item [Analog_Fan_RPM[Fan1]] on host [xxx] failed: another network error, wait for 15 seconds
 18214:20120402:145951.955 temporarily disabling IPMI checks on host [xxx]: host unavailable
 18214:20120402:145955.697 resuming IPMI checks on host [xxx]: connection restored
 18214:20120402:162901.379 Got signal [signal:15(SIGTERM),sender_pid:21889,sender_uid:0,reason:0]. Exiting ...

Thanks,
Alex



 Comments   
Comment by Alex Vorona [ 2012 Apr 25 ]

I'm still hitting this bug, now for 3 processes instead of 1

Comment by richlv [ 2012 Apr 26 ]

hmm, so is it a hang or a crash ? does server shut down in this case ?

Comment by Alex Vorona [ 2012 Apr 26 ]

>hmm, so is it a hang
yes. Unreachable poller process(es) are eating CPU in endless cycle. Zabbix queue grows. But Zabbix at all still works due to some count of still working unreachable poller processes.
> or a crash
no
> does server shut down in this case
no.

Comment by richlv [ 2012 Apr 26 ]

which libopenipmi version is zabbix server compiled against ?

Comment by Alex Vorona [ 2012 Apr 26 ]

OpenIPMI-2.0.16-12.el6.x86_64
OpenIPMI-libs-2.0.16-12.el6.x86_64
OpenIPMI-devel-2.0.16-12.el6.x86_64

Comment by Alexei Vladishev [ 2012 Aug 29 ]

Do you still have this issues with the latest 1.8.x or 2.0.x?

Comment by Alex Vorona [ 2012 Aug 29 ]

Just updated to 1.8.15 and re-enabled IPMI again. Keep an eye on it.

Comment by Alexei Vladishev [ 2012 Sep 21 ]

Alex, any new server hangs or crashes?

Comment by Alex Vorona [ 2012 Sep 21 ]

No hangs were detected, I had checked log - no crashes. Seems the problem has been resolved.

Comment by Alex Vorona [ 2012 Sep 29 ]

Well, I got a hangs of 4 of 5 unreachable pollers today on 1.8.15, same strace and backtrace for all four, as before.

(gdb) bt full
#0 0x00007f5f7724fce3 in __select_nocancel () from /lib64/libc.so.6
No symbol table info available.
#1 0x00007f5f77dfd67d in ?? () from /usr/lib64/libOpenIPMIposix.so.0
No symbol table info available.
#2 0x00007f5f77dfdc5e in sel_select () from /usr/lib64/libOpenIPMIposix.so.0
No symbol table info available.
#3 0x00007f5f77dfb57c in ?? () from /usr/lib64/libOpenIPMIposix.so.0
No symbol table info available.
#4 0x00000000004175c3 in init_ipmi_host ()
No symbol table info available.
#5 0x00000000004185f2 in get_value_ipmi ()
No symbol table info available.
#6 0x000000000041a530 in get_values ()
No symbol table info available.
#7 0x000000000041ad2f in main_poller_loop ()
No symbol table info available.
#8 0x0000000000411a6d in MAIN_ZABBIX_ENTRY ()
No symbol table info available.
#9 0x00000000004402c7 in daemon_start ()
No symbol table info available.
#10 0x00007f5f7718dcdd in __libc_start_main () from /lib64/libc.so.6
No symbol table info available.
#11 0x000000000040dde9 in _start ()
No symbol table info available.

Comment by Alex Vorona [ 2012 Sep 29 ]

And I got hang with just re-started zabbix server again with 2 lines related to this PID:
14789:20120929:092105.824 server #63 started unreachable poller #1
14789:20120929:092425.842 enabling IPMI checks on host [xxx]: host became available

Comment by richlv [ 2013 Feb 22 ]

while it's been a while... does strace reveal anything interesting ?

Comment by Alex Vorona [ 2013 Feb 22 ]

>does strace reveal anything interesting ?
strace was the same as per 1'st post. I had just to disable IPMI on all hosts. We are in process of migrating to Zabbix 2, I will try to re-enable IPMI on Zabbix 2.

Comment by richlv [ 2013 Feb 22 ]

ok, thanks. it might also be worth checking with the latest version of openipmi.

closing the issue, please reopen is still reproducible

Comment by Oleksii Zagorskyi [ 2015 Aug 05 ]

I'm reopening current issue to investigate it as there is a zabbix installation v 2.4.5 where it's reproducible.

Comment by Oleksii Zagorskyi [ 2015 Aug 05 ]

I'm attaching filterd zabbix_server.log with debug mode risen for single running unreachable poller which catches the problem.
In the log file are two sections: one success for "mcu-r1" host, and another failed for "mcu-r2" host.

strace results for the unreachable poller was also captured at the same time. I trimmed it too - from a point where the failed attempt started.

Notice please my in line comments in the files.

What is also collected:
The distro and installed packages have been just recently updated to RHEL 6.7 (Santiago).
Picture was the same on RHEL 6.5 (Santiago) (2.6.32-358.14.1.el6.x86_64 #1 SMP Mon Jun 17 15:54:20 EDT 2013 x86_64 x86_64 x86_64 GNU/Linux)


Was and has left the same after the upgrade:
# rpm -qa | grep -i ipmi
 ipmitool-1.8.11-28.el6.x86_64
 OpenIPMI-libs-2.0.16-14.el6.x86_64
 OpenIPMI-2.0.16-14.el6.x86_64
 OpenIPMI-devel-2.0.16-14.el6.x86_64
 ipmiutil-devel-2.8.2-1.el6.x86_64
 ipmiutil-2.8.2-1.el6.x86_64

Later, for several "hanged" unreachable pollers:
shell output of commands:

# gcore 25807
[Thread debugging using libthread_db enabled]
 0x0000003ecdee1403 in __select_nocancel () from /lib64/libc.so.6
 Saved corefile core.25807

# gcore 8881
[Thread debugging using libthread_db enabled]
 0x00000032af8e15c3 in __select_nocancel () from /lib64/libc.so.6
 Saved corefile core.8881

Ideas how to troubleshoot it next - are welcome.

Comment by Andris Mednis [ 2015 Aug 05 ]

A similar problem has been described for PostgreSQL (in 2006) waiting in __select_nocancel() at
http://postgresql.nabble.com/cvs-tip-stats-buffer-process-consuming-100-cpu-td1939377.html

Comment by Andris Mednis [ 2015 Aug 05 ]

If possible try with newer OpenIPMI, e.g. 2.0.21. OpenIPMI changelog mentions a change "to allow software using library to see "connection timeout" error, that happens when BMC indeed does not respond.", introduced for 2.0.20 on 2013-08-27.

Comment by Vladimir Kamarzin [ 2015 Sep 04 ]

I'm getting the same issue on zabbix 2.4.6. Unreacheble pollers taking 100% CPU doing

select(1, [0], [], [], {3, 850611})     = -1 EBADF (Bad file descriptor)
select(1, [0], [], [], {3, 850589})     = -1 EBADF (Bad file descriptor)
select(1, [0], [], [], {3, 850567})     = -1 EBADF (Bad file descriptor)
select(1, [0], [], [], {3, 850545})     = -1 EBADF (Bad file descriptor)

This happens suddenly on 2.4.5. Before this IPMI checks worked without problem for more than a year.

# dpkg -l |grep ipmi
ii  ipmitool                         1.8.11-5ubuntu1                   utility for IPMI control with kernel driver or LAN interface
ii  libopenipmi0                     2.0.18-0ubuntu3.1ppa1             Intelligent Platform Management Interface - runtime
ii  openipmi                         2.0.18-0ubuntu3.1ppa1             Intelligent Platform Management Interface (for servers)
Comment by Oleksii Zagorskyi [ 2015 Sep 04 ]

What I have is a feedback from a production system that decreasing StartIPMIPollers to just 1 has resolved the issue.
So it requires more investigation.

Comment by Imri Zvik [ 2016 Jan 16 ]

I see this on Zabbix 2.2.11

Comment by Alexander Vladishev [ 2016 Aug 16 ]

Reproducible in 3.0.5rc1 r61474. OpenIPMI version: 2.0.18.

strace

...
select(1, [0], [], [], {6, 377766})     = -1 EBADF (Bad file descriptor)
select(1, [0], [], [], {6, 377739})     = -1 EBADF (Bad file descriptor)
select(1, [0], [], [], {6, 377712})     = -1 EBADF (Bad file descriptor)
select(1, [0], [], [], {6, 377685})     = -1 EBADF (Bad file descriptor)
select(1, [0], [], [], {6, 377657})     = -1 EBADF (Bad file descriptor)
select(1, [0], [], [], {6, 377630})     = -1 EBADF (Bad file descriptor)
select(1, [0], [], [], {6, 377602})     = -1 EBADF (Bad file descriptor)
select(1, [0], [], [], {6, 377575})     = -1 EBADF (Bad file descriptor)
select(1, [0], [], [], {6, 377548})     = -1 EBADF (Bad file descriptor)
select(1, [0], [], [], {6, 377520})     = -1 EBADF (Bad file descriptor)
select(1, [0], [], [], {6, 377493})     = -1 EBADF (Bad file descriptor)
select(1, [0], [], [], {6, 377466})     = -1 EBADF (Bad file descriptor)
select(1, [0], [], [], {6, 377438})     = -1 EBADF (Bad file descriptor)
select(1, [0], [], [], {6, 377411})     = -1 EBADF (Bad file descriptor)
select(1, [0], [], [], {6, 377384})     = -1 EBADF (Bad file descriptor)
select(1, [0], [], [], {6, 377356})     = -1 EBADF (Bad file descriptor)
select(1, [0], [], [], {6, 377329})     = -1 EBADF (Bad file descriptor)
select(1, [0], [], [], {6, 377302})     = -1 EBADF (Bad file descriptor)
...

backtrace

#0  0x00007fadd7950ae3 in select () at ../sysdeps/unix/syscall-template.S:81
#1  0x00007fadd8a0702a in process_fds (sel=sel@entry=0x1de0c60, timeout=timeout@entry=0x7fff26345b80, cb_data=0x0, thread_id=0, send_sig=0x0) at selector.c:602
#2  0x00007fadd8a079e1 in sel_select (sel=0x1de0c60, send_sig=send_sig@entry=0x0, thread_id=thread_id@entry=0, cb_data=cb_data@entry=0x0, timeout=0x7fff26345c50) at selector.c:739
#3  0x00007fadd8a05aec in perform_one_op (os_hnd=<optimized out>, timeout=<optimized out>) at posix_os_hnd.c:357
#4  0x0000000000421365 in init_ipmi_host (ip=<optimized out>, port=<optimized out>, authtype=<optimized out>, privilege=<optimized out>, username=username@entry=0x7fff26348aae "*******", 
    password=password@entry=0x7fff26348abf "*******") at checks_ipmi.c:1152
#5  0x0000000000422732 in get_value_ipmi (item=0x7fff26348970, value=0x7fff26346970) at checks_ipmi.c:1273
#6  0x0000000000425528 in get_value (add_results=<optimized out>, result=<optimized out>, item=<optimized out>) at poller.c:434
#7  get_values (poller_type=poller_type@entry=1 '\001', nextcheck=0x7fff263459c0, nextcheck@entry=0x7fff264b95dc) at poller.c:663
#8  0x00000000004256f8 in poller_thread (args=<optimized out>) at poller.c:854
#9  0x000000000046c5e5 in zbx_thread_start (handler=0x425610 <poller_thread>, thread_args=thread_args@entry=0x7fff264b9650) at threads.c:128
#10 0x0000000000419275 in MAIN_ZABBIX_ENTRY (flags=flags@entry=0) at server.c:953
#11 0x000000000046b3a0 in daemon_start (allow_root=<optimized out>, user=<optimized out>, flags=0) at daemon.c:392
#12 0x0000000000414348 in main (argc=<optimized out>, argv=<optimized out>) at server.c:757

full-backtrace.txt

Comment by Pavel Smykov [ 2016 Oct 31 ]

This issue has affect in my Zabbix installation 3.2.1 on Centos 7.2.
Problem begins after I update Zabbix server to 3.2.1, and downgrade back to 3.2.0 didn't help.

Comment by Andris Mednis [ 2016 Dec 12 ]

How to reproduce:

  1. Configure an IPMI host with a sensor to be checked, get it working.
  2. Stop Zabbix server.
  3. Disconnect LAN cable from the Zabbix server machine.
  4. Start Zabbix server.
  5. 'ipmi poller' or 'unreachable poller' hangs when checking the sensor. Connect to hanging process with "gdb -p <PID>", observe "bt full" output.

strace shows looping on select():

select(8, [7], [], [], {9, 768419})     = 0 (Timeout)
sendto(7, "\6\0\377\7\0\0\0\0\0\0\0\0\0\t \30\310\201x8\216\2?", 23, 0, {sa_family=AF_INET, sin_port=htons(623), sin_addr=inet_addr("192.168.xxx.xxx")}, 16) = -1 ENETUNREACH (Network is unreachable)
sendto(7, "\6\0\377\7\0\0\0\0\0\0\0\0\0\7 \30\310\201|\1\2", 21, 0, {sa_family=AF_INET, sin_port=htons(623), sin_addr=inet_addr("192.168.xxx.xxx")}, 16) = -1 ENETUNREACH (Network is unreachable)
select(8, [7], [], [], {0, 0})          = 0 (Timeout)
select(8, [7], [], [], {9, 999973})     = 0 (Timeout)
sendto(7, "\6\0\377\7\0\0\0\0\0\0\0\0\0\t \30\310\201\2008\216\0027", 23, 0, {sa_family=AF_INET, sin_port=htons(623), sin_addr=inet_addr("192.168.xxx.xxx")}, 16) = -1 ENETUNREACH (Network is unreachable)
sendto(7, "\6\0\377\7\0\0\0\0\0\0\0\0\0\7 \30\310\201\204\1\372", 21, 0, {sa_family=AF_INET, sin_port=htons(623), sin_addr=inet_addr("192.168.xxx.xxx")}, 16) = -1 ENETUNREACH (Network is unreachable)
select(8, [7], [], [], {0, 0})          = 0 (Timeout)
select(8, [7], [], [], {9, 999905})     = 0 (Timeout)

Sorry, I got only simple process hanging (still a bug) , but no "-1 EBADF (Bad file descriptor)".

Comment by Aleksandrs Saveljevs [ 2016 Dec 14 ]

(1) Unrelated to this task, but we call zbx_init_ipmi_handler() in main() of server and proxy. It would be nice to call it only for IPMI and unreachable pollers.

Similarly, we call zbx_free_ipmi_handler() in zbx_on_exit() of server and proxy, but only in the main process. That does not seem to make a lot of sense.

wiper Agreed. Initialization was fixed in ZBXNEXT-3386. Regarding zbx_free_ipmi_handler() - it's not a good idea to call it from signal handler, so we can't don much until the child process termination is reworked.
CLOSED

Comment by Andris Zeila [ 2017 Feb 08 ]

There were fixes improving stability of IPMI checks (ZBX-10983) and how the IPMI checks are used (ZBXNEXT-3386). It might also have fixed this problem. We will monitor it and close If it doesn't repeat with current IPMI code.





[ZBX-4270] IPMI insufficient resource Created: 2011 Oct 25  Updated: 2017 May 30  Resolved: 2012 Nov 01

Status: Closed
Project: ZABBIX BUGS AND ISSUES
Component/s: Server (S)
Affects Version/s: 1.8.8
Fix Version/s: None

Type: Incident report Priority: Trivial
Reporter: hamid sfandiari Assignee: Unassigned
Resolution: Duplicate Votes: 0
Labels: ipmi
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

FreeBSD , ipmitool-1.8.11_2, openipmi-2.0.18_4, DL 360 G7 ilo v3.0


Attachments: PNG File IPMI item graph.png    
Issue Links:
Duplicate
duplicates ZBX-3243 Network error while retrieving IPMI data Closed

 Description   

I've set 3 items for my server to get sensor data each 30 seconds but it Frequently changed to not supported
zabbix_server.log:
8042:20111025:103124.814 Enabling IPMI host [zproxy01]
8282:20111025:103134.311 IPMI Host [zproxy01]: first network error, wait for 15 seconds
8281:20111025:103141.545 Disabling IPMI host [zproxy01]
8130:20111025:103225.023 Enabling IPMI host [zabbix]
8290:20111025:103242.032 IPMI Host [zabbix]: first network error, wait for 15 seconds
8295:20111025:103336.089 IPMI Host [zabbix]: first network error, wait for 15 seconds
8291:20111025:103342.064 Disabling IPMI host [zabbix]
8123:20111025:103456.984 Enabling IPMI host [zabbix]
8102:20111025:103501.839 Enabling IPMI host [zproxy01]
8287:20111025:103539.066 IPMI Host [zproxy01]: first network error, wait for 15 seconds
8289:20111025:103540.094 IPMI Host [zproxy01]: another network error, wait for 15 secon

when it changed to not supported i had check with manual "ipmitool -U zabbix -H 192.168.XX.XX -I lanplus -L ADMINISTRATOR sensor"
i've got the following message:
Error in open session response message : insufficient resources for session

Error: Unable to establish IPMI v2 / RMCP+ session

also the result of "ipmitool -U zabbix -H 192.168.XX.XX -I lanplus -L ADMINISTRATOR session info all" maybe usefull (at the next possible time)
session handle : 1
slot count : 4
active sessions : 60
user id : 2
privilege level : ADMINISTRATOR
session type : IPMIv2/RMCP+
channel number : 0x02
console ip : 192.168.XX.XX
console mac : 00:00:00:00:00:00
console port : 4040

session handle : 2
slot count : 4
active sessions : 60
user id : 2
privilege level : ADMINISTRATOR
session type : IPMIv2/RMCP+
channel number : 0x02
console ip : 192.168.XX.XX
console mac : 00:00:00:00:00:00
console port : 59029

session handle : 0
slot count : 4
active sessions : 60

session handle : 4
slot count : 4
active sessions : 60
user id : 2
privilege level : ADMINISTRATOR
session type : IPMIv2/RMCP+
channel number : 0x02
console ip : 192.168.XX.XX
console mac : 00:00:00:00:00:00
console port : 25072



 Comments   
Comment by richlv [ 2011 Oct 25 ]

could it be that you're just overloading the target ipmi device, thus starving it of the resources ?
if so, i don't think there's anything simple zabbix could help with

Comment by hamid sfandiari [ 2011 Oct 25 ]

do you have think that 3 item for an iLo by the interval of 60 sec caused to Overloading?
it's better that zabbix get all result of sensor items from iLo in one step and gathering them in local buffer then dispatching them as they needed (because of serial aspect of data gathering from IPMI)
each date fetching need authentication and it's time consuming step

i think that the following pesudo script could solve this issue and preventing iLo from overloading

interval 1
#> ipmitool IP .... sensor > buffer
#> grep "temp 02" buffer | zabbix_send $1
#> grep "temp 03" buffer | zabbix_send $1

interval 2
#> ipmitool IP .... sensor > buffer
#> grep "temp 02" buffer | zabbix_send $1
#> grep "temp 03" buffer | zabbix_send $1

Comment by richlv [ 2011 Oct 25 ]

do i understand it correctly that your suggestion is to merge requests to ipmi devices if they happen to be scheduled close enough ?

as for what could overload devices, hard to say. if ipmitool returns you an error, then that's what is happening
having said that, i have seen very serious issues with hp cli tools for querying disk & array status when calling them often enough. that was definitely one thing their testing did not try at all..

Comment by hamid sfandiari [ 2011 Oct 25 ]

yes, you understood correctly

but still i am thinking that closing session after querying (don't waiting for session timeout happen on ilo side) is more effective solution
by this explanation IPMI tool is not useful way for hp in ZABBIX
(sorry for my bad English )

Comment by Cristian Mammoli [ 2012 Mar 02 ]

Similar thing happens on Fujitsu Primergu Servers.

My env:
Zabbix 1.8.10
Server Fujitsu Primergy RX300 S6

Very often I get errors in Zabbix logs and when I query the serevr with ipmitool I get:

[root@srvzabbix ~]# ipmitool -H <IP>-U admin -P admin sensor get "Power Unit"
Locating sensor record...
Get Session Challenge command failed: Out of space
Error: Unable to establish LAN session
Get Device ID command failed
Unable to open SDR for reading
Sensor data record "Power Unit" not found!

Instead of:

[root@srvzabbix ~]# ipmitool -H <IP> -U admin -P admin sensor get "Power Unit"
Locating sensor record...
Sensor ID : Power Unit (0x41)
Entity ID : 19.0
Sensor Type (Discrete): Power Unit
States Asserted : Redundancy State
[Redundancy Lost]

Comment by Alexei Vladishev [ 2012 Aug 27 ]

I am not sure we can do anything with this issue. We are looking forward to optimizing IPMI and SNMP polling by using bulk requests. It may help once implemented.

Comment by richlv [ 2012 Nov 01 ]

should be solved by ZBX-3243





[ZBX-3243] Network error while retrieving IPMI data Created: 2010 Nov 29  Updated: 2017 May 30  Resolved: 2017 Feb 08

Status: Closed
Project: ZABBIX BUGS AND ISSUES
Component/s: Proxy (P), Server (S)
Affects Version/s: 1.8.4rc2
Fix Version/s: None

Type: Incident report Priority: Major
Reporter: Sergey Syreskin Assignee: Unassigned
Resolution: Duplicate Votes: 7
Labels: ipmi
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

Centos 5.5, PostgreSQL 8.4.4, Apache 2.2.3, PHP 5.2.10, Zabbix 1.8.4rc2 (15477);
IBM System x3550 M2 as IPMI client.


Attachments: File ipmi_error_report.tgz    
Issue Links:
Duplicate
duplicates ZBXNEXT-3386 IPMI connection to a single device is... Closed
duplicates ZBX-3188 IPMI host unreachable Closed
is duplicated by ZBX-4270 IPMI insufficient resource Closed

 Description   

Zabbix periodically fails to retrieve some sensor's data via IPMI. The reported error is "End of read_ipmi_sensor():NETWORK_ERROR". This leads to gaps on graphs.
Graphs with gaps, debug log, zabbix server conf, ipmi and host templates are attached in a tgz file.



 Comments   
Comment by Sergey Syreskin [ 2010 Nov 29 ]

The network connection is stable, this is confirmed by ping graphs. Ping frequency is 3 seconds, IPMI requests frequency is 300 seconds. IPMI library version is 2.0.16-7.el5. StartIPMIPollers=2, 3 hosts are monitored via IPMI and there are some other hosts monitored via agents, snmp and simple checks.

Comment by Aleksandrs Saveljevs [ 2010 Nov 29 ]

Seems to be the same problem as in ZBX-3188.

Comment by Sergey Syreskin [ 2010 Nov 29 ]

Aleksandrs Saveljevs, no this is not the same problem as ZBX-3188, I have no "host unreachable" errors in zabbix log. Besides that, all sensors work in my setup, the gaps in data happen periodically, after some time sensors data becomes reachable again, then after some time network error occurs, then it becomes ok again. Just see the graphs attached to my first post.

Comment by richlv [ 2011 Apr 25 ]

could it be that by polling ipmi too often it becomes slow, locks up or just applies some connection throttling ?
how many ipmi items you have ? do they all have the same interval ?

Comment by Sergey Syreskin [ 2011 Apr 26 ]

There are 3 hosts with 125 IPMI items each. Polling interval is set to 300 seconds for each item.
I'm using Zabbix 1.8.5 now and don't experience this problem any more.
I can't remember when the problem disappeared, it could be Zabbix update or changes in the IPMI template,
that I have done some time ago.
The only thing I can say for sure, is that I didn't change any settings on the IPMI devices.

Looking at the template in the attached ipmi_error_report.tgz archive, I can see that my current template is
definitely different from the old one. The old template had only 19 items.

Comment by Chris Witte [ 2012 Mar 27 ]

Same erros with Zabbix 1.8.10 / 2.0.0 RC1 / 2.0.0 RC2

Installed new machine (Debian 6.0.4 -x64 / Virtual machine on VMware ESXi 5) and installed Zabbix 2.0 RC1 - Compiled with openipmi-2.0.19 (tried older version as well).

Zabbix 2.0 is monitoring just ONE host with ONE item (directly, no template) for testing and the errors in the zabbix_server.log appear.
(Interval: 15 sec, no flixible intervals)

I assumed that the BMC was too busy and made checks with openipmish (two checks per second).
Result: All requests were answered correct and in time.

Monitored Host: Dell PowerEdge R610 + R710 with iDRAC6 - Ver: 1.80 (also tested with Ver. 1.71)

    1. configure##
      ./configure --enable-server --enable-agent --with-mysql --enable-ipv6 --with-net-snmp --with-libcurl --with-ssh2 --with-ldap --enable-proxy --openipmi --prefix=/opt/zabbix
      ###
    1. zabbix_server.conf ##
      StartPollers=5
      StartIPMIPollers=5 # incremented step-by-step but no changes
      ###
    1. zabbix_server.log - Zabbix 2.0.0 RC1 ##
      13292:20120327:113333.407 IPMI item [System_Level] on host [F2-CN-01] failed: first network error, wait for 15 seconds
      13271:20120327:113350.583 IPMI item [System_Level] on host [F2-CN-01] failed: another network error, wait for 15 seconds
      13271:20120327:113406.933 resuming IPMI checks on host [F2-CN-01]: connection restored
      13288:20120327:113425.415 IPMI item [System_Level] on host [F2-CN-01] failed: first network error, wait for 15 seconds
      13271:20120327:113440.949 resuming IPMI checks on host [F2-CN-01]: connection restored
      13288:20120327:113504.423 IPMI item [System_Level] on host [F2-CN-01] failed: first network error, wait for 15 seconds
      13271:20120327:113519.964 resuming IPMI checks on host [F2-CN-01]: connection restored
      13289:20120327:113555.014 IPMI item [System_Level] on host [F2-CN-01] failed: first network error, wait for 15 seconds
      13271:20120327:113610.980 resuming IPMI checks on host [F2-CN-01]: connection restored
      13291:20120327:113625.014 IPMI item [System_Level] on host [F2-CN-01] failed: first network error, wait for 15 seconds
      13271:20120327:113640.994 resuming IPMI checks on host [F2-CN-01]: connection restored
      13291:20120327:113649.023 IPMI item [System_Level] on host [F2-CN-01] failed: first network error, wait for 15 seconds
      13271:20120327:113705.008 resuming IPMI checks on host [F2-CN-01]: connection restored
      13289:20120327:113718.027 IPMI item [System_Level] on host [F2-CN-01] failed: first network error, wait for 15 seconds
      13271:20120327:113733.026 resuming IPMI checks on host [F2-CN-01]: connection restored
      13288:20120327:113955.094 IPMI item [System_Level] on host [F2-CN-01] failed: first network error, wait for 15 seconds
      13271:20120327:114017.064 IPMI item [System_Level] on host [F2-CN-01] failed: another network error, wait for 15 seconds
      13271:20120327:114033.070 IPMI item [System_Level] on host [F2-CN-01] failed: another network error, wait for 15 seconds
      13271:20120327:114050.561 resuming IPMI checks on host [F2-CN-01]: connection restored
      13289:20120327:114110.112 IPMI item [System_Level] on host [F2-CN-01] failed: first network error, wait for 15 seconds
      13271:20120327:114125.586 resuming IPMI checks on host [F2-CN-01]: connection restored
      13288:20120327:114134.110 IPMI item [System_Level] on host [F2-CN-01] failed: first network error, wait for 15 seconds
      13271:20120327:114149.602 resuming IPMI checks on host [F2-CN-01]: connection restored
      13289:20120327:114219.124 IPMI item [System_Level] on host [F2-CN-01] failed: first network error, wait for 15 seconds
      13271:20120327:114234.618 resuming IPMI checks on host [F2-CN-01]: connection restored
      13291:20120327:114640.743 IPMI item [System_Level] on host [F2-CN-01] failed: first network error, wait for 15 seconds
      13271:20120327:114702.663 IPMI item [System_Level] on host [F2-CN-01] failed: another network error, wait for 15 seconds
      13271:20120327:114718.673 IPMI item [System_Level] on host [F2-CN-01] failed: another network error, wait for 15 seconds
      13271:20120327:114736.178 resuming IPMI checks on host [F2-CN-01]: connection restored
      13289:20120327:114755.756 IPMI item [System_Level] on host [F2-CN-01] failed: first network error, wait for 15 seconds
      13271:20120327:114810.189 resuming IPMI checks on host [F2-CN-01]: connection restored
      13291:20120327:114819.756 IPMI item [System_Level] on host [F2-CN-01] failed: first network error, wait for 15 seconds
      13271:20120327:114834.205 resuming IPMI checks on host [F2-CN-01]: connection restored
      13290:20120327:115155.014 IPMI item [System_Level] on host [F2-CN-01] failed: first network error, wait for 15 seconds
      13271:20120327:115217.250 IPMI item [System_Level] on host [F2-CN-01] failed: another network error, wait for 15 seconds
      13271:20120327:115233.257 IPMI item [System_Level] on host [F2-CN-01] failed: another network error, wait for 15 seconds
      13271:20120327:115250.893 resuming IPMI checks on host [F2-CN-01]: connection restored
      13292:20120327:115310.016 IPMI item [System_Level] on host [F2-CN-01] failed: first network error, wait for 15 seconds
      13271:20120327:115325.904 resuming IPMI checks on host [F2-CN-01]: connection restored
      13292:20120327:115333.023 IPMI item [System_Level] on host [F2-CN-01] failed: first network error, wait for 15 seconds
      13271:20120327:115348.920 resuming IPMI checks on host [F2-CN-01]: connection restored
      13290:20120327:115418.036 IPMI item [System_Level] on host [F2-CN-01] failed: first network error, wait for 15 seconds
      13271:20120327:115433.941 resuming IPMI checks on host [F2-CN-01]: connection restored
      13291:20120327:115510.801 IPMI item [System_Level] on host [F2-CN-01] failed: first network error, wait for 15 seconds
      13271:20120327:115525.959 resuming IPMI checks on host [F2-CN-01]: connection restored
      13291:20120327:115534.808 IPMI item [System_Level] on host [F2-CN-01] failed: first network error, wait for 15 seconds
      13271:20120327:115549.975 resuming IPMI checks on host [F2-CN-01]: connection restored
      13292:20120327:115940.013 IPMI item [System_Level] on host [F2-CN-01] failed: first network error, wait for 15 seconds
      13271:20120327:120002.020 IPMI item [System_Level] on host [F2-CN-01] failed: another network error, wait for 15 seconds
      13271:20120327:120018.026 IPMI item [System_Level] on host [F2-CN-01] failed: another network error, wait for 15 seconds
      13271:20120327:120035.679 resuming IPMI checks on host [F2-CN-01]: connection restored
      13290:20120327:120055.023 IPMI item [System_Level] on host [F2-CN-01] failed: first network error, wait for 15 seconds
      13271:20120327:120110.695 resuming IPMI checks on host [F2-CN-01]: connection restored
      13290:20120327:120119.035 IPMI item [System_Level] on host [F2-CN-01] failed: first network error, wait for 15 seconds
      13271:20120327:120134.711 resuming IPMI checks on host [F2-CN-01]: connection restored
      13292:20120327:120149.035 IPMI item [System_Level] on host [F2-CN-01] failed: first network error, wait for 15 seconds
      13271:20120327:120204.725 resuming IPMI checks on host [F2-CN-01]: connection restored
      13291:20120327:120255.625 IPMI item [System_Level] on host [F2-CN-01] failed: first network error, wait for 15 seconds
      13271:20120327:120317.749 IPMI item [System_Level] on host [F2-CN-01] failed: another network error, wait for 15 seconds
      13271:20120327:120333.765 IPMI item [System_Level] on host [F2-CN-01] failed: another network error, wait for 15 seconds
      13271:20120327:120351.272 resuming IPMI checks on host [F2-CN-01]: connection restored
      13291:20120327:120404.641 IPMI item [System_Level] on host [F2-CN-01] failed: first network error, wait for 15 seconds
      13271:20120327:120419.283 resuming IPMI checks on host [F2-CN-01]: connection restored
      13288:20120327:121655.018 IPMI item [System_Level] on host [F2-CN-01] failed: first network error, wait for 15 seconds
      13271:20120327:121717.398 IPMI item [System_Level] on host [F2-CN-01] failed: another network error, wait for 15 seconds
      13271:20120327:121732.402 IPMI item [System_Level] on host [F2-CN-01] failed: another network error, wait for 15 seconds
      13271:20120327:121749.902 resuming IPMI checks on host [F2-CN-01]: connection restored
      13288:20120327:121803.030 IPMI item [System_Level] on host [F2-CN-01] failed: first network error, wait for 15 seconds
      13271:20120327:121818.918 resuming IPMI checks on host [F2-CN-01]: connection restored
      13291:20120327:121855.640 IPMI item [System_Level] on host [F2-CN-01] failed: first network error, wait for 15 seconds
      13271:20120327:121910.935 resuming IPMI checks on host [F2-CN-01]: connection restored
      13291:20120327:121919.648 IPMI item [System_Level] on host [F2-CN-01] failed: first network error, wait for 15 seconds
      13271:20120327:121934.950 resuming IPMI checks on host [F2-CN-01]: connection restored
      13289:20120327:122249.007 IPMI item [System_Level] on host [F2-CN-01] failed: first network error, wait for 15 seconds
      13271:20120327:122311.990 IPMI item [System_Level] on host [F2-CN-01] failed: another network error, wait for 15 seconds
      13271:20120327:122327.995 IPMI item [System_Level] on host [F2-CN-01] failed: another network error, wait for 15 seconds
      13271:20120327:122345.767 resuming IPMI checks on host [F2-CN-01]: connection restored
      13291:20120327:122355.249 IPMI item [System_Level] on host [F2-CN-01] failed: first network error, wait for 15 seconds
      13271:20120327:122410.781 resuming IPMI checks on host [F2-CN-01]: connection restored
      ###

-EDIT: 2012 Mar 28-

1. Converted VM from VMware to VirtualBox (Windows) on another host (win7) in another network segment (to exclude hypervisor, Host-OS, network connectivity from error source)
2. Compiled Zabbix 2.0.0 RC2 and updated the system
3. Added host and templates

Result:

    1. zabbix_server.log - Zabbix 2.0.0 RC2 ##
      1539:20120328:123650.011 Starting Zabbix Server. Zabbix 2.0.0rc2 (revision 26343).
      1539:20120328:123650.011 ****** Enabled features ******
      1539:20120328:123650.011 SNMP monitoring: YES
      1539:20120328:123650.012 IPMI monitoring: YES
      1539:20120328:123650.012 WEB monitoring: YES
      1539:20120328:123650.012 Jabber notifications: NO
      1539:20120328:123650.012 Ez Texting notifications: YES
      1539:20120328:123650.012 ODBC: NO
      1539:20120328:123650.012 SSH2 support: YES
      1539:20120328:123650.012 IPv6 support: YES
      1539:20120328:123650.012 ******************************
      1541:20120328:123650.068 server #2 started db watchdog #1
      1540:20120328:123650.070 server #1 started configuration syncer #1
      1548:20120328:123650.126 server #9 started trapper #1
      1549:20120328:123650.128 server #10 started trapper #2
      1550:20120328:123650.130 server #11 started trapper #3
      1551:20120328:123650.158 server #12 started trapper #4
      1544:20120328:123650.161 server #5 started poller #3
      1542:20120328:123650.163 server #3 started poller #1
      1545:20120328:123650.164 server #6 started poller #4
      1543:20120328:123650.165 server #4 started poller #2
      1546:20120328:123650.167 server #7 started poller #5
      1547:20120328:123650.170 server #8 started unreachable poller #1
      1552:20120328:123650.173 server #13 started trapper #5
      1553:20120328:123650.179 server #14 started icmp pinger #1
      1554:20120328:123650.185 server #15 started alerter #1
      1555:20120328:123650.192 server #16 started housekeeper #1
      1555:20120328:123650.192 executing housekeeper
      1566:20120328:123650.204 server #17 started timer #1
      1567:20120328:123650.206 server #18 started http poller #1
      1569:20120328:123650.215 server #20 started history syncer #1
      1570:20120328:123650.217 server #21 started history syncer #2
      1571:20120328:123650.220 server #22 started history syncer #3
      1572:20120328:123650.223 server #23 started history syncer #4
      1579:20120328:123650.244 server #24 started escalator #1
      1580:20120328:123650.247 server #25 started ipmi poller #1
      1581:20120328:123650.250 server #26 started ipmi poller #2
      1582:20120328:123650.253 server #27 started ipmi poller #3
      1568:20120328:123650.262 server #19 started discoverer #1
      1586:20120328:123650.273 server #29 started ipmi poller #5
      1587:20120328:123650.275 server #30 started proxy poller #1
      1539:20120328:123650.280 server #0 started [main process]
      1585:20120328:123650.284 server #28 started ipmi poller #4
      1592:20120328:123650.289 server #31 started self-monitoring #1
      1555:20120328:123651.371 housekeeper deleted: 10190 records from history and trends, 500 records of deleted items, 0 events, 0 alerts, 0 sessions
      1547:20120328:123655.299 temporarily disabling IPMI checks on host [F2-VH-01]: host unavailable
      1580:20120328:123700.385 IPMI item [FAN_MOD_1B_RPM] on host [F2-VH-02] failed: first network error, wait for 15 seconds
      1547:20120328:123712.952 resuming IPMI checks on host [F2-CN-02]: connection restored
      1547:20120328:123712.967 temporarily disabling IPMI checks on host [F2-CN-01]: host unavailable
      1581:20120328:123713.314 IPMI item [FAN_4_RPM] on host [F2-CN-02] failed: first network error, wait for 15 seconds
      1547:20120328:123715.974 IPMI item [FAN_MOD_1B_RPM] on host [F2-VH-02] failed: another network error, wait for 15 seconds
      1547:20120328:123728.994 resuming IPMI checks on host [F2-CN-02]: connection restored
      1547:20120328:123731.006 IPMI item [FAN_MOD_4A_RPM] on host [F2-VH-02] failed: another network error, wait for 15 seconds
      1586:20120328:123739.330 IPMI item [Ambient_Temp] on host [F2-CN-02] failed: first network error, wait for 15 seconds
      1547:20120328:123746.025 temporarily disabling IPMI checks on host [F2-VH-02]: host unavailable
      1547:20120328:123754.040 resuming IPMI checks on host [F2-CN-02]: connection restored
      1547:20120328:123758.285 enabling IPMI checks on host [F2-VH-01]: host became available
      1585:20120328:123809.353 IPMI item [Ambient_Temp] on host [F2-CN-02] failed: first network error, wait for 15 seconds
      1547:20120328:123815.477 enabling IPMI checks on host [F2-CN-01]: host became available
      1586:20120328:123816.364 IPMI item [FAN_4_RPM] on host [F2-CN-01] failed: first network error, wait for 15 seconds
      1581:20120328:123816.364 IPMI item [Ambient_Temp] on host [F2-VH-01] failed: first network error, wait for 15 seconds
      1547:20120328:123824.499 resuming IPMI checks on host [F2-CN-02]: connection restored
      1547:20120328:123831.521 resuming IPMI checks on host [F2-VH-01]: connection restored
      1547:20120328:123831.529 resuming IPMI checks on host [F2-CN-01]: connection restored
      1585:20120328:123839.381 IPMI item [Ambient_Temp] on host [F2-CN-02] failed: first network error, wait for 15 seconds
      1581:20120328:123842.384 IPMI item [Ambient_Temp] on host [F2-CN-01] failed: first network error, wait for 15 seconds
      1586:20120328:123846.388 IPMI item [Ambient_Temp] on host [F2-VH-01] failed: first network error, wait for 15 seconds
      1547:20120328:123854.560 resuming IPMI checks on host [F2-CN-02]: connection restored
      1547:20120328:123857.572 resuming IPMI checks on host [F2-CN-01]: connection restored
      1547:20120328:123901.582 resuming IPMI checks on host [F2-VH-01]: connection restored
      1586:20120328:123911.420 IPMI item [FAN_2_RPM] on host [F2-CN-02] failed: first network error, wait for 15 seconds
      1582:20120328:123912.411 IPMI item [Ambient_Temp] on host [F2-CN-01] failed: first network error, wait for 15 seconds
      1582:20120328:123917.457 IPMI item [FAN_MOD_1B_RPM] on host [F2-VH-01] failed: first network error, wait for 15 seconds
      1547:20120328:123926.620 resuming IPMI checks on host [F2-CN-02]: connection restored
      1547:20120328:123927.629 resuming IPMI checks on host [F2-CN-01]: connection restored
      1547:20120328:123932.640 resuming IPMI checks on host [F2-VH-01]: connection restored
      1585:20120328:123942.166 IPMI item [Ambient_Temp] on host [F2-CN-01] failed: first network error, wait for 15 seconds
      1585:20120328:123946.191 IPMI item [Ambient_Temp] on host [F2-VH-01] failed: first network error, wait for 15 seconds
      1547:20120328:123957.679 resuming IPMI checks on host [F2-CN-01]: connection restored
      1547:20120328:124001.691 resuming IPMI checks on host [F2-VH-01]: connection restored
      1586:20120328:124013.463 IPMI item [FAN_4_RPM] on host [F2-CN-02] failed: first network error, wait for 15 seconds
      1586:20120328:124013.477 IPMI item [FAN_1_RPM] on host [F2-CN-01] failed: first network error, wait for 15 seconds
      1586:20120328:124016.486 IPMI item [Ambient_Temp] on host [F2-VH-01] failed: first network error, wait for 15 seconds
      1547:20120328:124028.718 resuming IPMI checks on host [F2-CN-02]: connection restored
      1547:20120328:124028.726 resuming IPMI checks on host [F2-CN-01]: connection restored
      1547:20120328:124031.735 resuming IPMI checks on host [F2-VH-01]: connection restored
      1586:20120328:124039.505 IPMI item [Ambient_Temp] on host [F2-CN-02] failed: first network error, wait for 15 seconds
      1582:20120328:124042.515 IPMI item [Ambient_Temp] on host [F2-CN-01] failed: first network error, wait for 15 seconds
      1586:20120328:124046.517 IPMI item [Ambient_Temp] on host [F2-VH-01] failed: first network error, wait for 15 seconds
      1547:20120328:124049.716 enabling IPMI checks on host [F2-VH-02]: host became available
      1547:20120328:124054.727 resuming IPMI checks on host [F2-CN-02]: connection restored
      1547:20120328:124057.741 resuming IPMI checks on host [F2-CN-01]: connection restored
      1582:20120328:124059.529 IPMI item [Ambient_Temp] on host [F2-VH-02] failed: first network error, wait for 15 seconds
      1547:20120328:124101.759 resuming IPMI checks on host [F2-VH-01]: connection restored
      1586:20120328:124109.537 IPMI item [Ambient_Temp] on host [F2-CN-02] failed: first network error, wait for 15 seconds
      1580:20120328:124112.541 IPMI item [Ambient_Temp] on host [F2-CN-01] failed: first network error, wait for 15 seconds
      1547:20120328:124114.776 resuming IPMI checks on host [F2-VH-02]: connection restored
      1582:20120328:124116.547 IPMI item [Ambient_Temp] on host [F2-VH-01] failed: first network error, wait for 15 seconds
      1547:20120328:124124.796 resuming IPMI checks on host [F2-CN-02]: connection restored
      1547:20120328:124127.807 resuming IPMI checks on host [F2-CN-01]: connection restored
      1586:20120328:124129.554 IPMI item [Ambient_Temp] on host [F2-VH-02] failed: first network error, wait for 15 seconds
      1547:20120328:124131.818 resuming IPMI checks on host [F2-VH-01]: connection restored
      1582:20120328:124141.569 IPMI item [FAN_2_RPM] on host [F2-CN-02] failed: first network error, wait for 15 seconds
      1586:20120328:124142.568 IPMI item [Ambient_Temp] on host [F2-CN-01] failed: first network error, wait for 15 seconds
      1547:20120328:124144.832 resuming IPMI checks on host [F2-VH-02]: connection restored
      1581:20120328:124146.017 IPMI item [Ambient_Temp] on host [F2-CN-02] failed: another network error, wait for 15 seconds
      1581:20120328:124147.024 IPMI item [Ambient_Temp] on host [F2-VH-01] failed: first network error, wait for 15 seconds
      1547:20120328:124157.854 resuming IPMI checks on host [F2-CN-01]: connection restored
      1581:20120328:124159.046 IPMI item [Ambient_Temp] on host [F2-VH-02] failed: first network error, wait for 15 seconds
      1547:20120328:124201.865 resuming IPMI checks on host [F2-CN-02]: connection restored
      1547:20120328:124202.872 resuming IPMI checks on host [F2-VH-01]: connection restored
      1581:20120328:124209.063 IPMI item [Ambient_Temp] on host [F2-CN-02] failed: first network error, wait for 15 seconds
      1581:20120328:124212.072 IPMI item [Ambient_Temp] on host [F2-CN-01] failed: first network error, wait for 15 seconds
      1547:20120328:124214.886 resuming IPMI checks on host [F2-VH-02]: connection restored
      1581:20120328:124216.085 IPMI item [Ambient_Temp] on host [F2-VH-01] failed: first network error, wait for 15 seconds
      ...
      ####
Comment by Chris Witte [ 2012 Mar 28 ]

It seems that the BMC gets too many requests/connections.
From time to time I get following message when running ipmitool:

# ipmitool sdr -H <HOSTNAME> -U <USER> -P <PASSWORD> -L USER

Get Session Challenge command failed: Node busy
Error: Unable to establish LAN session
Get Device ID command failed
Unable to open SDR for reading

Does Zabbix use a sdr cache ? This could increase the performance.

ipmitool offers this parameter:

-S <sdr_cache_file>
              Use local file for remote SDR cache.  Using a local  SDR  cache
              can  drastically increase performance for commands that require
              knowledge of the entire SDR to perform their  function.   Local
              SDR cache from a remote system can be created with the sdr dump
              command.

BMC busy topic: http://old.nabble.com/possible-causes-for-%22ipmi_ctx_open_outofband%3A-BMC-busy%22-td31448014.html

Comment by Chris Witte [ 2012 Mar 30 ]

Posted this problem on Dell Community:

http://en.community.dell.com/support-forums/servers/f/177/p/19442918/20078853.aspx#20078853

Comment by Sergey Syreskin [ 2012 May 22 ]

My colleague has done some testing on this issue, and he came to the conclusion that IPMI CPU is unable to handle all those requests. As he says, for each request to IPMI host Zabbix opens one separate connection and IBM System x IMM module is unable to handle all the requests. So he had to write a wrapper script that requests all IPMI items from the host at a time, stores them in a cache file, and gives items to Zabbix when it requests.

Comment by Chris Witte [ 2012 May 22 ]

Thanks for your reply. Could you post the wrapper script here ?

What about caching the sdr query like ipmitool does whe using the parameter -s ?

-S <sdr_cache_file> 
              Use local file for remote SDR cache. Using a local SDR cache 
              can drastically increase performance for commands that require 
              knowledge of the entire SDR to perform their function. Local 
              SDR cache from a remote system can be created with the sdr dump 
              command. 

I know that freeipmi automaticaly creates a cachefile of the sdr. But Zabbix uses openipmi.
For sure Zabbix's IPMI-Engine would have a better performace when using the caching option by default.

Chris

Comment by Sergey Syreskin [ 2012 May 22 ]

The script is rather simple, it just stores values in a local file with a timestamp. Then, when Zabbix requests a value, script examines the timestamp, and either renews its cache first, or just gives out data from cache, if it's recent enough.

Comment by Sergey Syreskin [ 2012 Aug 14 ]

This issue is covered by ZBXNEXT-1210, which is related to ZBXNEXT-98.

Comment by Aaron Smart [ 2012 Aug 24 ]

I'm experiencing the same network errors in the server log as Chris is above (running 2.0.2), trying to connect to a Dell PowerEdge 1950 (BMC) and PowerEdge R210 II (iDRAC 6 Express). Is there some way to make the IPMI poller more accommodating for slow devices?

Comment by dimir [ 2012 Aug 24 ]

There is a discussion going recently about fixing this one. We will report as soon as there is more information.

Comment by Falk G. [ 2012 Sep 09 ]

i have the same issue ... and its not related to DELL. I am using Supermicro IPMI to monitor RAM and Environment Temperature and i got the same issues:

24006:20120909:171011.011 IPMI item [P2-DIMM2B_Temp] on host [Supermicro SC836] failed: first network error, wait for 15 seconds
23988:20120909:171033.326 IPMI item [P2-DIMM2B_Temp] on host [Supermicro SC836] failed: another network error, wait for 15 seconds
23988:20120909:171048.330 IPMI item [P2-DIMM3B_Temp] on host [Supermicro SC836] failed: another network error, wait for 15 seconds
23988:20120909:171104.539 resuming IPMI checks on host [Supermicro SC836]: connection restored
24006:20120909:171111.023 IPMI item [P1-DIMM1A_Temp] on host [Supermicro SC836] failed: first network error, wait for 15 seconds
23988:20120909:171126.547 resuming IPMI checks on host [Supermicro SC836]: connection restored
24005:20120909:171135.955 IPMI item [Fan5] on host [Supermicro SC836] failed: first network error, wait for 15 seconds
23988:20120909:171150.556 resuming IPMI checks on host [Supermicro SC836]: connection restored
24005:20120909:171559.993 IPMI item [Fan5] on host [Supermicro SC836] failed: first network error, wait for 15 seconds
24007:20120909:171603.995 IPMI item [P2-DIMM2A_Temp] on host [Supermicro SC836] failed: another network error, wait for 15 seconds
23988:20120909:171625.608 IPMI item [Fan6] on host [Supermicro SC836] failed: another network error, wait for 15 seconds
23988:20120909:171641.611 IPMI item [Fan3] on host [Supermicro SC836] failed: another network error, wait for 15 seconds
23988:20120909:171657.737 resuming IPMI checks on host [Supermicro SC836]: connection restored
24007:20120909:171718.004 IPMI item [P1-DIMM3A_Temp] on host [Supermicro SC836] failed: first network error, wait for 15 seconds
23988:20120909:171733.748 resuming IPMI checks on host [Supermicro SC836]: connection restored
24006:20120909:172329.683 IPMI item [Fan2] on host [Supermicro SC836] failed: first network error, wait for 15 seconds
23988:20120909:172351.819 IPMI item [Fan2] on host [Supermicro SC836] failed: another network error, wait for 15 seconds
23988:20120909:172407.825 IPMI item [P2-DIMM1A_Temp] on host [Supermicro SC836] failed: another network error, wait for 15 seconds
23988:20120909:172424.027 resuming IPMI checks on host [Supermicro SC836]: connection restored
24006:20120909:172429.695 IPMI item [P2-DIMM2A_Temp] on host [Supermicro SC836] failed: first network error, wait for 15 seconds
23988:20120909:172444.037 resuming IPMI checks on host [Supermicro SC836]: connection restored
24005:20120909:172453.725 IPMI item [P1-DIMM2B_Temp] on host [Supermicro SC836] failed: first network error, wait for 15 seconds
23988:20120909:172508.047 resuming IPMI checks on host [Supermicro SC836]: connection restored
24005:20120909:172511.730 IPMI item [P1-DIMM1A_Temp] on host [Supermicro SC836] failed: first network error, wait for 15 seconds
23988:20120909:172526.056 resuming IPMI checks on host [Supermicro SC836]: connection restored
24006:20120909:172559.839 IPMI item [Fan2] on host [Supermicro SC836] failed: first network error, wait for 15 seconds
23988:20120909:172614.069 resuming IPMI checks on host [Supermicro SC836]: connection restored
24006:20120909:172615.843 IPMI item [P1-DIMM2A_Temp] on host [Supermicro SC836] failed: first network error, wait for 15 seconds
23988:20120909:172630.078 resuming IPMI checks on host [Supermicro SC836]: connection restored
24007:20120909:173101.923 IPMI item [Fan3] on host [Supermicro SC836] failed: first network error, wait for 15 seconds
23988:20120909:173123.135 IPMI item [Fan3] on host [Supermicro SC836] failed: another network error, wait for 15 seconds
23988:20120909:173139.139 IPMI item [P2-DIMM1A_Temp] on host [Supermicro SC836] failed: another network error, wait for 15 seconds
23988:20120909:173155.263 resuming IPMI checks on host [Supermicro SC836]: connection restored
24007:20120909:173156.932 IPMI item [P2-DIMM2A_Temp] on host [Supermicro SC836] failed: first network error, wait for 15 seconds
23988:20120909:173211.665 resuming IPMI checks on host [Supermicro SC836]: connection restored
24005:20120909:173347.011 IPMI item [P1-DIMM1A_Temp] on host [Supermicro SC836] failed: first network error, wait for 15 seconds
23988:20120909:173409.694 IPMI item [P2-DIMM1A_Temp] on host [Supermicro SC836] failed: another network error, wait for 15 seconds
23988:20120909:173425.699 IPMI item [System_Temp] on host [Supermicro SC836] failed: another network error, wait for 15 seconds

23678:20120904:202202.840 Starting Zabbix Server. Zabbix 2.0.2 (revision 29214).
23678:20120904:202202.840 ****** Enabled features ******
23678:20120904:202202.840 SNMP monitoring: YES
23678:20120904:202202.840 IPMI monitoring: YES
23678:20120904:202202.840 WEB monitoring: NO
23678:20120904:202202.840 Jabber notifications: NO
23678:20120904:202202.840 Ez Texting notifications: NO
23678:20120904:202202.840 ODBC: NO
23678:20120904:202202.840 SSH2 support: NO
23678:20120904:202202.840 IPv6 support: NO
23678:20120904:202202.840 ******************************
23680:20120904:202202.900 server #1 started configuration syncer #1
23681:20120904:202202.900 server #2 started db watchdog #1
23682:20120904:202202.901 server #3 started poller #1
23683:20120904:202202.902 server #4 started poller #2
23684:20120904:202202.904 server #5 started poller #3
23685:20120904:202202.905 server #6 started poller #4
23686:20120904:202202.906 server #7 started poller #5
23678:20120904:202202.906 server #0 started [main process]
23704:20120904:202202.906 server #25 started ipmi poller #1
23687:20120904:202202.907 server #8 started unreachable poller #1
23705:20120904:202202.907 server #26 started ipmi poller #2
23706:20120904:202202.907 server #27 started ipmi poller #3
23707:20120904:202202.907 server #28 started proxy poller #1
23708:20120904:202202.908 server #29 started self-monitoring #1
23692:20120904:202202.910 server #13 started trapper #5
23693:20120904:202202.910 server #14 started icmp pinger #1
23698:20120904:202202.911 server #19 started discoverer #1
23697:20120904:202202.911 server #18 started http poller #1
23696:20120904:202202.912 server #17 started timer #1
23695:20120904:202202.912 server #16 started housekeeper #1
23695:20120904:202202.912 executing housekeeper
23694:20120904:202202.912 server #15 started alerter #1
23699:20120904:202202.913 server #20 started history syncer #1
23688:20120904:202202.913 server #9 started trapper #1
23689:20120904:202202.913 server #10 started trapper #2
23690:20120904:202202.913 server #11 started trapper #3
23691:20120904:202202.913 server #12 started trapper #4
23702:20120904:202202.914 server #23 started history syncer #4
23701:20120904:202202.914 server #22 started history syncer #3
23700:20120904:202202.914 server #21 started history syncer #2
23703:20120904:202202.915 server #24 started escalator #1

Comment by Milosz Modrzewski [ 2012 Sep 27 ]

Same problem for me:

Zabbix server v2.0.2 --> Zabbix proxy v2.0.2 (revision 29214) --> Dell Remote Access Controller 5 A01 Firmware Version 1.60 (11.03.03) IP: 172.30.5.96
Zabbix server v2.0.2 --> Zabbix proxy v2.0.2 (revision 29214) --> iLO4 Firmware Version 1.05 ILOCZ22240991 IP: 172.30.5.98

1689:20120927:144113.549 resuming IPMI checks on host [172.30.5.98]: connection restored
1679:20120927:144123.137 Received configuration data from server. Datalen 7766
1709:20120927:144206.227 IPMI item [ipmi.ambient_temp] on host [172.30.5.96] failed: first network error, wait for 15 seconds
1679:20120927:144223.268 Received configuration data from server. Datalen 7766
1689:20120927:144228.939 resuming IPMI checks on host [172.30.5.96]: connection restored
1679:20120927:144323.726 Received configuration data from server. Datalen 7766
1711:20120927:144416.828 IPMI item [ipmi.ambient_temp] on host [172.30.5.98] failed: first network error, wait for 15 seconds
1679:20120927:144423.899 Received configuration data from server. Datalen 7766
1689:20120927:144432.106 IPMI item [ipmi.ambient_temp] on host [172.30.5.98] failed: another network error, wait for 15 seconds
1689:20120927:144457.405 resuming IPMI checks on host [172.30.5.98]: connection restored
1679:20120927:144524.008 Received configuration data from server. Datalen 7766

Comment by Anton Samets [ 2012 Nov 22 ]

I know the solution for this issue:
as for me, if you are using LO-100 you must set password size to 16 bytes (not 20). After that monitoring of IPMI will start to work.
So, zabbix don't use ipmi 2.0 and I can't find where I can set it.

Print out of commands if you have password size set to 20 bytes:

ipmitool  -H 10.145.1.129 -U admin -P admin chassis status
Invalid user name
Error: Unable to establish LAN session
Error sending Chassis Status command
ipmitool -I lanplus -H 10.145.1.129 -U admin -P admin chassis status
System Power         : on
Power Overload       : false
Power Interlock      : inactive
Main Power Fault     : false
Power Control Fault  : false
Power Restore Policy : previous
Last Power Event     : 
Chassis Intrusion    : inactive
Front-Panel Lockout  : inactive
Drive Fault          : false
Cooling/Fan Fault    : false
Sleep Button Disable : allowed
Diag Button Disable  : allowed
Reset Button Disable : allowed
Power Button Disable : allowed
Sleep Button Disabled: false
Diag Button Disabled : false
Reset Button Disabled: false
Power Button Disabled: false

So, where we can set parameters for ipmi-tools?

Comment by Anton Samets [ 2012 Nov 22 ]

Hm, I found that if you set Authentication algorithm from "none" to "RMCP+" all is works fine.

Comment by Andrej Kacian [ 2013 Mar 14 ]

I too had same problem (monitoring 5 hosts with around 10 items each), and was getting unsupported items intermittently every minute or so. Based on a suggestion from forums[1], I changed number of IPMI pollers to just one. Since then, there was no problem with getting IPMI values at all. This was on zabbix 2.0.3 at that time, and still works flawlessly on 2.05 with just one IPMI poller.

1. https://www.zabbix.com/forum/showpost.php?s=783bdc9aff7d3ea26999f74f4d223e59&p=118389&postcount=4

Comment by Alexey Pustovalov [ 2014 Feb 24 ]

if IPMI sensor is located at the end of table of sensors, getting value can take about 40-50 seconds and sometimes can be failed with network error:

 10673:20140224:182441.012 In get_value() key:'ipmi.cpu[FAN 1]'
 10673:20140224:182441.012 In get_value_ipmi() key:'Zabbix server:ipmi.cpu[FAN 1]'
 10673:20140224:182441.012 In init_ipmi_host() host:'[10.100.52.28]:623'
 10673:20140224:182441.012 In get_ipmi_host() host:'[10.100.52.28]:623'
 10673:20140224:182441.012 End of get_ipmi_host():0x2f9bbf0
 10673:20140224:182441.013 End of init_ipmi_host():0x2f9bbf0
 10673:20140224:182441.013 In get_ipmi_sensor_by_id() sensor:'FAN 1@[10.100.52.28]:623'
 10673:20140224:182441.013 End of get_ipmi_sensor_by_id():0x307fcf8
 10673:20140224:182441.013 In read_ipmi_sensor() sensor:'FAN 1@[10.100.52.28]:623'
 10673:20140224:182448.020 In got_thresh_reading()
 10673:20140224:182448.020 got_thresh_reading() fail: [16777411] Unknown error 16777411
 10673:20140224:182448.020 End of got_thresh_reading():NETWORK_ERROR
 10673:20140224:182448.020 End of read_ipmi_sensor():NETWORK_ERROR
 10673:20140224:182448.020 Item [Zabbix server:ipmi.cpu[FAN 1]] error: error 0x10000c3 while reading threshold sensor
 10673:20140224:182448.020 End of get_value():NETWORK_ERROR
Comment by Michael Sphar [ 2014 Apr 14 ]

I have had the same experience, that reducing the number of IPMI pollers to just one has stopped the frequent Network Error messages and gaps.

I had both my production server and a small test VM server, the production server was only polling a few IPMI items and a lot of other non-IPMI monitoring, and the test server was only polling a few IPMI items and doing no other monitoring. Both servers were showing frequent Network Errors and gaps from the items then going unsupported. I reduced the number of IPMI pollers to just one last week and have yet to see a Network Error warning since. Neither server showed all the IPMI pollers as busy.

This is with Zabbix 2.2.2.

Thinking it might have something to do with two different ipmi pollers polling the same device at the same time, I did a simple test where from two different hosts I issued an ipmitool sensor command to the same IPMI device. What I observed is that the resulting output from the IPMI is only sent to one device at a time. The effect I observe is that one ipmitool output starts scrolling while the other is paused for a few seconds, then the other starts scrolling and the first one pauses, and this goes back and forth a few times until both are complete.

Comment by Norbert Wögerbauer [ 2014 Oct 09 ]

Same here with 2.2.6
I observed that I had an item with an invalid sensor id. Seems that if there is a problem with any one item, further processing just breaks.
I disabled all items that did not give a value and the problem disappears.
Note that this happens even if the sensor is listed and basically available, but simply doesn't provide a value (e.g. I have sensor FAN4 but no fan connected to it)!

Comment by Jeroen van den Berg [ 2015 Jun 27 ]

This issue still exists in 2.4.5, and I can confirm that if you disable unavailable sensors it works without problems.
Looks like the handling of not available sensors is incorrect.

Comment by Ilya Kruchinin [ 2015 Aug 12 ]

Disabling unavailable sensors did not help (zabbix_server v2.4.5) in my case - those sensors that had been successfully receiving data still had issues.
However, when I set IPMIPollers to 1, the issue disappeared.

Comment by pfoo [ 2015 Dec 13 ]

Similar behaviour on my supermicro board monitored using latest zabbix server and agent from zabbix debian repository (2.4.7-1+jessie).
I was actually able to fix the issue by lowering the update interval of one IPMI item from 300s to 60s (all others ipmi items were kept to their 300s interval). If I set this ipmi item to 90s, the issue appears again.
Could be some ipmi session handling issue.

Comment by Sascha Plumhoff [ 2016 Jun 14 ]

Same here with DELL PowerEdge R510.

It seems to be a know issue with OpenIPMI, see https://www.zabbix.com/documentation/3.0/manual/config/items/itemtypes/ipmi :

"IPMI session inactivity timeout for LAN is 60 +/-3 seconds. [...] then the next IPMI check after the timeout expires will time out due to individual message timeouts, retries or receive error."

Reducing the check interval to 45 seconds fixed the problem for me.
The issue appears naturally more frequently in testing environments e.g. when checking only 1 item on a server.

Comment by Alexander Vladishev [ 2017 Feb 08 ]

Already fixed under ZBXNEXT-3386.





[ZBX-3188] IPMI host unreachable Created: 2010 Nov 05  Updated: 2017 May 30  Resolved: 2015 Feb 08

Status: Closed
Project: ZABBIX BUGS AND ISSUES
Component/s: Server (S)
Affects Version/s: 1.8.2
Fix Version/s: None

Type: Incident report Priority: Major
Reporter: Alfred Sawaya Assignee: Unassigned
Resolution: Cannot Reproduce Votes: 0
Labels: ipmi
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

Debian GNU/Linux Lenny 5.0.4


Issue Links:
Duplicate
is duplicated by ZBX-3243 Network error while retrieving IPMI data Closed

 Description   

IPMI partially work.

IPMI is working fine with ipmitool, but partially with zabbix.

A request to sensor "Planar Temp" works well but a request "Voltage 1" or "Voltage 2" doesn't work while it works with ipmitool :

Log zabbix :
In substitute_simple_macros (data:'ipmi.get.voltage1')
In substitute_simple_macros (data:'10.10.10.65')
In get_value() key:'ipmi.get.voltage1'
In get_value_ipmi(key:ipmi.get.voltage1)
In init_ipmi_host([10.10.10.65]:623)
In get_ipmi_host([10.10.10.65]:623)
In get_ipmi_sensor_by_name() Voltage 1@[10.10.10.65]:623
In read_ipmi_sensor() Voltage 1@[10.10.10.65]:623
In got_thresh_reading()
Error 0x10000cb while read threshold sensor
End of get_value():NETWORK_ERROR
In substitute_simple_macros (data:'ipmi.get.voltage2')
In substitute_simple_macros (data:'10.10.10.65')
IPMI Host 10059 is unreachable. Skipping [ipmi.get.voltage2]

But by ssh on the zabbix server :

  1. ipmitool -H10.10.10.65 -UXXX -PYYY sdr get 'Voltage 1' | fgrep "Sensor Reading" | awk ' {print $4}'
    232
    # ipmitool -H10.10.10.65 -UXXX -PYYY sdr get 'Voltage 2' | fgrep "Sensor Reading" | awk '{print $4}

    '
    232

So...



 Comments   
Comment by Alfred Sawaya [ 2010 Nov 05 ]

Sorry, it was on 1.8.2 Version ! (can't I modify this ?!)

Comment by Alexei Vladishev [ 2012 Aug 27 ]

Apologies for the long delay, is there any updates on this issue?

Comment by Alexander Vladishev [ 2015 Feb 08 ]

Feel free to reopen if issue still exists in latest versions of Zabbix.





[ZBX-3186] Patch to add IPMI connection hack support Created: 2010 Nov 05  Updated: 2019 Aug 28  Resolved: 2019 Aug 28

Status: Closed
Project: ZABBIX BUGS AND ISSUES
Component/s: Server (S)
Affects Version/s: 1.8.3
Fix Version/s: None

Type: Incident report Priority: Major
Reporter: Christoph Berg Assignee: Unassigned
Resolution: Won't fix Votes: 2
Labels: ipmi, macosx, patch
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

Debian Linux 5.0 running a self compiled Zabbix server 1.4.3 linked against a self compiled libopenipmi 2.0.16 (Debian disabled OpenSSL support for whoever knows)


Attachments: Text File zabbix-1.8.3-support-for-ipmi-hacks-dbschema-1.patch     Text File zabbix-1.8.3-support-for-ipmi-hacks-frontend-php.patch     Text File zabbix-1.8.3-support-for-ipmi-hacks-server-1.patch    
Issue Links:
Duplicate

 Description   

We use Zabbix to monitor MacOS X Server hardware having an integrated IPMI solution provided by Intel. As Intel IPMI controllers have a bug when using RMCP+ for authentication and MacOS X does not seem to support using an alternative algorithm, Zabbix IPMI agent is not able to read values from the chip in its current form.

Provided is a patch, which allows to activate all supported hacks by OpenIPMI for connections. The only problem I could not solve is the non working mass update in the PHP frontend, because of the multiple checkboxes used to activate the hacks. Perhaps you have an idea how to solve this.

One issue I encountered with the patch is, if I change the refresh rate of the IPMI sensors from the default 30 to 60 seconds, I get these messages:

31437:20101105:103732.519 Item [dn21-mac-web5:sensor[temp2]] error: Error 0x10000c3 while reading threshold sensor
31437:20101105:103732.525 IPMI Host [dn21-mac-web5]: first network error, wait for 15 seconds
31437:20101105:103826.528 Item [dn21-mac-web5:sensor[temp2]] error: Error 0x10000ff while reading threshold sensor
31437:20101105:103826.537 Disabling IPMI host [dn21-mac-web5]
31437:20101105:104029.840 Enabling IPMI host [dn21-mac-web5]
31437:20101105:104232.891 Item [dn21-mac-web5:sensor[temp2]] error: Error 0x10000c3 while reading threshold sensor
31437:20101105:104232.896 IPMI Host [dn21-mac-web5]: first network error, wait for 15 seconds
31437:20101105:104326.900 Item [dn21-mac-web5:sensor[temp2]] error: Error 0x10000ff while reading threshold sensor
31437:20101105:104326.909 Disabling IPMI host [dn21-mac-web5]
31437:20101105:104530.332 Enabling IPMI host [dn21-mac-web5]
31437:20101105:104732.379 Item [dn21-mac-web5:sensor[temp2]] error: Error 0x10000c3 while reading threshold sensor
31437:20101105:104732.384 IPMI Host [dn21-mac-web5]: first network error, wait for 15 seconds
31437:20101105:104826.388 Item [dn21-mac-web5:sensor[temp2]] error: Error 0x10000ff while reading threshold sensor
31437:20101105:104826.397 Disabling IPMI host [dn21-mac-web5]

Changing it back to the default fixes this issue. I don't know if this is due to the patch or if allow IPMI sensors have this problem.

I splitted the patches in three parts, so it is - hopefully - easier to review.



 Comments   
Comment by richlv [ 2010 Nov 05 ]

hmm, very interesting. the biggest problems i see are that this is something very specific, and that it involves database schema changes.

maybe it's worth starting a thread on the forum (if there's not one already) to find out how many other people are seeing these issues

Comment by Christoph Berg [ 2010 Nov 08 ]

I just noticed, that the activation of the hack is not necessary on two of the three machines we want to monitor. The series having this bug is the Apple Xserve 1.1 series, but I'm in no position to replace this hardware, so we have to live with its "interpretation" of the spec.

As suggested I will start a forum thread, but looking through the posts dealing with IPMI it sounds like nobody had this problem before.

Comment by Christian Nancy [ 2010 Nov 09 ]

Slightly different although somewhat related:

I just upgraded from 1.8.2 to 1.8.3. Monitoring a Dell PE 2950 via IPMI. Before the upgrade my log was showing:

25622:20101108:214502.685 IPMI Host [Dell PowerEdge 2950]: another network error, wait for 15 seconds
25638:20101108:214616.733 IPMI Host [Dell PowerEdge 2950]: first network error, wait for 15 seconds
25638:20101108:214922.853 IPMI Host [Dell PowerEdge 2950]: first network error, wait for 15 seconds

But the sensor would be read and the data would show in Zabbix.

After the upgrade, the log shows:

27596:20101108:222031.838 Enabling IPMI host [Dell PowerEdge 2950]
27595:20101108:222422.152 Item [Dell PowerEdge 2950:Ambient_Temp] error: Error 0x10000c3 while reading threshold sensor
27595:20101108:222422.166 IPMI Host [Dell PowerEdge 2950]: first network error, wait for 15 seconds
27595:20101108:222516.173 Item [Dell PowerEdge 2950:Ambient_Temp] error: Error 0x10000ff while reading threshold sensor
27595:20101108:222516.178 Disabling IPMI host [Dell PowerEdge 2950]

And repeat again and again, also no readout in Zabbix.

Nothing else was changed on the Zabbix server, ipmitool on command line returns correct reading. I did not apply the patch discussed in this bug report as it doesn't pertain to my situation.

So something in the way Zabbix uses the ipmitool has changed from 1.8.2 to 1.8.3 which makes reading no longer working (unless of course this is an operator error, which is entirely possible).

Please let me know if this is off-topic and where should I post if that's the case.

Thanks,
Christian

Comment by Christoph Berg [ 2010 Nov 09 ]

As I said above, you can try reducing the time between two sensor checks. In my case switching back from 60 to 30 seconds worked.

Comment by Christoph Berg [ 2010 Nov 09 ]

Added a forum post for discussion: http://www.zabbix.com/forum/showthread.php?p=75170

Comment by Christian Nancy [ 2010 Nov 10 ]

It appears that in my case the error only happens every few minutes (varies) and most other times, it polls the data correctly. It has happened 427 times since I upgraded and restarted at 11/08 22:09. I've changed the interval to 30 seconds as suggested but that didn't have any impact.

I think I'll leave it at that for now. I changed the interval to 5 minutes to minimize the chatty logs.

Thank,
Christian

Comment by Rick [ 2010 Nov 13 ]

I've encountered the same issue trying to use OpenIPMI's ipmilan tool to create a udp server for Zabbix to query. If you run ipmilan with the -n and -d options you can see that Zabbix leaves the connection open but the ipmilan server "times the connection out". After that successive queries fail with an invalid SID message.

I see a patch in 1.8.4 that seems to refer to dropping and re-establishing connections which I'm going to investigate this week and will post my observations

Comment by Sergey Syreskin [ 2010 Nov 29 ]

It seems that I have a similar problem: https://support.zabbix.com/browse/ZBX-3243?focusedCommentId=35764

Comment by Blakkheim.GW [ 2011 Mar 07 ]

I have exactly the same issue than then one described by Christian Nancy. Since upgrade to 1.8.3 from 1.8.2, I have these errors in Zabbix logs and data is not polled correctly, only one check is done from time to time. Reducing the interval has no effect.

Error 0x10000c3 while reading threshold sensor
and then :
Error 0x10000ff while reading threshold sensor

I've just upgraded to 1.8.4 today but the issue stills here.

Comment by Aleksandrs Saveljevs [ 2016 Jan 26 ]

Does anybody still experience this issue with the latest versions of Zabbix?

Comment by Tyler French [ 2017 Oct 02 ]

@Aleksandrs Savelijevs Late and random but i just happened to notice this issue occurs and then "fixes" it self on some SuperMicro X10DRW-E devices that we are monitoring via IPMI. The error reported was

error 0x10000ff while reading threshold sensor
Comment by Glebs Ivanovskis (Inactive) [ 2017 Nov 29 ]

Dear frenchtoasters, I would suggest to open a new ticket. This one seems to be long dead.

Comment by Vladislavs Boborikins (Inactive) [ 2019 Aug 28 ]

Hello,

Since this version of Zabbix is no longer supported, we've decided not to prioritize this bug for the near future and close the issue with "Won't fix" resolution.

Please let us know if this decision should be reconsidered.

Regards
Vladislavs





[ZBX-2164] ipmi monitoring does not work with 2.0.13 libopenipmi from ubuntu Created: 2010 Mar 12  Updated: 2017 May 30  Resolved: 2012 Feb 20

Status: Closed
Project: ZABBIX BUGS AND ISSUES
Component/s: Proxy (P), Server (S)
Affects Version/s: None
Fix Version/s: None

Type: Incident report Priority: Critical
Reporter: richlv Assignee: Unassigned
Resolution: Won't fix Votes: 0
Labels: ipmi
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Description   

ipmi monitoring does not work with 2.0.13 libopenipmi from ubuntu. returns 0x16 connection error.
should be figured out why



 Comments   
Comment by Aleksandrs Saveljevs [ 2010 Apr 20 ]

IPMI does not work with libOpenIPMI 2.0.14 on Debian either. Upgrading to libOpenIPMI 2.0.16 solved the problem.

Comment by Alexei Vladishev [ 2011 Aug 27 ]

Not sure if this should be considered as a bug.

Comment by Alexei Vladishev [ 2012 Feb 20 ]

It appeared to be libopenipmi related problem.





[ZBX-2071] Zabbix frontend fails on IPMI sensors with trailing spaces Created: 2010 Feb 24  Updated: 2017 May 30  Resolved: 2012 Jan 31

Status: Closed
Project: ZABBIX BUGS AND ISSUES
Component/s: Frontend (F)
Affects Version/s: 1.8.1
Fix Version/s: 2.0.0rc1

Type: Incident report Priority: Major
Reporter: Yuri Vasilevski Assignee: Unassigned
Resolution: Fixed Votes: 0
Labels: ipmi
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Duplicate
duplicates ZBXNEXT-903 Redesign Configuration->Items Closed

 Description   

Zabbix frontend should not sanitize the input on IPMI sensor field as sensor names can contain trailing spaces.

Here at UNAM we have a cluster with HP ProLiant DL145 G2 computing nodes that support IPMI via iLO, the problem is that among the 22 sensors it has, two of them have names with trailing spaces (i.e. "VBAT " and "VCC5V ") which zabbix is able to monitor only if I manually change the "ipmi_sensor" column from table "items" in my database.

So the problem is that I'm unable to define the items to monitor for this IPMI sensors via GUI, be it manually adding sensor information or exporting to xml -> modifying sensor name in xml file -> importing from xml.

Thanks,
Yuri.



 Comments   
Comment by Eduards Samersovs (Inactive) [ 2012 Jan 25 ]

Resolved in ZBXNEXT-903

Comment by Alexei Vladishev [ 2012 Jan 30 ]

Reopening, should be closed only after proper testing.

Comment by Alexei Vladishev [ 2012 Jan 31 ]

Tested under ZBXNEXT-903, works fine.

Comment by richlv [ 2013 Mar 28 ]

also see ZBX-6445





Generated at Thu Mar 28 18:24:39 EET 2024 using Jira 9.12.4#9120004-sha1:625303b708afdb767e17cb2838290c41888e9ff0.