Resolution: Unsupported version
Server: HPE - ProLiant DL380 Gen10 (P19720-B21)
OS: CentOS Linux release 7.9.2009
Kernel: Linux 3.10.0-1160.2.2.el7.x86_64
After a host with multiple NVMe's has been running a zabbix agent with the template (Template Module Linux block devices by Zabbix agent) for x period of time, the auto disk discovery will discover an garbage disk that is not present in the system, most likely an bad parsing is causing an shifted indent in the preprocessing, which causes an garbage device to be added to the lld list
Steps to reproduce:
- Setup an zabbix agent on HPE - ProLiant DL380 Gen10 (P19720-B21) server (might work just on server with multiple NVMe's)
- Create an Host on zabbix server using Template Module Linux block devices by Zabbix agent
- Let the agent run for few hours and check if there have occurred any strange disk discoveries, example: Block devices discovery: Disk 6006249559
- In into Monitoring -> Latest data -> current host -> Zabbix raw items, and see the history for Get /proc/diskstats, and find the entry that caused the garbage item to be discovered
(See P1.png) After a host ran for 20 minutes a device was discovered by the auto discovery
(See failure-zabbix.png) After going though step 4 (in Steps to reproduce) you can see that there are two errors in the output gotten from the preprocessing, one is that there is one 'undefined' key in the .values object with only an array of an length 1, and another key '6006249559' that is shifted by one. They keys are generated by indexing into the array at the index 2, hence the undefined key is generated because there is only item at index 0.
So, in short once in a while the data for one disk will be split into two arrays, which leads to shifted device discovery, which then leads to errors on the servers side when extracting information from the agent since there are no disks with the given name.