Would be nice if you could provide more information. Timestamps and values from Latest Data, configuration of this host, whether it is monitored by proxy, etc.

Ill give you my details. So I am running 3.4.8. The affected device is a HP 5406 chassis, its reporting to a Zabbix 3.4.8 proxy server on the same subnet that is uploading data to a cloud based zabbix server.

Hosting  using Template Module Interfaces SNMPv2

Ill have to post latest data when the error occurs. Its not all the time. I see on the forum post someone was able to post the last reported data over the last 10 reports (with one missing), I don't know how to do that, any advice.

Discovered by Network Interfaces Discovery

Name: Interface G3(HV-1-SAN-C): Outbound packets with errors
Type: SNMPv2 agent
Key: net.if.in[ifHCInOctets.147]
Host interface: x.x.x.x:161
SNMP OID: 1.3.6.1.2.1.31.1.1.1.6.147
SNMP community: {$SNMP_COMMUNITY}
Port: None
Type of information: Numeric (unsigned)
Units: bps
Update interval: 90

Preprocessing steps
Change Per second: None
Custom Mutipler: 8

Worked it out

I see on the forum post someone was able to post the last reported data over the last 10 reports (with one missing), I don't know how to do that, any advice.

If you mean this post then it's probably directly from the database, but you can get a similar result by going to Latest Data, selecting item, clicking Graph and choosing Values from the dropdown on the right.

Can you please create a clone of this item with the same SNMP OID but without any preprocessing steps and provide Latest Data for both items when the issue happens again?

Sorry for delay I was rest of yesterday. OK I have 3 ports running clones without pre-processing. They'll most likely fail tonight so Ill come back to you asap

Great! Thank you!

Ok This is the first reading. Over data reported @ 23:02 on Failed Data chart. It does look like it missing the previous reading which is what one of the posters from the forum mentioned. There is nothing in the proxy logs. There's a slow query in the server logs kind of at the same time but it doesn't say anything obvious to indicate its to do with this.

Second one. There appears to be much more test data than failed data. There is nothing about slow querys in the server logs for this one.

Over data @ 06:36:53

I've slightly rearranged your data:

It is very strange that raw value goes up and down, as it was coming from two different sources...

Zabbix has a built-in logic to ignore negative values of delta calculations – this is why we see less values for delta item.

I'm not very experienced with SNMP, but it seems that the only way to get to the bottom of the problem is to sniff network traffic between Zabbix and your device.

Since you are using 3.4 please change the type of your original item to be Dependent on the raw value item you've just added. We will get "synchronized" values this way and hopefully will be able to make more sense of them.

I wonder if Ive done something wrong then. Let me try once more a different way of getting the data.

To me this looks so strange, that I start to suspect some sort of device bug. After all, you are not the only one experiencing such troubles.

Just to confirm, when I clone the item it say 

Item with key "net.if.out[ifHCOutOctets.150]" already exists on "TGS CORE SWITCH 5400"

I assume I can pick a random number for OutOctet?

For SNMP and dependent items you can choose any key, it just has to be unique on the host/template level.

Oh ok, then I haven't cocked up, phew. Ill try and do the dependent link then. I was going to clone the host completely but remove the preprocessing from the cloned host items.

Ill do that as well as cloning the item and doing the dependency

Tim

Thank you!

By the way, do you see this happening on just one host or are there other devices affected? Any similarities between them?

Does this look ok?

Yes I do see it on an edge switch but not as often and havent been able to catch it. The core switch that I'm getting most of the logs from is a HP 5406 and its the SAN VLAN that triggers during backup time. The edge switches are 10Gb HP 1950s. There is a CCTV cluster attached to one of the edge switches and if there is a lot of CCTV traffic that will sometimes triggers as well.

Does this look ok?

Yes, if master item has no preprocessing and dependent item has Change per second and Multiplier 8.

Ah Ive got it the wrong way around then. That was showing the test without pre-processing as being the dependant to the original entry.

Ill swap them around

Tim

Latest data should be similar to your previous comments, but with both values collected at the same time.

OK. Ive just realised because the original items are auto discovery items I can not edit them. So are you saying the way I had it configured will be ok with the clone being the none pre-processing dependant with the original auto discovery item (with pre-processing) being the master?)

No, this way both values will be the same, both pre-processed.

ok, Ill disable the discovery items and create new failing pre-processing items and then create the non-pre-processing items there after.

More data, event is 14:49:32. The times aren't the same building up to it.

There is simply one value missing for "delta item" (it would have been negative and Zabbix ignored it):

But the big picture is the same – there is a huge jump in raw value...

Ha, it's starting to make some kind of sense, look:

1. 158296926293136 − 157196886691440 = 1100039601696
2. 1100039601696‍₁₀ = 1000000000011110111000001111100010000‍₂ (note that lonely 1 in the highest order!)
3. 11110111000001111100010000‍₂ = 64757520‍₁₀ (ignoring that faulty bit)
4. 64757520 / 90 seconds × 8 bits per octet = 5756224 bps ≃ 5.5 Mbps (which kinda makes sense)

Could it be that bits in counters just "flip" randomly from time to time?

Dear timguyuk, do your graphs look like this one?

Simular yes although the jump is so high that it appears like the other data is flat, it isn't, just the averages with the big jump cancel them out.

Thanks for you help on this.

So Is it zabbix or do you think its the snmp device?

Tim

As we see, Zabbix correctly calculates speed per second from raw values it gets. Raw values cause the issue. So there could be two explanations:

1. Zabbix incorrectly gets raw values
2. SNMP device returns incorrect values

According to your data there is a step in raw values after which all gets back to normal. This would mean that Zabbix starts getting values incorrectly and then continues to get values incorrectly. This is highly unlikely and therefore I don't believe that first option explains the issue well enough.

On the other hand SNMP implementation in switches and routers is typically low level, very close to hardware. And probably hardware itself is a bit simplified if we compare it to server and desktop computers. I suppose it is a plausible explanation that from time to time certain physical factors like voltage spike in the mains can cause glitches in SNMP layer.

There was the same problem, tell please how you solved it?

Same problem here

I think the problem related to zabbix not device,

because we have a cacti and there isn't any abnormal pic on same interface at same time

The same problem requires a solution.

same problem, can provide the necessary data

problem not only snmp, but with delta values processing

Same problem here, on Zabbix 4.2.4.

See attached graph for a 1 Gbps interface, coming from Template Module Interfaces SNMPv2, Item prototype "Interface {#IFNAME}({#IFALIAS}): Bits sent".

The values around the time of the spike are:

With an interval of 3 minutes, you can see the data is missing around 13:36.

Added to known issues for 3.0, 4.0, 4.4, 5.0.

Hi Martins Valkovskis, could you please clarify the status of this defect ZBX-14318 which is shown as - Resolution:Fixed  and Fix Version/s:5.0 (plan)  - is it already fixed or there is just a plan to fix it in version 5.0 ?

I believe the current consensus is that it is not an issue that can be fixed by Zabbix. So it is "fixed" in as far as notes have been added to documentation.

In that case - how we can explain situation where we are able to see the spike on a Zabbix graph and there is no - on Cacti = for the same device/interface and timestamp?

Hi Martins Valkovskis, some extra information to my previous posts:

Usually, the spikes happens when Zabbix do the calculation, but were able to catch some spikes even with raw data, before any calculations. 

There are spikes on down interfaces. (not connected)

**//+same device/same interface/same SNMP/ - no issues on Cacti system.

Nail, I have reopened the issue so that Zabbix developers may have another look at this issue.

Is it simply that the server or proxy doesnt have the processing ability to crunch the amount of data coming from a 10Gb connection snmp trap and so zabbix lets it slip? Do we need faster servers for 10Gb connection processing?

In that case - how we can explain situation where we are able to see the spike on a Zabbix graph and there is no - on Cacti = for the same device/interface and timestamp?

Cacti system has spikekill class which analyses data, calculates average values, standard deviations, etc, and filters impossible spikes, according to settings.
Smailov, could you attach raw data, ifHCOutOctets, received from same device, interface and collected with Zabbix and Cacti around the spikes? If they are the same, the only solution would be to filter those spikes.

Running Zabbix 5.0.0alpha.

Interfaces UP for this specific switch:

Zabbix interface discovery:

The "-raw" doesn't have any preprocessing.

The "-raw dep" is dependent on "-raw" applying the default preprocessing.

Interface UP with spike (graph):

Interface UP with spike (value):

Displaying 32 of 32 found
 
 
Interface UP on Cacti:

 
 
Interface DOWN with spike (graph):

 
Interface DOWN with spike (value):
 

Displaying 35 of 35 found
 
 
Interface DOWN with spike (graph):

 
Interface DOWN with spike (value):

Displaying 19 of 19 found
 
 
Interface DOWN with spike (graph):

 
Interface DOWN with spike (value):

Displaying 12 of 12 found
 
As you can see these spikes looks totally random.
 
Besides the fact that Cacti interval is 5 minutes and Zabbix is now configured to 1 minute, I got the same issue using the Zabbix interval of 3 and 5 minutes as well.

Spike in bps could be explained by some oddity in the way how switch updates counter of sent bytes.
There is a jump in bytes sent counter, after that growth and calculated bps is normal again.

But case, where from zero bits sent is calculated some non zero value, could only happen in Zabbix server. However I could not reproduce it. And reading through code did not give me any idea, how this could happen.

Please, let me know if there's any debug or anything else that I can do to help the troubleshooting of this issue.

Is only delta speed for second affected if so then maybe it is somehow related to time in the system for example time goes back.

wiper: From the posted data they are getting random data spikes, so does not seem likely.

In our case, we figured out that this was a hardware issue. The issue was caused by the SNMP bulk request that the hardware didn't know how to handle properly. It will be fixed in a new firmware version.
As a workaround, we disabled the SNMP bulk request on the host config and now it's working properly.
Also, this is the reason it was working on Cacti but not on Zabbix since Cacti wasn't using bulk requests.
Hope this can help other Zabbix users!
Thanks for the support and help.

No No

It is not a hardware issue. if it was, we will see a lot of spikes in all graphs, but there are a few spikes in graphs on random interfaces.

I also noticed that my BulkRequest is not enabled at all and still I have Spike in graphs

We are unable to reproduce the issue, if you can consistently reproduce the issue on small amount of items then maybe it would be possible for us to provide a patch for you that will include additional debug ?

zabbix 4.4.10 Found the problem again

zabbix 3.4.14 In 2018, the problem was found to be upgraded to 4.0 to fix, and now the problem is found again
https://support.zabbix.com/browse/ZBX-15215

Closing, because of the inability to reproduce.

We noticed this kind of behaviour when polling a Cisco ASR in high availability mode (so there is two physical devices, one logical vía SNMP). When the cluster failover to one device to another the bytes counter changes from the "physical" one in one router to the other and then you will find the peak. Maybe being able to use in preprocessing the value of another item will be useful so you can filter this situation.

Could you please be so kind and take a look at ZBX-25917, it appears that issue can happen when device is restarted and polled at same moment so device would send 0 before it updates own counters.