net-snmp.x86_64 1:5.7.2-49.el7 @centos7-base-x86_64
net-snmp-agent-libs.i686 1:5.7.2-49.el7 @centos7-base-x86_64
net-snmp-agent-libs.x86_64 1:5.7.2-49.el7 @centos7-base-x86_64
net-snmp-devel.i686 1:5.7.2-49.el7 @centos7-base-x86_64
net-snmp-libs.i686 1:5.7.2-49.el7 @centos7-base-x86_64
net-snmp-libs.x86_64 1:5.7.2-49.el7 @centos7-base-x86_64
net-snmp-perl.x86_64 1:5.7.2-49.el7 @centos7-base-x86_64
net-snmp-utils.x86_64 1:5.7.2-49.el7 @centos7-base-x86_64
zabbix-agent.x86_64 5.0.7-1.el7 @zabbix5.0
zabbix-proxy-mysql.x86_64 5.0.7-1.el7 @zabbix5.0
zabbix-release.noarch 5.0-1.el7 @zabbix4.0

Have you had a chance to read ZBX-8385 ?

It describes also cases when monitored device does not follow RFC requirements, which cause the issue.

Hi Oleksii,

Thanks for your help and pointing me to ZBX-8385. After reading it I can say ZBX-8385 describes two possible reasons but does not fit to my discovered problem.

From ZBX-8385:

We can split the "usmStatsNotInTimeWindows" case to two possible reasons:
a) there indeed are monitored snmpV3 devices which have identical msgAuthoritativeEngineID

This is not the case. I doublechecked and there are no duplicate snmpEngineIDs.

b) some of devices have incorrect behavior and don't follow RFC3414. For example I know device which always use engineBoot=1, even after reboot.

This doesn't fit either. The monitored device did not reboot and in all SNMPv3 packets of this device the snmpEngineBoot value is 0. The snmpEngineTime further did not overflow, this the snmpEngineBoot value of 0 is correct.

But ZBX-8385 describes an interesting fact:

Zabbix server|proxy is a multi-process application. Every poller|unreachable_poller|and_some_other_process_type when starts - it loads libnetsnmp.

Every such process keeps its own libsnmp's in-RAM cache.What we want to consider is "etimelist" (see ldc_time.c)

This explains why it is even possible that zabbix starts to poll the device with correct snmpEngineTime and wrong snmpEngineTime in parallel.

In the moment that happened I see the "first network error, wait for 15 seconds" and "temporarily disabling SNMP agent checks on host: host unavailable" errors in the logs.

From this behaviour and RFC3414 I would suggest to update the poller code of zabbix. If any instance of poller receives an usmStatsNotInTimeWindows error report from a device, that same instance should update its snmpEngineTime and snmpEngineBoot values according to the values in the report and try again. The device should only be treated unreachable if the retry with updated snmpEngineTime and snmpEngineBoot values fails again. This would make zabbix further more stable from an enduser perspective.

This is your friend: https://www.zabbix.com/documentation/5.0/manual/introduction/whatsnew500#manual_snmp_cache_clearing

Nothing add on zabbix side, this issue to be closed.

Don't get me wrong on closing current issue. But taking into account that you found new important details, this one should be closed.

If you still think that something is wrong, create a new issue taking into account all new details.

Yes, I absolutely don't understand why you close this case. I found an issue in zabbix, documented and reported it and suggested how to solve it. You pointed me to manually clear the snmp cache to "solve" this issue. This might be a workaround until the code gets fixed. Zabbix is promoted to "Automate monitoring of large, dynamic environments". How does this fit to manually clear snmp cache from time to time because we won't fix our code. Zabbix behaves unstable and the users have to manually intervene to have it working until it gets again unstable. I did invest a lot of time to analyze this issue to make zabbix a better software and you just ignore it. Great job, Thanks.

Taking into account all new details it is still the same: Zabbix behaves unstable because it doesn't correctly track snmpEngineTime with multiple pollers. Manually clearing the snmp cache is not a solution, it is only a workaround.

I've spent quite a lot of time to work with SNMP, especially v3, so now I'm pretty confident to say that there is no bug in zabbix.
Maybe there us something not like you expected or wanted, but that's not a bug, I'm sorry.

The RFC does require to not "track snmpEngineTime" but reject it because of security aspects.
If device does not follow RFC - that's not zabbix fault.

So, how do you explain that the zabbix poller process starts to use a complete made up snmpEngineTime value? In all my traces the monitored devices behave RFC compliant. They increase their values correctly like they should. Only Zabbix starts out of nothing to use weird snmpEngineTime values without any known reason. I even filtered my traces to find any other device having a snmpEngineTime value like the one Zabbix made up, but didn't find one. Nothing tells Zabbix to invent snmpEngineTime values.

That all made by library itself, not by zabbix.
It's possible that small number of zabbix server pollers "caught" the higher EngineTime for duplicated EngineID some time ago and continued to "count" the clock (in library's cache). Or device was rebooted.

snmpEngineBoot=0 (as you said) is almost 100% grantee (IMO) that the device does not follow RFC.

Try to restart zabbix or clean the cache I implemented.
I'm sorry, but I do not see much sense to argue here.

Related to ZBX-21557 ?