Test performed with single poller configured in server config file.
Tested two snmp hosts: 1st - a Dlink swith, 2nd - snmp (net-snmp) daemong running of localhost.
To show captured packets I use customized columns set and snmp traffic is decrypted (Wireshark has such a feature).
Minor note:
Difference with D-link (agent) that it provides engineBoots engineTime which are =0 for first "usmStatsUnknownEngineIDs" report, so zabbix (manager) should perform additional get-request with correct engineID to receive real engineBoots, engineTime.
Local hosts provides real engineBoots, engineTime for first "usmStatsUnknownEngineIDs" report, so additional get-request is not required and is not performed.
Here and next I show 2 consecutive polls for a single item, divided by a line break.
Vanilla zabbix server, right after restart:
For D-link:
No. Time Source Destination Length Protocol Info SRC port Engine-Boots Engine-Time Priv Auth Engine-MAC Enterprise
47 20:03:28.039796000 my-zabbix snmp-3550-6 108 SNMP get-request 60977 0 0 Not set Not set
48 20:03:28.047956000 snmp-3550-6 my-zabbix 148 SNMP report SNMP-USER-BASED-SM-MIB::usmStatsUnknownEngineIDs.0 161 0 0 Not set Not set 00:1b:11:b5:d3:d2 D-Link Systems, Inc.
49 20:03:28.048244000 my-zabbix snmp-3550-6 176 SNMP get-request SNMPv2-MIB::sysDescr.0 60977 0 0 Set Set 00:1b:11:b5:d3:d2 D-Link Systems, Inc.
50 20:03:28.057678000 snmp-3550-6 my-zabbix 182 SNMP report SNMP-USER-BASED-SM-MIB::usmStatsNotInTimeWindows.0 161 12 1165647 Set Set 00:1b:11:b5:d3:d2 D-Link Systems, Inc.
51 20:03:28.057927000 my-zabbix snmp-3550-6 178 SNMP get-request SNMPv2-MIB::sysDescr.0 60977 12 1165647 Set Set 00:1b:11:b5:d3:d2 D-Link Systems, Inc.
52 20:03:28.067628000 snmp-3550-6 my-zabbix 222 SNMP get-response SNMPv2-MIB::sysDescr.0 161 12 1165647 Set Set 00:1b:11:b5:d3:d2 D-Link Systems, Inc.
99 20:03:38.132274000 my-zabbix snmp-3550-6 108 SNMP get-request 43853 0 0 Not set Not set
100 20:03:38.166511000 snmp-3550-6 my-zabbix 148 SNMP report SNMP-USER-BASED-SM-MIB::usmStatsUnknownEngineIDs.0 161 0 0 Not set Not set 00:1b:11:b5:d3:d2 D-Link Systems, Inc.
101 20:03:38.166866000 my-zabbix snmp-3550-6 178 SNMP get-request SNMPv2-MIB::sysDescr.0 43853 12 1165657 Set Set 00:1b:11:b5:d3:d2 D-Link Systems, Inc.
102 20:03:38.185150000 snmp-3550-6 my-zabbix 222 SNMP get-response SNMPv2-MIB::sysDescr.0 161 12 1165657 Set Set 00:1b:11:b5:d3:d2 D-Link Systems, Inc.
For localhost:
No. Time Source Destination Length Protocol Info SRC port Engine-Boots Engine-Time Priv Auth Engine-MAC Enterprise
153 20:03:50.260837000 127.0.0.1 127.0.0.1 106 SNMP get-request 41899 0 0 Not set Not set
154 20:03:50.261015000 127.0.0.1 127.0.0.1 160 SNMP report SNMP-USER-BASED-SM-MIB::usmStatsUnknownEngineIDs.0 161 9 124074 Not set Not set net-snmp
155 20:03:50.261213000 127.0.0.1 127.0.0.1 190 SNMP get-request SNMPv2-MIB::sysDescr.0 41899 9 124074 Set Set net-snmp
156 20:03:50.261320000 127.0.0.1 127.0.0.1 254 SNMP get-response SNMPv2-MIB::sysDescr.0 161 9 124074 Set Set net-snmp
319 20:04:20.538908000 127.0.0.1 127.0.0.1 106 SNMP get-request 55691 0 0 Not set Not set
320 20:04:20.539180000 127.0.0.1 127.0.0.1 160 SNMP report SNMP-USER-BASED-SM-MIB::usmStatsUnknownEngineIDs.0 161 9 124104 Not set Not set net-snmp
321 20:04:20.539618000 127.0.0.1 127.0.0.1 190 SNMP get-request SNMPv2-MIB::sysDescr.0 55691 9 124105 Set Set net-snmp
322 20:04:20.540077000 127.0.0.1 127.0.0.1 254 SNMP get-response SNMPv2-MIB::sysDescr.0 161 9 124104 Set Set net-snmp
As we see in case of D-link after zabbix restart a zabbix process (for details see ZBX-8385) once learned by engineBoots, engineTime it will reuse them, so any next polls by this process will not perform second get-request to get the engineBoots, engineTime. 2nd and all next sessions will have 2 get-response pairs.
Localhost sessions shows 2 get-response pairs every time.
And here is a question - why do we spend some time to discover engineID?
Zabbix a daemon, it could cache and reuse it later!
What I did - I hardcoded engineID in zabbix sources (at start for "zbx_snmp_open_session" function, for snmpV3) to see what I can get in a result.
// u_char engine_id[] = { 0x80, 0x00, 0x00, 0xab, 0x03, 0x00, 0x1b, 0x11, 0xb5, 0xd3, 0xd2 }; //D-link
u_char engine_id[] = { 0x80, 0x00, 0x1f, 0x88, 0x80, 0x93, 0x4e, 0xde, 0x66, 0xfb, 0x80, 0x95, 0x53, 0x00, 0x00, 0x00, 0x00 }; //localhost
session.securityEngineID = engine_id;
session.securityEngineIDLen = sizeof(engine_id);
And I was happy to see success!
Patched zabbix server, right after restart:
hardcoded D-link's engineID:
No. Time Source Destination Length Protocol Info SRC port Engine-Boots Engine-Time Priv Auth Engine-MAC Enterprise
17 14:36:18.073879 my-zabbix snmp-3550-6 176 SNMP get-request SNMPv2-MIB::sysDescr.0 35757 0 0 Set Set 00:1b:11:b5:d3:d2 D-Link Systems, Inc.
18 14:36:18.083926 snmp-3550-6 my-zabbix 182 SNMP report SNMP-USER-BASED-SM-MIB::usmStatsNotInTimeWindows.0 161 12 714020 Set Set 00:1b:11:b5:d3:d2 D-Link Systems, Inc.
19 14:36:18.084084 my-zabbix snmp-3550-6 178 SNMP get-request SNMPv2-MIB::sysDescr.0 35757 12 714020 Set Set 00:1b:11:b5:d3:d2 D-Link Systems, Inc.
20 14:36:18.094461 snmp-3550-6 my-zabbix 222 SNMP get-response SNMPv2-MIB::sysDescr.0 161 12 714020 Set Set 00:1b:11:b5:d3:d2 D-Link Systems, Inc.
23 14:36:28.165773 my-zabbix snmp-3550-6 178 SNMP get-request SNMPv2-MIB::sysDescr.0 38450 12 714030 Set Set 00:1b:11:b5:d3:d2 D-Link Systems, Inc.
24 14:36:28.176490 snmp-3550-6 my-zabbix 222 SNMP get-response SNMPv2-MIB::sysDescr.0 161 12 714031 Set Set 00:1b:11:b5:d3:d2 D-Link Systems, Inc.
hardcoded localhost's engineID:
No. Time Source Destination Length Protocol Info SRC port Engine-Boots Engine-Time Priv Auth Engine-MAC Enterprise
1 13:30:50.918574000 127.0.0.1 127.0.0.1 188 SNMP get-request SNMPv2-MIB::sysDescr.0 48864 0 0 Set Set net-snmp
2 13:30:50.918964000 127.0.0.1 127.0.0.1 176 SNMP report SNMP-USER-BASED-SM-MIB::usmStatsNotInTimeWindows.0 161 20 2419 Not set Set net-snmp
3 13:30:50.919199000 127.0.0.1 127.0.0.1 189 SNMP get-request SNMPv2-MIB::sysDescr.0 48864 20 2419 Set Set net-snmp
4 13:30:50.919468000 127.0.0.1 127.0.0.1 253 SNMP get-response SNMPv2-MIB::sysDescr.0 161 20 2419 Set Set net-snmp
5 13:31:20.995176000 127.0.0.1 127.0.0.1 189 SNMP get-request SNMPv2-MIB::sysDescr.0 46660 20 2449 Set Set net-snmp
6 13:31:20.995528000 127.0.0.1 127.0.0.1 253 SNMP get-response SNMPv2-MIB::sysDescr.0 161 20 2449 Set Set net-snmp
We can see that starting from 2nd session, both hosts have only one get/response pair to get a value.
Zabbix (manager) when initiating session just specified engineID, and library itself "on the fly" added correct (recalculated, according to RFC) engineBoots, engineTime to very first get request.
So traffic is decreased, time spent decreases, load on monitored device decreased!
I checked also bulk-get behavior (for several snmp items, starting from 2.2.3) - for 1st get-request zabbix sent the same 106|108 bytes packet, which contain empty "variable-bindings". So from 2.2.3 traffic and time "overhead" is not so big as previously, but still.
What we need is to to cache engineID-IP pairs and reuse engineID later.
Where cache it - maybe for example in zabbix configuration cache together with interfaces IP.
Just FYI - libnetsnmp have interesting functions:
- free_enginetime(unsigned char *engineID, size_t engineID_len) - to remove particular engineID entry from the cache
- free_etimelist() - to clean out etimelist cache completely.
During experiments I tried to use both - both are working as expected.
Maybe this function should be used as well for specific cases like host (with IP) has been removed from zabbix etc.
It can provide some more possibilities - for example detecting duplicated enineIDs and printing very nice error messages to zabbix log (and do not cache them of course) etc.
During investigation I used other functions as well, for example dump_etimelist(); (which requires net-snmp recompiled with --enable-testing-code option.)
So I was able to print to zabbix log (as library's STDERR) the etimelist and make sure that it works as I understand it.
oh, btw, thanks to asaveljevs who helped me to construct correctly engine_id[] array after 2 hours of my own unsuccessful attempts 
Can this be an option?
Like snmp bulk requests?
If option is not set, than zabbix per each request does discovering engineid and time ?
Cross link - ZBXNEXT-3940 - asks to have ability to manually drop library's cache.
If Zabbix is aware of snmpEngineId at the daemon level it could expose it as an item. Then one could create a trigger on engineId change (e.g. to reload net-snmp cache).
ZBXNEXT-8823 is related to this