ZABBIX FEATURE REQUESTS

More efficient SNMP trapping

Details

  • Type: New Feature New Feature
  • Status: Closed Closed
  • Priority: Major Major
  • Resolution: Fixed
  • Affects Version/s: 1.9.3 (alpha)
  • Fix Version/s: None
  • Component/s: Proxy (P), Server (S)
  • Labels:
    None
  • Zabbix ID:
    Reviewed 2.0

Description

Zabbix will support new SNMP trap handling by using native (script-less) integration with NET-SNMP trap daemon. The integration will not require shell scripts or any other heavy processing, it NET-SNMP trap daemon will report directly to Zabbix Server using standard (one of: file, fifo, pipes, etc) high performance IPC methods.

Optional configurable trap pre-processors (such as SNMPTT) will be supported.

It will be implemented so that Zabbix will automatically sort traps and put corresponding hosts based on IP address.

Issue Links

Activity

Hide
richlv added a comment - - edited

any updates on where the development is currently heading ?

<rudolfs> haven't got around to the final tests (working on other issues...) snmptrapd have various issues when configuring for a direct output (to a file/fifo). it seems that the handler does not invoke forking or any other heavy processing.
so the most effective way might be a simple C application configured as the handler that writes the data to a file. the file will then be read by special trappers on the server/proxy. so this will be configurable on the main server or on any of the proxies. the final processing might consist of finding the correct host in the DB and run the trap against regexes configured in Zabbix.

Show
richlv added a comment - - edited any updates on where the development is currently heading ? <rudolfs> haven't got around to the final tests (working on other issues...) snmptrapd have various issues when configuring for a direct output (to a file/fifo). it seems that the handler does not invoke forking or any other heavy processing. so the most effective way might be a simple C application configured as the handler that writes the data to a file. the file will then be read by special trappers on the server/proxy. so this will be configurable on the main server or on any of the proxies. the final processing might consist of finding the correct host in the DB and run the trap against regexes configured in Zabbix.
Hide
Oleksiy Zagorskyi added a comment -

I decided to publish here some my thought (part of some discussion with devs) for public discussion.

Q: passing traps directly to zabbix server w/o intermediate layers would be much faster

A: "classic" shell handler runs shell script and after zabbix_sender (current official implementation) every time when snmp trap is received. As result a system is loaded and snmptrapd works slowly because of constant process forking. My deep experiments gave me a result - snmptrapd can't receive and log in real time more than ~ 16-20 traps/second. With the speed of the traps flow more that this value, snmptrapd starts to buffering received traps and log they with delay.

"perl" handler starts when snmptrapd starting, and it stay resident in the memory while snmptrapd running. When trap received - it makes TCP connection to zabbix_server and sent the message (trap) like a zabbix_sender. so, no any forks and as result - it is very-very fast. You can send raw message in some optimal way to the zabbix_server and parse this message on the server side (C code). I do not know any other "more direct" way without w/o intermediate layers.
additional benefits of "perl" handler - output format for traps v1 and v2 are identical an fixed.

Additional details of my experiment (in Russian) are available for developers at the internal wiki page.

Show
Oleksiy Zagorskyi added a comment - I decided to publish here some my thought (part of some discussion with devs) for public discussion. Q: passing traps directly to zabbix server w/o intermediate layers would be much faster A: "classic" shell handler runs shell script and after zabbix_sender (current official implementation) every time when snmp trap is received. As result a system is loaded and snmptrapd works slowly because of constant process forking. My deep experiments gave me a result - snmptrapd can't receive and log in real time more than ~ 16-20 traps/second. With the speed of the traps flow more that this value, snmptrapd starts to buffering received traps and log they with delay. "perl" handler starts when snmptrapd starting, and it stay resident in the memory while snmptrapd running. When trap received - it makes TCP connection to zabbix_server and sent the message (trap) like a zabbix_sender. so, no any forks and as result - it is very-very fast. You can send raw message in some optimal way to the zabbix_server and parse this message on the server side (C code). I do not know any other "more direct" way without w/o intermediate layers. additional benefits of "perl" handler - output format for traps v1 and v2 are identical an fixed. Additional details of my experiment (in Russian) are available for developers at the internal wiki page.
Hide
Oleksiy Zagorskyi added a comment -

After implementing ZBX-3105 we have:
zabbix_server [24855]: unknown parameter [StartSNMPTrappers] in config file [/etc/zabbix/zabbix_server.conf], line 2
/usr/local/etc/rc.d/zabbix_server_fast: WARNING: failed to start zabbix_server
Rudolfs, do not forget to provide new configuration parameters (StartSNMPTrappers and SNMPTrapperFile) and do not see such error

Show
Oleksiy Zagorskyi added a comment - After implementing ZBX-3105 we have: zabbix_server [24855]: unknown parameter [StartSNMPTrappers] in config file [/etc/zabbix/zabbix_server.conf], line 2 /usr/local/etc/rc.d/zabbix_server_fast: WARNING: failed to start zabbix_server Rudolfs, do not forget to provide new configuration parameters (StartSNMPTrappers and SNMPTrapperFile) and do not see such error
Hide
Rudolfs Kreicbergs added a comment -

This is still a development branch, thus a lot of weird things can happen So you probably would not want to use this branch. Currently I'm waiting for more specs to continue work on this branch.

Show
Rudolfs Kreicbergs added a comment - This is still a development branch, thus a lot of weird things can happen So you probably would not want to use this branch. Currently I'm waiting for more specs to continue work on this branch.
Hide
Oleksiy Zagorskyi added a comment -

A note just for case.
I registered already a old time some bug at a net-snmp bug-tracker http://sourceforge.net/tracker/?func=detail&aid=3053954&group_id=12694&atid=112694

First of all i must to say that i confused because of my old experiments with a different syslog daemons or with stopped syslogd. So system messages came to different places (physical console or log files).

Now description of problem will be more clear (i hope)

When snmptrapd receives a trap message this buggy message is added to /var/log/messages - "perl callback function 0x8240694 returned a scalar of type 6 instead of an integer, assuming 1 (NETSNMPTRAPD_HANDLER_OK)"

When i add some line (return 1 to the perl-script (according to Rufolf's suggestion) then the buggy message start coming to /var/log/debug.log - "perl callback function 0x82644fc returns 1"

If syslogd is stopped, then all messages come to the physical console.

Maybe we can somehow suppress all this messages at all? I recall that this problem did not exist in the net-snmp 5.3

Show
Oleksiy Zagorskyi added a comment - A note just for case. I registered already a old time some bug at a net-snmp bug-tracker http://sourceforge.net/tracker/?func=detail&aid=3053954&group_id=12694&atid=112694 First of all i must to say that i confused because of my old experiments with a different syslog daemons or with stopped syslogd. So system messages came to different places (physical console or log files). Now description of problem will be more clear (i hope) When snmptrapd receives a trap message this buggy message is added to /var/log/messages - "perl callback function 0x8240694 returned a scalar of type 6 instead of an integer, assuming 1 (NETSNMPTRAPD_HANDLER_OK)" When i add some line (return 1 to the perl-script (according to Rufolf's suggestion) then the buggy message start coming to /var/log/debug.log - "perl callback function 0x82644fc returns 1" If syslogd is stopped, then all messages come to the physical console. Maybe we can somehow suppress all this messages at all? I recall that this problem did not exist in the net-snmp 5.3
Hide
Rudolfs Kreicbergs added a comment -

Thank you for the info, Oleksiy. Added return statements for both OK and FAIL cases. The second message seems like a debug message so it should be ok, those can probably be switched off in the daemon configuration file.

Show
Rudolfs Kreicbergs added a comment - Thank you for the info, Oleksiy. Added return statements for both OK and FAIL cases. The second message seems like a debug message so it should be ok, those can probably be switched off in the daemon configuration file.
Hide
Rudolfs Kreicbergs added a comment -

Available in pre-1.9.6 r21580.

Show
Rudolfs Kreicbergs added a comment - Available in pre-1.9.6 r21580.

People

Vote (6)
Watch (6)

Dates

  • Created:
    Updated:
    Resolved: