[ZBXNEXT-9135] Zabbix agent does not send events from "Forwarded Events" event log Created: 2017 Jun 26  Updated: 2024 Apr 15

Status: Reopened
Project: ZABBIX FEATURE REQUESTS
Component/s: Agent (G)
Affects Version/s: None
Fix Version/s: None

Type: Change Request Priority: Minor
Reporter: Oleg Ivanivskyi Assignee: Andris Zeila
Resolution: Unresolved Votes: 0
Labels: eventlog
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

Windows server > 2008

Issue Links:
Duplicate
Team: Team A
Sprint: Sprint 11, Sprint 12, Sprint 13, Sprint 14, Sprint 15, Sprint 16, Sprint 17, Sprint 18, Sprint 19, Sprint 20
Story Points: 0.25

 Description   

It’s possible for a Windows server to forward its events to a “subscribing” server. In this scenario the collector server can become a central repository for Windows logs from other servers in the network. Zabbix agent can be installed on the collector and can be used to monitor received logs from other servers.

The issue is that Zabbix agent does not send events from the "Forwarded Events" event log from collector. It does not generate any errors also.

Details:

  • Zabbix agent has 2 items: 
    eventlog[ForwardedEvents,,,,4625,,]
eventlog[Security,,,,4625,,]
  • It sends events from the Security log, but does not send for ForwardedEvents
  • It does not generate an error for ForwardedEvents also
  • Tried to change user from System to administrator - didn't help

Debug has no clue about the root cause:

  6328:20170625:203913.873 got [{"response":"success","data":[{"key":"eventlog[ForwardedEvents,,,,4625,,]","delay":1,"lastlogsize":0,"mtime":0},{"key":"eventlog[Security,,,,4625,,]","delay":1,"lastlogsize":47652,"mtime":0}]}]

  6328:20170625:203923.490 In process_active_checks() server:'<some IP>' port:10051
  6328:20170625:203923.502 In initialize_eventlog6() source:'ForwardedEvents' previous lastlogsize:0
  6328:20170625:203923.502 In zbx_open_eventlog6()
  6328:20170625:203923.502 End of zbx_open_eventlog6():SUCCEED FirstID:5544 LastID:12572 numIDs:7028
  6328:20170625:203923.502 In zbx_get_handle_eventlog6(), previous lastlogsize:0
  6328:20170625:203923.502 End of zbx_get_handle_eventlog6():SUCCEED
  6328:20170625:203923.519 End of initialize_eventlog6():SUCCEED
  6328:20170625:203923.519 In process_eventlog6() source: 'ForwardedEvents' previous lastlogsize: 0, FirstID: 5544, LastID: 12572
  6328:20170625:203923.519 In zbx_get_eventlog_message6() EventRecordID:5544
  6328:20170625:203923.614 End of zbx_get_eventlog_message6():SUCCEED
  6328:20170625:203923.627 End of process_eventlog6():SUCCEED
  6328:20170625:203923.627 In finalize_eventlog6()
  6328:20170625:203923.627 End of finalize_eventlog6():SUCCEED
  6328:20170625:203923.627 In need_meta_update() key:eventlog[ForwardedEvents,,,,4625,,]
  6328:20170625:203923.645 End of need_meta_update():FAIL

  6328:20170625:203924.738 In process_active_checks() server:'<some IP>' port:10051
  6328:20170625:203924.738 In initialize_eventlog6() source:'ForwardedEvents' previous lastlogsize:0
  6328:20170625:203924.738 In zbx_open_eventlog6()
  6328:20170625:203924.755 End of zbx_open_eventlog6():SUCCEED FirstID:5544 LastID:12584 numIDs:7040
  6328:20170625:203924.755 In zbx_get_handle_eventlog6(), previous lastlogsize:0
  6328:20170625:203924.755 End of zbx_get_handle_eventlog6():SUCCEED
  6328:20170625:203924.770 End of initialize_eventlog6():SUCCEED
  6328:20170625:203924.770 In process_eventlog6() source: 'ForwardedEvents' previous lastlogsize: 0, FirstID: 5544, LastID: 12584
  6328:20170625:203924.770 In zbx_get_eventlog_message6() EventRecordID:5544
  6328:20170625:203924.865 End of zbx_get_eventlog_message6():SUCCEED
  6328:20170625:203924.865 End of process_eventlog6():SUCCEED
  6328:20170625:203924.879 In finalize_eventlog6()
  6328:20170625:203924.879 End of finalize_eventlog6():SUCCEED
  6328:20170625:203924.879 In need_meta_update() key:eventlog[ForwardedEvents,,,,4625,,]
  6328:20170625:203924.879 End of need_meta_update():FAIL

Note, log size is ~30MB (not "lastlogsize:0"). Agent can see that the log has new events but does not send them (e.g. LastID: 12572 --> LastID: 12584).

 Comments   
Comment by Andris Zeila [ 2017 Jul 03 ]

It appears that query Event/System[EventRecordID>1234] for forwarded events works only if the event with corresponding event record id exists. So for example if we have event with record id 1, then query Event/System[EventRecordID>0] will return empty set while Event/System[EventRecordID>=1] will work.

The first event record id is detected by querying all events and reading the record id of the first event. However for forwarded events there is an event without event record id.

Also the forwarded events are not sorted by event record ids, querying by date/time doesn't seem to be working either. Using one log per forwarded event source might be used as a workaround (at least it should fix event ordering by record ids), however there is no easy way to do it (would require downloading sdk and compiling dll just to create new logs to store forwarded events).

Possible solution might to use bookmarked events from last query instead of reading events with record id greater than the last read event. However it still must be investigated and also would require redesigning part of eventlog processing.

Comment by Andris Zeila [ 2017 Oct 20 ]

Bookmarking last read event and then reading next batch of events from the bookmark does work, not sure about perfomance hit though, as we have to query all events (EvtQuery(NULL, channel, L"Event/System[EventRecordID != 0]"), not only events starting with the specified EventRecordId (EvtQuery(NULL, channel, L"Event/System[EventRecordID > <lastid>]).

However the problem is identifying the last event. The EventRecordId in forwarded events is the original event record id on the source computer. When collecting events from multiple computers we could have matching EventRecordID (actually that could happen when collecting events from different event channels of a single computer). We could try using hash function from EventRecordID + Channel + Computer, however we would need some additional logic to handle possible hashsum matching. Otherwise event reading would stuck in a loop reading the same events over and over again.

Comment by Andris Zeila [ 2017 Oct 23 ]

So the possibe workaround for forwarded events:

  • identify the eventlog items working with forwarded events. I understand the user can choose any channel to collect forwarded events (or in rare cases even create new one) - so we can't use channel name. That leaves adding another parameter (8th!). For channel collecting forwarded events we need to apply the following logic:
    • on the first request (after agent has been started):
      • query all events and scan for the last processed event (by hashsum created from EventRecordID + Channel + Computer and stored in lastlogsize property).
      • bookmark the found request (last read event bookmark must be kept for each eventlog request)
    • query all events and use EvtSeek to find the next event after bookmarked
    • read the events from query, for each event:
      • update the last read event bookmark
      • calculate the last event hashsum from EventRecordID + Channel + Computer

The concerns:

  • Adding another parameter to already cluttered eventlog parameter list.
  • In the case of hashsum collision after agent restart a range of events would be duplicated.
  • Performance question. With original logic we were querying only events with EventRecordId > <lastlogsize> - so basically only new events. Will querying all events and then locating the bookmarked event result in performance impact?
  • Resource usage. Agent will keep open bookmarks between eventlog requests. It should be minimal impact to number of open handles unless agent has to process a large number of eventlog items.
Comment by Andris Zeila [ 2017 Oct 23 ]

Oh, one more thought about eventlogs. Currently we are reading all events from channel and then doing filtering on agent side with regexp parameter. I'm not sure how 'friendly' it is for Windows administrators and if allowing them to specify eventlog queries (Event/System[EventID=4] for example) wouldn't be better.

Comment by Martins Valkovskis [ 2017 Nov 06 ]

Added to documentation for 2.2, 3.0, 3.2, 3.4, 4.0 (see the comment column for the 'eventlog' item).

Comment by dimir [ 2024 Apr 15 ]

From internal discussion:

This is not an easy task because forwarded events do not have an ID that agent needs to associate the event with lastlogsize.

Moving to ZBXNEXT.

Generated at Wed Apr 24 02:25:27 EEST 2024 using Jira 9.12.4#9120004-sha1:625303b708afdb767e17cb2838290c41888e9ff0.