[ZBXNEXT-967] Logfile monitoring - regular expressions to search multiple lines by agent Created: 2011 Sep 14 Updated: 2024 Apr 08 |
|
| Status: | Reopened |
| Project: | ZABBIX FEATURE REQUESTS |
| Component/s: | Agent (G) |
| Affects Version/s: | 1.8.7 |
| Fix Version/s: | 1.8.8 |
| Type: | New Feature Request | Priority: | Major |
| Reporter: | Frank Hirschner | Assignee: | Zabbix Support Team |
| Resolution: | Unresolved | Votes: | 6 |
| Labels: | logmonitoring, regexps | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Environment: |
CentOS 5.6 |
||
| Issue Links: |
|
||||
| Description |
|
Logfile monitoring with regular expression is ok. But sometimes you need to search for log entries in more than one line. Example: foo foofoofoo foofoofoofoo error and much more here is the description foofoofoo foofoofoofoo regular expression: /^error.*\n.*$/m With this regular expression you will get both lines. But today it is not possible in zabbix |
| Comments |
| Comment by richlv [ 2011 Sep 14 ] |
|
most likely a dupe of |
| Comment by Oleksii Zagorskyi [ 2021 Apr 05 ] |
|
This is not duplicate, reopening it, as time to time people ask for that. |
| Comment by Gaetano Schuppe [ 2024 Apr 03 ] |
|
We really need this feature. It will likely be the dealbreaker that will make us switch to using the grafana stack for monitoring infra and services. |
| Comment by Matthew Steeves [ 2024 Apr 03 ] |
|
Hey Rashawn_Johnson, what's you use case? |
| Comment by Gaetano Schuppe [ 2024 Apr 03 ] |
|
We're trying to monitor the logs of a rails app. Below is an excerpt of the logs
I, [2024-04-03T20:00:25.689375 #1] INFO -- : [d2ec09f4-e1fd-4b05-a49f-2177b80a012b] Started GET "/" for 159.203.36.31 at 2024-04-03 20:00:25 +0000 [34ea335d-9627-4fc6-8b1f-957d6e2b46e9] app/controllers/application_controller.rb:172:in `set_timezone' [34ea335d-9627-4fc6-8b1f-957d6e2b46e9] app/controllers/application_controller.rb:56:in `execute_around_action' [34ea335d-9627-4fc6-8b1f-957d6e2b46e9] app/controllers/application_controller.rb:60:in `switch_locale' [34ea335d-9627-4fc6-8b1f-957d6e2b46e9] app/controllers/service_cards_controller.rb:57:in `endpoint' [34ea335d-9627-4fc6-8b1f-957d6e2b46e9] app/models/endpoint.rb:90:in `perform_action_as_user' [34ea335d-9627-4fc6-8b1f-957d6e2b46e9] app/models/endpoint.rb:83:in `perform_action' [34ea335d-9627-4fc6-8b1f-957d6e2b46e9] app/models/endpoint.rb:83:in `public_send' [34ea335d-9627-4fc6-8b1f-957d6e2b46e9] app/models/trust_adapter.rb:107:in `perform_post' [34ea335d-9627-4fc6-8b1f-957d6e2b46e9] app/services/http_utils.rb:92:in `perform_post' [34ea335d-9627-4fc6-8b1f-957d6e2b46e9] app/services/http_utils.rb:103:in `make_request' [34ea335d-9627-4fc6-8b1f-957d6e2b46e9] app/services/http_utils.rb:124:in `make_call' [34ea335d-9627-4fc6-8b1f-957d6e2b46e9] [34ea335d-9627-4fc6-8b1f-957d6e2b46e9] URI::InvalidURIError (bad URI(is not URI?): " http://127.0.0.1:2375/v1.24/containers/json"): E, [2024-04-03T19:59:27.412915 #1] ERROR -- : [34ea335d-9627-4fc6-8b1f-957d6e2b46e9] I, [2024-04-03T19:59:27.411767 #1] INFO -- : [34ea335d-9627-4fc6-8b1f-957d6e2b46e9] Completed 500 Internal Server Error in 22ms (ActiveRecord: 16.4ms | Allocations: 2452) I, [2024-04-03T19:59:26.614839 #1] INFO -- : [34ea335d-9627-4fc6-8b1f-957d6e2b46e9] Started GET "/fr/identity/password_reset?%F6%7Bconstructor.constructor(13250300)%7D" for 52.60.149.201 > I, [2024-04-03T17:06:41.466371 #1] INFO -- : [8eb439ee-af77-486c-93ee-6c4ee3584e0b] Completed 200 OK in 24ms (Views: 5.5ms | ActiveRecord: 13.5ms | Allocations: 4152) I, [2024-04-03T17:06:41.466201 #1] INFO -- : [8eb439ee-af77-486c-93ee-6c4ee3584e0b] Rendered layout layouts/public.html.erb (Duration: 14.7ms | Allocations: 3289) I, [2024-04-03T17:06:41.441846 #1] INFO -- : [8eb439ee-af77-486c-93ee-6c4ee3584e0b] Parameters: {"{constructor.constructor(13246203)}"=>nil, "locale"=>"fr"} I, [2024-04-03T17:06:41.441700 #1] INFO -- : [8eb439ee-af77-486c-93ee-6c4ee3584e0b] Processing by Identity::PasswordResetsController#new as HTML We need to be able to extract the id of requests with error status (`34ea335d-9627-4fc6-8b1f-957d6e2b46e9` in this excerpt) and the error message information which can be found in subsequent lines after the error is logged (on line starting with 'E,'). Different errors will generate different number of error message lines. The request id and message info will then be used in alert messaging. We've tried different regex patterns that work in test but always fail with `pattern match not found` till we saw this comment.
|
| Comment by Matthew Steeves [ 2024 Apr 06 ] |
|
Yup, that syntax [[:space:]] should work. 2 other ways are the (?s) inline modifier or the \s token. If you're ok with ingesting the entire log into the item, then problem solved! One of the above multi-line regex approaches combined with regex lookahead should be enough to match the lines. To capture only the lines of interest into the item, you'll likely need to "pre-process" your log with a script. The script will need to keep track of where it is in the file in between runs, and use awk or perl with the range operator to define the start and end regex's. Then use zabbix-sender to feed that into an item. }}{{The trigger then becomes simple - every time a new item value is received, create a problem. Alternatively, could get the data into Zabbix via }}{{vfs.file.contents[] or log[] item. I have a pre-processing script that you could use as a starting point, if you're interested I'll post.
|
| Comment by Gaetano Schuppe [ 2024 Apr 08 ] |
|
Thank you Matthew.Steeves for these options. We hadn't looked into pre-processing the logs yet so I'm interested in your script. |