[ZBXNEXT-4865] Prometheus format native support Created: 2018 Nov 19  Updated: 2019 Apr 23  Resolved: 2019 Apr 12

Status: Closed
Project: ZABBIX FEATURE REQUESTS
Component/s: Frontend (F), Server (S)
Affects Version/s: None
Fix Version/s: 4.2.0rc1, 4.2 (plan)

Type: Change Request Priority: Major
Reporter: Vitaly Zhuravlev Assignee: Andris Zeila
Resolution: Fixed Votes: 7
Labels: preprocessing, prometheus
Σ Remaining Estimate: Not Specified Remaining Estimate: Not Specified
Σ Time Spent: Not Specified Time Spent: Not Specified
Σ Original Estimate: Not Specified Original Estimate: Not Specified

Attachments: PDF File ACC Prometheus integration.pdf     PNG File Screenshot from 2019-04-03 15-04-02.png     PNG File Selection_207.png    
Issue Links:
Sub-task
depends on ZBXNEXT-4087 Allow preprocessing in LLD rules Closed
depends on ZBXNEXT-4163 Using LLD macros on item preprocessing Closed
depends on ZBXNEXT-78 Zabbix is unable to interpret floats ... Closed
part of ZBXNEXT-4635 Zabbix Integration with Kubernetes Closed
Sub-Tasks:
Key
Summary
Type
Status
Assignee
ZBXNEXT-4961 Frontend changes to Prometheus format... Specification change (Sub-task) Closed Ivo Kurzemnieks  
Team: Team A
Sprint: Sprint 48, Jan 2019, Sprint 49 (Feb 2019), Sprint 50 (Mar 2019), Sprint 51 (Apr 2019)
Story Points: 7

 Description   

Number of collectors/applications that support Prometheus line protocol is growing. It includes such things as:

Docker(experimental https://docs.docker.com/config/thirdparty/prometheus/)
Kubernetes <-- what we are interested in
GItlab
Ceph
Collectd
etcd
InfluxDB
InfluxDB Telegraf
and more.

There are also tons of collectors available from prometheus or its community here: https://prometheus.io/docs/instrumenting/exporters/

As exposition format is quite simple and there is an HTTP agent in zabbix (4.0+) it could be relatively straightforward to implement its support in Zabbix.



 Comments   
Comment by Glebs Ivanovskis [ 2018 Nov 20 ]

Such requests could be offloaded to community if ZBXNEXT-2651 was implemented.

Comment by Vitaly Zhuravlev [ 2018 Dec 28 ]

First version of acceptance criteria with suggested design is attached. ACC Prometheus integration.pdf

Comment by Glebs Ivanovskis [ 2018 Dec 28 ]

Apparently, it's not a MS Word document...

Comment by Andrea Biscuola [ 2018 Dec 28 ]

Yep. Can you provide a pdf instead?

Comment by Vitaly Zhuravlev [ 2018 Dec 28 ]

Weird. I can only open file from the PC, not from mobile phone. Re-uploaded as PDF.

Comment by Glebs Ivanovskis [ 2018 Dec 28 ]

Item prototype example uses {#KEY} as item key. LLD macros are supported in item key parameters, not in item key itself.

Comment by Dimitri Bellini [ 2018 Dec 31 ]

Very good work, I hope to test soon to understand if is easy to start to monitoring application like "OpenShift".
Another topic on Kubernetes cluster could be, how i could automatically create PODS on Zabbix using the Prometheus exporters data? Host Prototypes? Have you some ideas?

Comment by Dimitri Bellini [ 2019 Feb 13 ]

Hi Dev Team,
this new feature is still planned for Zabbix 4.2?
Thanks

Comment by Vitaly Zhuravlev [ 2019 Feb 14 ]

dimitri.bellini, yes

Comment by Dimitri Bellini [ 2019 Feb 14 ]

Very good, we need a customer case to setup
Do you think we could start to monitoring Openshift/Kubernetes with the actual feature sets?

Comment by Andris Zeila [ 2019 Mar 15 ]

Released in:

  • pre-4.2.0rc1 r91109
Comment by Ivo Kurzemnieks [ 2019 Mar 19 ]

API documentation updated:

Comment by Kirill Dudka [ 2019 Mar 22 ]

dimitri.bellini Did you manage to try to automatically create pods?

Comment by Dimitri Bellini [ 2019 Mar 22 ]

@Kirill Dudka at the moment i could not test it, i'm waiting the Zabbix Documentation

Comment by Mark Broyles [ 2019 Mar 28 ]

Is there interest in the Zabbix community in forming a working-group to create a comprehensive Kubernetes monitoring Template that has feature parity with Prometheus? The ability to auto-discover pods, containers, etc. Would be great to build a robust solution, share it with the community, and possibly maintain it as a group to support future versions.

I’m able to provide compute resources for running a development environment for us to perform testing and provide a space for collaboration.  If anyone is interested, please message me privately ([email protected]).

cc: vzhuravlev

 

Comment by Dimitri Bellini [ 2019 Apr 03 ]

Hi Zabbix Team,
i tried the new feature on Zabbix 4.2.0 but i'm very confusing, i have tested with a simple exporter (https://github.com/nlamirault/speedtest_exporter). From that exporter i could receive the below metrics:

# HELP go_gc_duration_seconds A summary of the GC invocation durations.
# TYPE go_gc_duration_seconds summary
go_gc_duration_seconds{quantile="0"} 5.233e-05
go_gc_duration_seconds{quantile="0.25"} 6.3163e-05
go_gc_duration_seconds{quantile="0.5"} 7.107000000000001e-05
go_gc_duration_seconds{quantile="0.75"} 7.271e-05
go_gc_duration_seconds{quantile="1"} 8.303000000000001e-05
go_gc_duration_seconds_sum 0.000624217
go_gc_duration_seconds_count 9
# HELP go_goroutines Number of goroutines that currently exist.
# TYPE go_goroutines gauge
go_goroutines 24
# HELP go_memstats_alloc_bytes Number of bytes allocated and still in use.
# TYPE go_memstats_alloc_bytes gauge
go_memstats_alloc_bytes 3.370152e+06
# HELP go_memstats_alloc_bytes_total Total number of bytes allocated, even if freed.
# TYPE go_memstats_alloc_bytes_total counter
go_memstats_alloc_bytes_total 3.2114744e+07
# HELP go_memstats_buck_hash_sys_bytes Number of bytes used by the profiling bucket hash table.
# TYPE go_memstats_buck_hash_sys_bytes gauge
go_memstats_buck_hash_sys_bytes 1.448797e+06
# HELP go_memstats_frees_total Total number of frees.
# TYPE go_memstats_frees_total counter
go_memstats_frees_total 334739

I have created an HTTP Check Item with a Prometheus step, below the detail:

  • Parameters: go_gc_duration_seconds_count

every time i will test it i sow an error that say "", check the screenshot.

What i'm wrong?

Thanks very much!

Comment by Andris Zeila [ 2019 Apr 03 ]

When sending mutliline input data frontend terminates lines with CR LF (\r\n) while promeheus protocol expects only LF (\n).

Created ZBX-15933. Note that only the testing is affected. Data obtained from prometheus agents should be parsed without errors, but of course it's hard to setup it without testing.

Comment by Dimitri Bellini [ 2019 Apr 03 ]

Ok we have found the trick
I think is very useful to find a "fix" for the testing feature.
Thank you!

Comment by Alexander Vladishev [ 2019 Apr 12 ]

Updated documentation:

Generated at Fri Mar 29 01:12:44 EET 2024 using Jira 9.12.4#9120004-sha1:625303b708afdb767e17cb2838290c41888e9ff0.