[ZBX-25325] Prometheus: Preprocessing failed - multiple comments found Created: 2024 Oct 01  Updated: 2024 Oct 25  Resolved: 2024 Oct 25

Status: Closed
Project: ZABBIX BUGS AND ISSUES
Component/s: Server (S)
Affects Version/s: 7.0.4
Fix Version/s: None

Type: Problem report Priority: Trivial
Reporter: Arina Lebedeva Assignee: Tomasz Grzechulski
Resolution: Cannot Reproduce Votes: 0
Labels: items
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: PNG File item_preprocessing.png     PNG File latest_data.png     PNG File zabbix_error.png     File zbx_export_template_prometheus (1).yaml    

 Description   

After upgrade from 6.4 to 7.0.4 there is a problem with pre-processing of data in Prometneus format.

I'm monitoring an Azure Kubernetes cluster and have an item that makes multiple API calls to get cAdvisor metrics from x nodes. The item's output contains metrics from x nodes in Prometheus format (see below):

 

# HELP cadvisor_version_info A metric with a constant '1' value labeled by kernel version, OS version, docker version, cadvisor version & cadvisor revision.
# TYPE cadvisor_version_info gauge
cadvisor_version_info{cadvisorRevision="",cadvisorVersion="",dockerVersion="",kernelVersion="5.15.0-1071-azure",osVersion="Ubuntu 22.04.4 LTS"} 1
# HELP container_blkio_device_usage_total Blkio Device bytes usage
# TYPE container_blkio_device_usage_total counter
container_blkio_device_usage_total{container="",device="",id="/",image="",major="11",minor="0",name="",namespace="",operation="Read",pod=""} 2.7058176e+07 1727772781281
container_blkio_device_usage_total{container="",device="",id="/",image="",major="11",minor="0",name="",namespace="",operation="Write",pod=""} 0 1727772781281
container_blkio_device_usage_total{container="",device="/dev/sda",id="/",image="",major="8",minor="0",name="",namespace="",operation="Read",pod=""} 1.713661952e+09 1727772781281

 

When I try to extract any metric using preprocessing (Prometheus pattern), I get an error:

Preprocessing failed for: # HELP cadvisor_version_info A metric with a constant '1' value labeled by kernel version, OS ver... 
1. Failed: cannot apply Prometheus pattern: data parsing error at row 3213 "# HELP cadvisor_version_info A metric with a const...": multiple HELP comments found for metric "cadvisor_version_info"

After upgrade I didn’t change any configuration.

 Comments   
Comment by Tomasz Grzechulski [ 2024 Oct 03 ]

Hello Arina,

Thank you for reporting this issue,

If that would be possible, could you please share with us logs of your Zabbix Server?

And to have better understanding - this item that you're using is your custom item, right? Or it's from some of Zabbix templates?

Regards,

Tomasz

Comment by Arina Lebedeva [ 2024 Oct 04 ]

Hello Tomasz,

unfortunately, I’m not able to share logs with you but I uploaded a template where you find all the necessary information to understand what exactly I'm doing.
Best regards,

Arina

Comment by Tomasz Grzechulski [ 2024 Oct 15 ]

Hello Arina,

Thank you for template,
I was able to reproduce this issue with use of AKS, and because of that i have one question,
Can you please share with us source of this template?
That would be valuable information,

Thank you,
Regards,
Tomasz

Comment by Arina Lebedeva [ 2024 Oct 16 ]

Hello Tomasz,

it's my own template.

Best regards,
Arina

 

Comment by Tomasz Grzechulski [ 2024 Oct 17 ]

Hello Arina,

Thank you for clarification,
I would suggest one workaround - if that would possible, could you please checkout this template?:
https://git.zabbix.com/projects/ZBX/repos/zabbix/browse/templates/app/kubernetes_http/kubernetes_kubelet_http?at=refs%2Fheads%2Frelease%2F7.0
During troubleshooting of template attached in this ticket, i've also used this template to just check how it does look when it's working correctly, and it's working correctly.
Inside AKS i've created test environment with pod "nginx-test" with nginx listening on public ipv4 inside it - with use of template linked above i was able to read network traffic metrics without issues.
Because of that, i would suggest using either that template, or different one from our builtin templates - then, you can edit that template, or built your own using those ones.
But, at first i would suggest using our builtin templates to test if metrics are correctly preprocessed - if there still would be a problem, notify us.link to every of our Kubernetes related templates for 7.0LTS: https://git.zabbix.com/projects/ZBX/repos/zabbix/browse/templates/app/kubernetes_http?at=release%2F7.0

Thank you,
Regards,
Tomasz

Comment by Arina Lebedeva [ 2024 Oct 25 ]

Hello Tomasz,

you may not have been able to reproduce the error because you were likely using only one node. When cAdvisor metrics are collected from multiple nodes, the output includes duplicate comments, which causes issues during preprocessing. If you remove all comments with JavaScript, the preprocessing completes without errors.

I use custom templates to collect metrics not included in Zabbix’s default templates. Additionally, I have created template for namespaces, as Zabbix doesn’t provide templates for namespaces itself.

This isn’t a critical issue for me since I can increase the script timeout, but this problem wasn’t present in previous versions. I just wanted to let you know about this new inconvenience in the latest version of Zabbix.

Best regards,
Arina

Generated at Tue Apr 08 05:47:23 EEST 2025 using Jira 9.12.4#9120004-sha1:625303b708afdb767e17cb2838290c41888e9ff0.