image-2022-11-16-18-14-43-996.png
[ZBX-21933] Items created by LLD can be executed before preprocessing rules are available Created: 2022 Nov 16 Updated: 2023 Jan 12 Resolved: 2023 Jan 12 |
|
| Status: | Closed |
| Project: | ZABBIX BUGS AND ISSUES |
| Component/s: | Server (S) |
| Affects Version/s: | 6.0.10 |
| Fix Version/s: | None |
| Type: | Incident report | Priority: | Trivial |
| Reporter: | Mathew | Assignee: | Edgar Akhmetshin |
| Resolution: | Cannot Reproduce | Votes: | 0 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Environment: |
Kubernetes |
||
| Attachments: |
image-2022-11-16-18-14-43-996.png
|
| Description |
|
We add and remove items heavily for jobs on our Kubernetes cluster. We are running a modified version of the Kubernetes monitoring template optimized for this.
One of the otimisations we are running is that we are using a regex preprocessing step to level down data
Item 1: All kuberentes stats in a line seperated format i.e
production/x4b-job1/{json data}
production/x4b-job2/{json data}
Item 2. Inside a LLD rule extract the json data for that namespace and job using regex Item 3. Inside the same LLD rule JSONPath or similar on this to get the required value We have found this to be a high performance solution for this sort of data
What we have also found is a bug. Sometimes we get an attempt to store data before the preprocessor does it's thing in the final (item 3) item. This should be impossible.
Result: ` 414:20221116:063300.678 item "kubernetes.newark.production.k8s.x4b.net:kube.job.status_failed[production/x4b-kq-serverupdate-1190750]" became not supported: Value of type "string" is not suitable for value type "Numeric (unsigned)". Value "production/x4b-kq-serverupdate-1191082/{"kube_job_annotations":1,"kube_job_labels":1,"kube_job_info":1,"kube_job_created":1668580323,"kube_job_spec_parallelism":1,"kube_job_spec_completions":1,"kube_job_status_succeeded":1,"kube_job_status_failed":0,"kube_job_status_active":0,"kube_job_complete":1,"kube_job_status_start_time":1668580323,"kube_job_status_completion_time":1668580371,"kube_job_owner":1} production/x4b-kq-serverupdate-1191083/{"kube_job_annotations":1,"kube_job_labels":1,"kube_job_info":1,"kube_job_created":1668580323,"kube_job_spec_parallelism":1,"kube_job_spec_completions":1,"kube_job_status_succeeded":0,"kube_job_status_failed":0,"kube_job_status_active":1,"kube_job_status_start_time":1668580323,"kube_job_owner":1} production/x4b-kq-serverupdate-1191084/{"kube_job_annotations":1,"kube_job_labels":1,"kube_job_info":1,"kube_job_created":1668580323,"kube_job_spec_parallelism":1,"kube_job_spec_completions":1,"kube_job_status_succeeded":1,"kube_job_status_failed":0,"kube_job_status_active":0,"kube_job_complete":1,"kube_job_status_start_time":1668580323,"kube_job_status_completion_time":1668580369,"kube_job_owner":1} production/x4b-kq-serverupdate-1191085/{"kube_job_annotations":1,"kube_job_labels":1,"kube_job_info":1,"kube_job_created":1668580323,"kube_job_spec_parallelism":1,"kube_job_spec_completions":1,"kube_job_status_succeeded":0,"kube_job_status_failed":0,"kube_job_status_active":1,"kube_job_status_start_time":1668580323,"kube_job_owner":1} production/x4b-kq-serverupdate-1191086/{"kube_job_annotations":1,"kube_job_labels":1,"kube_job_info":1,"kube_job_created":1668580323,"kube_job_spec_parallelism":1,"kube_job_spec_completions":1,"kube_job_status_succeeded":1,"kube_job_status_failed":0,"kube_job_status_active":0,"kube_job_complete":1,"kube_job_status_start_time":1668580323,"kube_job_status_completion_time":1668580373,"kube_job_owner":1}
Expected: An integer value and no error.
Any error that occurs should have been handled by the preprocessor fail handling which tells me the item is being executed before the preprocessor steps are created by LLD or at-least before Zabbix Server (S) sees that it is. |
| Comments |
| Comment by Edgar Akhmetshin [ 2022 Nov 18 ] |
|
Could you please provide template used and raw data example in txt file attached to the issue? |
| Comment by Mathew [ 2022 Nov 21 ] |
|
@edgar.akhmetshin I'm not sure thats possible. It's well known that Zabbix template import & export doesnt directly support Dependent Items like this (theres a ZBXNEXT on it that I've contributed a workaround for) - ZBXNEXT-1305
I've tested the example steps provided.
My running theory was it's caused by Zabbix Server picking up the item before the preprocessor steps are created due to a lack of transactional integrety protection when the LLD process creates the item. But that doesnt appear to be the case from a read of the code (so it's possibly something else). |
| Comment by Edgar Akhmetshin [ 2022 Nov 30 ] |
|
Mathew, Could you please provide debug log for LLD part during the issue? |
