production

This also applies for :

Veeam by HTTP Browse Zabbix / Zabbix - ZABBIX GIT

Hello!

Actually, if you don't store the master item data in the DB (history is set to 0, which is the default for master items used for raw data), the only limitation you have here is the size of history cache, which is 16 MB by default. So this 65535 characters limit only applies for data that you actually store, and it's not the reason of your problem in that case. Here's more info about preprocessing:
https://www.zabbix.com/documentation/6.2/en/manual/config/items/preprocessing/preprocessing_details

First of all, you could test the "Get state metrics" item manually and see if there is actually any data about existing persistent volume claims in the output. If there is not, but you have these claims set up, probably there's something with kube-state-metrics addon which doesn't give any data about them for some reason. Also could be some preprocessing issues.

Share us a file with the actual Prometheus data from that item.

Hello,

The test on "Get state metrics" item was already present in previous screenshot but I'll retrace my steps :

I have created a pvc called "www-broken-web-0" on one of my clusters.

Searching for that pattern in the result of the "Get state metrics" item is unfruitful (text extracted from the "Get value and test" from the "Get state metrics" into a file named extract_from_zabbix) :

No results.

The prometheus extract as I said previously is the result of :

kubectl port-forward svc/zabbix-agent-kube-state-metrics -n zabbix-agent 8080 

and extract of metrics into a file named extract_from_prometheus.

In the prometheus extract from state metrics I do see the metrics/values :

So we do see the data in prometheus but not in the "Get state metrics" item.
I previously had added some history to the "Get state metrics" in order to see if the results were changing overtime.

Further more when I go to check the latest data here is what I can see, so no values but weirdly enough the entries are there :

Maybe I haven't understood what you mean by "First of all, you could test the "Get state metrics" item manually and see if there is actually any data about existing persistent volume claims in the output."
I maybe also am not extracting the prometheus data the way zabbix gets them as I do not know how to execute the following script directly on my kubernetes cluster:

Am at a loss, maybe I'm missing something... 
So is the above correct and are you noticing the same issue ?

Hope this helps.

The screenshots were reduced visibly, sorry.

Here is the data for that pvc :

➜ grep broken /tmp/extract_from_prometheus
kube_persistentvolumeclaim_labels{namespace="zabbix-test",persistentvolumeclaim="www-broken-web-0"} 1
kube_persistentvolumeclaim_info{namespace="zabbix-test",persistentvolumeclaim="www-broken-web-0",storageclass="doesnotexist",volumename=""} 1
kube_persistentvolumeclaim_status_phase{namespace="zabbix-test",persistentvolumeclaim="www-broken-web-0",phase="Lost"} 0
kube_persistentvolumeclaim_status_phase{namespace="zabbix-test",persistentvolumeclaim="www-broken-web-0",phase="Bound"} 0
kube_persistentvolumeclaim_status_phase{namespace="zabbix-test",persistentvolumeclaim="www-broken-web-0",phase="Pending"} 1
kube_persistentvolumeclaim_resource_requests_storage_bytes{namespace="zabbix-test",persistentvolumeclaim="www-broken-web-0"} 1.073741824e+09
kube_persistentvolumeclaim_access_mode{namespace="zabbix-test",persistentvolumeclaim="www-broken-web-0",access_mode="ReadWriteOnce"} 1 

Maybe I haven't understood what you mean by "First of all, you could test the "Get state metrics" item manually and see if there is actually any data about existing persistent volume claims in the output." I maybe also am not extracting the prometheus data the way zabbix gets them as I do not know how to execute the following script directly on my kubernetes cluster:

Yes, you've done it right with "Get value and test" (that also works just fine for testing items collected by active proxies).

Searching for that pattern in the result of the "Get state metrics" item is unfruitful (text extracted from the "Get value and test"

Not sure if the frontend might limit the actual output, you can try to copy it directly from webpage source code: 
1. Right-click the output result of the test and select "Inspect".
2. Copy the inner HTML of the result (this should include all the gathered Prometheus data):

In my case on my testing cluster the overall Prometheus data length received by Zabbix was 209 425 characters which is the same as I would get from the actual Prom endpoint myself.

I previously had added some history to the "Get state metrics" in order to see if the results were changing overtime.

In that case you'll get truncated master item with length of 65535 characters, but dependent items will actually see the whole master item data without any truncation.

So, I've checked everything and the reason is the mistake in the label on metrics preprocessing:

kube_persistentvolumeclaim_status_phase{namespace="{#NAMESPACE}", name="{#NAME}", phase="Pending"} 

This should be:

kube_persistentvolumeclaim_status_phase{namespace="{#NAMESPACE}", persistentvolumeclaim="{#NAME}", phase="Pending"} 

Also metrics for the phase "Available" should be removed as there is no such status for PVC (kube_persistentvolumeclaim_status_phase) according to the kube-state-metric documentation:
kube-state-metrics/persistentvolumeclaim-metrics.md at main · kubernetes/kube-state-metrics · GitHub

Fixed in:

• 6.0.15rc1 e202cf6aa93
• 6.4.1rc1 0c9f4a849ed
• 7.0.0alpha1 c3aa953fa33

Imported the new template and eureka :

Thank you so much for your promptness and solution.