|
Causes |
|||
| caused by |
|
Fix value type in ceph template | Closed |
Team INT
[ZBX-20442] zabbix-agent2 ceph plugin ceph.df.details does not return proper percent_used per pool Created: 2022 Jan 15 Updated: 2024 Apr 10 |
|
| Status: | Reopened |
| Project: | ZABBIX BUGS AND ISSUES |
| Component/s: | Agent2 plugin (G) |
| Affects Version/s: | 5.4.9 |
| Fix Version/s: | 5.0.38rc1, 6.0.22rc1, 6.4.7rc1, 7.0.0alpha5, 7.0 (plan) |
| Type: | Problem report | Priority: | Trivial |
| Reporter: | Rudolf Kastl | Assignee: | Aleksandre Sebiskveradze |
| Resolution: | Unresolved | Votes: | 1 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Environment: |
Red Hat Enterprise Linux release 8.4 (Ootpa) |
||
| Issue Links: |
|
||||||||
| Team: | Team INT |
||||||||
| Sprint: | Sprint 103 (Aug 2023), Sprint 104 (Sep 2023) | ||||||||
| Story Points: | 1 | ||||||||
| Description |
|
Steps to reproduce:
ceph.df.details[localhost:8003,zabbix,{$CEPH.API.KEY}][s|{"pools":{"cinder_pool":{"percent_used":0,"objects":2700348,"bytes_used":33478099716998,"rd_ops":131097973937,"rd_bytes":2854167607888896,"wr_ops":110086177173,"wr_bytes":1994239061645312,"stored_raw":33453536463750,"max_avail":30776729010176},"glance_pool":{"percent_used":1,"objects":8084276,"bytes_used":203193337176677,"rd_ops":203502055639,"rd_bytes":15367397057112064,"wr_ops":150701460,"wr_bytes":195993139178496,"stored_raw":203192739422821,"max_avail":30776729010176},"nova_pool":{"percent_used":0,"objects":565,"bytes_used":6819573244,"rd_ops":801415728,"rd_bytes":1014221914112,"wr_ops":757354406,"wr_bytes":12494521876480,"stored_raw":6801068028,"max_avail":30776729010176}},"rd_ops":335401445304,"rd_bytes":18222578886915072,"wr_ops":110994233039,"wr_bytes":2202726722700288,"num_pools":3,"total_bytes":384020542128128,"total_avail_bytes":146852942237696,"total_used_bytes":236683848986624,"total_objects":10785189}]
zabbix_agent2 -t ceph.df.details["localhost:8003","zabbix","\{$CEPH.API.KEY}"] Result: In the ceph df details output "used_percent" is a floating point number like e.g. for nova_pool -> "percent_used":7.3860952397808433e-05 While the zabbix agent 2 returns with ceph.df.details for nova_pool -> "percent_used":0 The two other pools show a similar symptom. Expected: |
| Comments |
| Comment by Rudolf Kastl [ 2022 Jan 15 ] | ||
|
I have also cross checked by directly querying the restful plugin with curl as you can see the problem must be on the processing side of the zabbix-agent2 ceph module: curl -k -H "Accept: application/json" -H "Content-Type: application/json" -X POST -d '{"prefix": "df", "format":"json"}' "https://zabbix:<hidden>@localhost:8003/request?wait=1" { "failed": [], "finished": [ { "command": "df format=json", "outb": "{\"stats\":{\"total_bytes\":384020542128128,\"total_avail_bytes\":146700027449344,\"total_used_bytes\":236835196022784,\"total_used_raw_bytes\":237320514678784,\"total_used_raw_ratio\":0.61798912286758423,\"num_osds\":100,\"num_per_pool_osds\":100},\"stats_by_class\":{\"hdd\":{\"total_bytes\":149767795507200,\"total_avail_bytes\":55950437056512,\"total_used_bytes\":93652328841216,\"total_used_raw_bytes\":93817358450688,\"total_used_raw_ratio\":0.6264188289642334},\"ssd\":{\"total_bytes\":234252746620928,\"total_avail_bytes\":90749590392832,\"total_used_bytes\":143182867181568,\"total_used_raw_bytes\":143503156228096,\"total_used_raw_ratio\":0.61259967088699341}},\"pools\":[{\"name\":\"glance_pool\",\"id\":1,\"stats\":{\"stored\":67730913747419,\"objects\":8084276,\"kb_used\":198430993337,\"bytes_used\":203193337176539,\"percent_used\":0.68794339895248413,\"max_avail\":30723375366144}},{\"name\":\"cinder_pool\",\"id\":2,\"stats\":{\"stored\":11201722993542,\"objects\":2713222,\"kb_used\":32841670107,\"bytes_used\":33629870189446,\"percent_used\":0.26732808351516724,\"max_avail\":30723375366144}},{\"name\":\"nova_pool\",\"id\":3,\"stats\":{\"stored\":2299460348,\"objects\":605,\"kb_used\":6756464,\"bytes_used\":6918618620,\"percent_used\":7.5057934736832976e-05,\"max_avail\":30723375366144}}]}\n", "outs": "" } ], "has_failed": false, "id": "140141297045912", "is_finished": true, "is_waiting": false, "running": [], "state": "success", "waiting": [] | ||
| Comment by Rudolf Kastl [ 2023 Feb 07 ] | ||
|
Still an issue for zabbix-agent2-6.2.6. | ||
| Comment by Rudolfs Zvejs (Inactive) [ 2023 Sep 01 ] | ||
|
Fixed in
| ||
| Comment by Rudolf Kastl [ 2023 Oct 26 ] | ||
|
While the calculation is fixed, it seems that the value is passed on as "string" while the template expects "Numeric (unsigned)" See e.g.: Value of type "string" is not suitable for value type "Numeric (unsigned)". Value "0.80994921922683716" | ||
| Comment by Rudolfs Zvejs (Inactive) [ 2023 Oct 27 ] | ||
|
Can't reproduce using ceph nautilus and latest Zabbix 6.4. Output of ❯ zabbix_get -s 127.0.0.1 -p 10050 -k 'ceph.df.details[Docker]' | jq
{
"pools": {
"cephfs.velo.data": {
"percent_used": 0,
"objects": 0,
"bytes_used": 0,
"rd_ops": 0,
"rd_bytes": 0,
"wr_ops": 0,
"wr_bytes": 0,
"stored_raw": 0,
"max_avail": 0
},
"cephfs.velo.meta": {
"percent_used": 0,
"objects": 0,
"bytes_used": 0,
"rd_ops": 0,
"rd_bytes": 0,
"wr_ops": 0,
"wr_bytes": 0,
"stored_raw": 0,
"max_avail": 0
},
"yumii": {
"percent_used": 0,
"objects": 0,
"bytes_used": 0,
"rd_ops": 0,
"rd_bytes": 0,
"wr_ops": 0,
"wr_bytes": 0,
"stored_raw": 0,
"max_avail": 0
}
},
"rd_ops": 0,
"rd_bytes": 0,
"wr_ops": 0,
"wr_bytes": 0,
"num_pools": 3,
"total_bytes": 0,
"total_avail_bytes": 0,
"total_used_bytes": 0,
"total_objects": 0
}
All zeroes, cause my env has no data, but clearly numerical. If the issue persists, please provide output of zabbix_get ceph.df.details and ceph df detail | ||
| Comment by Rudolf Kastl [ 2023 Nov 01 ] | ||
ceph df detail --- RAW STORAGE --- CLASS SIZE AVAIL USED RAW USED %RAW USED ssd 594 TiB 176 TiB 418 TiB 418 TiB 70.38 TOTAL 594 TiB 176 TiB 418 TiB 418 TiB 70.38 --- POOLS --- POOL ID PGS STORED (DATA) (OMAP) OBJECTS USED (DATA) (OMAP) %USED MAX AVAIL QUOTA OBJECTS QUOTA BYTES DIRTY USED COMPR UNDER COMPR device_health_metrics 1 1 0 B 0 B 0 B 0 0 B 0 B 0 B 0 36 TiB N/A N/A N/A 0 B 0 B glance_pool 2 2048 49 TiB 49 TiB 4.1 MiB 6.44M 147 TiB 147 TiB 12 MiB 57.86 36 TiB N/A N/A N/A 0 B 0 B cinder_pool 6 4096 88 TiB 88 TiB 30 MiB 12.13M 262 TiB 262 TiB 90 MiB 70.99 36 TiB N/A N/A N/A 0 B 0 B nova_pool 7 2048 8.5 GiB 8.5 GiB 11 MiB 19.95k 26 GiB 26 GiB 32 MiB 0.02 36 TiB N/A N/A N/A 0 B 0 B
zabbix_agent2 -t ceph.df.details[https://localhost:8003,user,apikey] ceph.df.details[https://localhost:8003,user,apikey][s|{"pools":{"cinder_pool":{"percent_used":0.7062910795211792,"objects":12072146,"bytes_used":286908304067472,"rd_ops":209011865943,"rd_bytes":3382408422080512,"wr_ops":188463442101,"wr_bytes":6981834848569344,"stored_raw":288241160814592,"max_avail":39769975291904},"device_health_metrics":{"percent_used":0,"objects":0,"bytes_used":0,"rd_ops":0,"rd_bytes":0,"wr_ops":0,"wr_bytes":0,"stored_raw":0,"max_avail":39769975291904},"glance_pool":{"percent_used":0.5754496455192566,"objects":6440212,"bytes_used":161716661399144,"rd_ops":150978453049,"rd_bytes":36850933300302848,"wr_ops":226662362,"wr_bytes":22167521027072,"stored_raw":161716541325312,"max_avail":39769975291904},"nova_pool":{"percent_used":0.00023126068117562681,"objects":19976,"bytes_used":27598077993,"rd_ops":455202196,"rd_bytes":1133556152320,"wr_ops":51954256,"wr_bytes":363315348480,"stored_raw":27483385856,"max_avail":39769975291904}},"rd_ops":360445521188,"rd_bytes":40234475278535680,"wr_ops":188742058719,"wr_bytes":7004365684944896,"num_pools":4,"total_bytes":652702069555200,"total_avail_bytes":194681798520832,"total_used_bytes":458020271034368,"total_objects":18532334}] I can actually see that the percent used value appears to be a floating number. This gets sent as active check to a 6.4.7 proxy from this 6.4.7 agent. The proxy pushes it to the server with the latest 6.4 template (server was just upgraded to 6.4.8 today, but the initially when reopening the issue the server was on 6.4.7 too). On the server side i see an error for every of the ceph storages i have the same error message, of course with slightly different values: "Value of type "string" is not suitable for value type "Numeric (unsigned)". Value "0.70573735237121582"" | ||
| Comment by Rudolf Kastl [ 2023 Nov 01 ] | ||
|
Reopening with the information above. Sorry for the late reply. | ||
| Comment by Rudolf Kastl [ 2023 Nov 01 ] | ||
|
As an additional example i enabled storage history for "ceph df" on the server side. This is one of the outputs i get from that value on the server side:
|