image-2020-11-26-19-29-05-984.png
image-2020-11-27-14-02-39-865.png
[ZBX-18709] Zabbix proxy stores junk data in the database for the dependent items Created: 2020 Nov 26 Updated: 2020 Nov 27 |
|
| Status: | Open |
| Project: | ZABBIX BUGS AND ISSUES |
| Component/s: | Proxy (P) |
| Affects Version/s: | 5.0.5 |
| Fix Version/s: | None |
| Type: | Incident report | Priority: | Trivial |
| Reporter: | Vadym Kalsin | Assignee: | Eduards Matuls (Inactive) |
| Resolution: | Unresolved | Votes: | 0 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Environment: |
Debian 10 |
||
| Attachments: |
image-2020-11-26-19-29-05-984.png
image-2020-11-27-14-02-39-865.png
|
| Description |
|
I'm trying to monitor logs in Zabbix. We have a master item which receives data in the JSON format:
{"time":"2020-11-26T14:56:02+00:00","method":"POST","status": "200","code":"uk","request_time":"0.140"}
{"time":"2020-11-26T14:56:02+00:00","method":"POST","status": "200","code":"au","request_time":"0.060"}
...
Also we have a discovery rule which detects different codes and creates dependent items:
To extract data we use the following preprocessing rule: So the the above example for the for the first log line we have:
But instead of discarding items if not found, zabbix proxy stores empty values in its database. slow query: 50.739578 sec, "insert into proxy_history (itemid,clock,ns,value,flags,write_clock) values (47220,1606400543,3637904,'',2,1606400554),(47238,1606400543,3637904,'',2,1606400554),(47217,1606400543,3637904,'',2,1606400554),(47226,1606400543,3637904,'',2,1606400554),(47230,1606400543,3637904,'',2,1606400554),(47219,1606400543,3637904,'',2,1606400554),(47234,1606400543,3637904,'',2,1606400554),(47221,1606400543,3637904,'',2,1606400554),(47216,1606400543,3637904,'',2,1606400554),(47225,1606400543,3637904,'',2,1606400554),(47224,1606400543,3637904,'',2,1606400554),(47223,1606400543,3637904,'',2,1606400554),(47218,1606400543,3637904,'',2,1606400554),(47227,1606400543,3637904,'',2,1606400554),(47228,1606400543,3637904,'',2,1606400554),(47229,1606400543,3637904,'0.19900000000000001',0,1606400554),(47222,1606400543,3637904,'',2,1606400554),(47231,1606400543,3637904,'',2,1606400554),(47232,1606400543,3637904,'',2,1606400554),(47233,1606400543,3637904,'',2,1606400554),(47215,1606400543,3637904,'',2,1606400554),(47235,1606400543,3637904,'',2,1606400554),(47236,1606400543,3637904,'',2,1606400554),(47237,1606400543,3637904,'',2,1606400554),(47214,1606400543,3637904,'',2,1606400554); Instead of this I expected something like this:
"insert into proxy_history (itemid,clock,ns,value,flags,write_clock) values (47229,1606400543,3637904,'0.19900000000000001',0,1606400554);
So if I have 100 dependent items, then 99% of the data in the database is just garbage. Why is the data not discarded, but stored as blank values? |
| Comments |
| Comment by Vadym Kalsin [ 2020 Nov 27 ] |
|
I think this issue relates to Zabbix proxy sends timestamp to distinguish items from which there is really no data. But this is wrong logic. If I set - discard value - I want the proxy to report when was the last real value and not to store empty values. For the empty value there is another option: Also, this is a bad idea for a high loaded proxies. Here is example. I have 1 master item and 25 dependent on 2 hosts. Here is the log of the Zabbix proxy: 5391:20201127:115401.056 executing housekeeper 5391:20201127:115403.365 housekeeper [deleted 401964 records in 2.308757 sec, idle for 1 hour(s)] 5391:20201127:115501.075 forced execution of the housekeeper 5391:20201127:115501.075 executing housekeeper 5391:20201127:115502.991 housekeeper [deleted 391462 records in 1.916273 sec, idle for 1 hour(s)] 5391:20201127:115601.095 forced execution of the housekeeper I'm executing housekeeper every minute. If I want to track also error codes, it will be millions of junk records in the proxy_history table. |
