Filter data return to the zabbix server based on a configurable criteria.

Reason: Dramatically Improve scalability

How:

Configure a dead band defined as a percentage of the last valid value returned to the server. The dead band is defined based on the granularity of data required. This would typically be implemented on a Zabbix proxy or Zabbix agent.
A local cache is established to remember the last valid value for each data point that must be returned by exception.

If the last value used is withing an established + or - X% of the last recorded value, do not publish an updated data point to Zabbix.

If the last value is outside the established band of + or - X% of the last recorded value, publish the data point to Zabbix and updated the local cache with the last value.

Note: The local cache update could update with the last value when an exception occurs, or it could use a little intelligence to wait for two values outside the range before updating the last value, or it could use a formula to update the cache value by cache value + new value /2 if the result is float or int. This would lead to smoother and meaningful graphs.

This is the concept of data updates via exceptions. Very useful for almost static time-series, booleans, composite KPIs, etc. Ex. Disk space, number of OSPF routes in table, interface up/down status, etc. This would solve a lot of of the scale problems that people run into, especially for state validation. None of the major open-source systems have implemented this yet.

This reduces the number of items per second treated by the server without sacrificing data quality and state evaluation.

Zabbix a;ready supports infrequent updates or updates via exception, but nothing has been implemented to actually filter the data at entry based on centrally managed criteria.

This will permit scaling older implementations without increasing the number of updates per second.

This would be a key differentiator versus all other open-source and commercial offerings. This feature is only seen in the industrial sector where scalability requirements surpass acual supervisory capabilities.

Thank you,

Example:

Memory utilization on a system.
Typical time-series of 1000 values.
150, 150,150,150, ..., 151, 150, 151, 150, 149, 151, 145, 122, 120, 121, 120, 120, 120, 121, ... , 121, 120
x x x x

Typical boolean time-series fo 1000 values.
1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 1, 0, 1, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, ..., 1, 1
x x x x x x x

By defining a 1% dead band, actual data returned to zabbix for the 1000 collected values would be:
150, 145, 122, 120,
1, 0, 1, 0, 1, 0, 1

This permits the sys-admin to view trigger events and actual values.

Actual data values can be kept much longer in the database as there is much less data actually stored, and if trends are required. They can be processed assuming the data value is constant between exceptions (value changes).

For treating data loss on acquisition, the following can be used:

Have Zabbix support the concept of VTQ: Value, Time-stamp, Quality
Have Zabbix support the concept of RRDtool Unknown value.

The System doing the data acquisition can figure out if the system actually return 0, 1, or nothing. If nothing is returned, then the quality value is bad. (There is a numeric scale, but essentially it it can be summed to Good, Bad and Uncertain due to problems with dependancies)
If the is no returned value, the system can choose to send the last valid value with a quality flag of Bad.
If there is no returned value, the system can choose to send U with a quality flag of Bad.

U = no value available or unrecognized value.

Having the Q data associated with the data and timestamp is useful when resolving dependancies or dealing with unknown values. It also deals with the fragmentation that occurs with merging data with treshold events or dependancy events. Right now in zabbix, all calculations need to take into account depdancies, with a Q value, they can be resolved once and then all other calculations can just look at the Q value.