We heavily use the delta storage item for counters for network traffic, mysql queries, etc. Works very well except when there are errors - which we usually report back as a small negative number, like -0.9901 to mean a script problem, etc.

The problem is when there is an error like this, the delta value is of course stored as a huge number, like 5 million SQL queries per second.

We are thinking of changing the server code to ignore negative values, but is there another way to do this ?

A better way to report errors in custom checks, but anything we return that is not real data will screw up the system hence why negative values work (and we can trigger on them), but they screw up the delta system.

Worse, you only store the delta, not the underlying value, making it even harder to actually get these errors or to know when they happen or even see the underlying data - a bit of a pain, but would be nice if you stored the actual values and then calculated the deltas dynamically if you could; this would solve a lot of problems.