-
Type:
Problem report
-
Resolution: Unresolved
-
Priority:
Minor
-
None
-
Affects Version/s: 7.0.31rc1, 7.4.15rc1, 8.0.0rc1 (master)
-
Component/s: Agent2 plugin (G)
-
None
-
Support backlog
hello,
Currently Docker monitoring is configured in a way that CPU related stat "TotalUsage" is set as uint64 with this calculation:
# https://git.zabbix.com/projects/ZBX/repos/zabbix/browse/src/go/plugins/docker/handlers/key_container_stats.go func (s *stats) setCPUPercentUsage() { // based on formula from docker api doc. delta := s.CPUStats.CPUUsage.TotalUsage - s.PreCPUStats.CPUUsage.TotalUsage systemDelta := s.CPUStats.SystemUsage - s.PreCPUStats.SystemUsage cpuNum := s.CPUStats.OnlineCPUs s.CPUStats.CPUUsage.PercentUsage = (float64(delta) / float64(systemDelta)) * float64(cpuNum) * 100.0
Same type of calculation in case of Docker CLI is done in this way:
func calculateCPUPercentUnix(previousCPU container.CPUStats, curCPUStats container.CPUStats) float64 {
var (
cpuPercent = 0.0
// calculate the change for the cpu usage of the container in between readings
cpuDelta = float64(curCPUStats.CPUUsage.TotalUsage) - float64(previousCPU.CPUUsage.TotalUsage)
// calculate the change for the entire system between readings
systemDelta = float64(curCPUStats.SystemUsage) - float64(previousCPU.SystemUsage)
onlineCPUs = float64(curCPUStats.OnlineCPUs)
)
if onlineCPUs == 0.0 {
onlineCPUs = float64(len(curCPUStats.CPUUsage.PercpuUsage))
}
if systemDelta > 0.0 && cpuDelta > 0.0 {
cpuPercent = (cpuDelta / systemDelta) * onlineCPUs * 100.0
}
return cpuPercent
}
In case of Docker CLI before any calculation is done, "TotalUsage" is at first transferred into float64, then at last step, delta is verified - This helps with situations where container reported abnormal utilisation of CPU due to for example restart of container.
In case of Zabbix, problem is that underflow can happen - Example:
Normally, the Docker CPU usage values are within a reasonable range. However, after container restarts, extremely high values can occur.
For example, one of the responses contained:
percent_usage = 956409283996.0422 online_cpus = 8
The corresponding CPU usage value in our Calculated item is therefore:
956409283996.0422 / 8 = 119551160499.5053%
Relevant raw data
For the affected sample:
current total_usage = 104859885 previous total_usage = 8740746497 current system_cpu_usage = 66207037850000000 previous system_cpu_usage = 66207022420000000 online_cpus = 8
Therefore:
delta = 104859885 - 8740746497 = -8635886612
The current total_usage is lower than the previous value because the container has been restarted and the CPU usage counter has started a new cycle.
TotalUsage is defined as uint64.
Therefore, when the current value is lower than the previous value, the subtraction is performed as an unsigned interger operation before the conversion to float64.
In the example above, instead of obtaining: -8635886612
the subtraction results in a uint64 underflow: 18446744065073665004
The subsequent calculation is therefore:
CPU% = (18446744065073665004 / 15430000000) * 8 * 100 = 956409283996.0422%
This exactly matches the percent_usage value observed in the raw docker.container_stats response.
It would be worth to consider change of Zabbix's calculation - example from docker cli code can be found here:
https://github.com/docker/cli/blob/master/cli/command/container/stats_helpers.go#L183