-
Incident report
-
Resolution: Won't fix
-
Blocker
-
None
-
3.2.6
-
None
I did open an issue here in parallel :
https://github.com/monitoringartist/zabbix-docker-monitoring/issues/73
i thought this issue was resolved in previous zabbix versions. I remember having this problem 1-2years ago where it was possible that passive items dont works anymore, but active items still works, because agent thread is in processing state.
So maybe on the module side there is something to do, but maybe this kind of issue can be prevented earlier in the code.
The last responsive line before the receiving thread is stuck is always the same: "Docker's socket query"
At about the same times i get a "Cannot open metric file"
When all threads are exhausted, they are all in processing states and passive items dont works anymore.
This is agent version 3.2.6
juin 20 23:32:48 aaaa zabbix_agentd[6747]: __zbx_zbx_setproctitle() title:'listener #3 [processing request]'
juin 20 23:32:48 aaaa zabbix_agentd[6747]: Requested [docker.stats[/ffff,memory_stats,usage]]
juin 20 23:32:48 aaaa zabbix_agentd[6747]: In zbx_module_docker_stats()
juin 20 23:32:48 aaaa zabbix_agentd[6747]: In zbx_module_docker_socket_query()
juin 20 23:32:48 aaaa zabbix_agentd[6747]: Docker's socket query: GET /containers/ffff/stats HTTP/1.0
juin 20 23:32:18 aaaa zabbix_agentd[6746]: __zbx_zbx_setproctitle() title:'listener #2 [processing request]'
juin 20 23:32:18 aaaa zabbix_agentd[6746]: Requested [docker.stats[/ffff,memory_stats,usage]]
juin 20 23:32:18 aaaa zabbix_agentd[6746]: In zbx_module_docker_stats()
juin 20 23:32:18 aaaa zabbix_agentd[6746]: In zbx_module_docker_socket_query()
juin 20 23:32:18 aaaa zabbix_agentd[6746]: Docker's socket query: GET /containers/ffff/stats HTTP/1.0
juin 20 23:35:48 aaaa zabbix_agentd[8196]: __zbx_zbx_setproctitle() title:'listener #2 [processing request]'
juin 20 23:35:48 aaaa zabbix_agentd[8196]: Requested [docker.stats[/ffff,memory_stats,usage]]
juin 20 23:35:48 aaaa zabbix_agentd[8196]: In zbx_module_docker_stats()
juin 20 23:35:48 aaaa zabbix_agentd[8196]: In zbx_module_docker_socket_query()
juin 20 23:35:48 aaaa zabbix_agentd[8196]: Docker's socket query: GET /containers/ffff/stats HTTP/1.0
juin 20 23:37:03 aaaa zabbix_agentd[8195]: Metric source file: /sys/fs/cgroup/memory/docker/f49261aea5c7135b5e7ff046d2606836671c40f4dca31a28e106a8cbdd78c01e/memory.stat
juin 20 23:37:03 aaaa zabbix_agentd[8195]: Looking metric total_rss in memory.stat file
juin 20 23:37:03 aaaa zabbix_agentd[8195]: Id: f49261aea5c7135b5e7ff046d2606836671c40f4dca31a28e106a8cbdd78c01e; metric: total_rss; value: 1294745600
juin 20 23:37:03 aaaa zabbix_agentd[8195]: Sending back [9884680192]
juin 20 23:37:03 aaaa zabbix_agentd[8195]: __zbx_zbx_setproctitle() title:'listener #1 [waiting for connection]'
juin 20 23:37:03 aaaa zabbix_agentd[8195]: __zbx_zbx_setproctitle() title:'listener #1 [processing request]'
juin 20 23:37:03 aaaa zabbix_agentd[8195]: Requested [docker.stats[/ffff,memory_stats,usage]]
juin 20 23:37:03 aaaa zabbix_agentd[8195]: In zbx_module_docker_stats()
juin 20 23:37:03 aaaa zabbix_agentd[8195]: In zbx_module_docker_socket_query()
juin 20 23:37:03 aaaa zabbix_agentd[8195]: Docker's socket query: GET /containers/ffff/stats HTTP/1.0
juin 20 23:13:36 rrrr zabbix_agentd[451]: Cannot open metric file: '/sys/fs/cgroup/cpu,cpuacct/docker/6705a06097c4311144edfab00f32fb320a91232e141f0f40f3c1bcb78c85d108/cpuacct.stat'
juin 20 23:13:36 rrrr zabbix_agentd[450]: Cannot open metric file: '/sys/fs/cgroup/memory/docker/6705a06097c4311144edfab00f32fb320a91232e141f0f40f3c1bcb78c85d108/memory.stat'
juin 20 23:13:36 rrrr zabbix_agentd[451]: Cannot open metric file: '/sys/fs/cgroup/memory/docker/6705a06097c4311144edfab00f32fb320a91232e141f0f40f3c1bcb78c85d108/memory.stat'
juin 20 23:13:36 rrrr zabbix_agentd[451]: Cannot open metric file: '/sys/fs/cgroup/cpu,cpuacct/docker/6705a06097c4311144edfab00f32fb320a91232e141f0f40f3c1bcb78c85d108/cpuacct.stat'
juin 20 23:15:16 rrrr zabbix_agentd[450]: Cannot open metric file: '/sys/fs/cgroup/memory/docker//r-zabbix-agent-ct03-dca-1-ac96fcff/memory.stat'
juin 20 23:15:18 rrrr zabbix_agentd[450]: Cannot open metric file: '/sys/fs/cgroup/cpu,cpuacct/docker//r-zabbix-agent-ct03-dca-1-ac96fcff/cpuacct.stat'
juin 20 23:15:19 rrrr zabbix_agentd[450]: Cannot open metric file: '/sys/fs/cgroup/memory/docker//r-zabbix-agent-ct03-dca-1-ac96fcff/memory.stat'
juin 20 23:15:21 rrrr zabbix_agentd[450]: Cannot open metric file: '/sys/fs/cgroup/cpu,cpuacct/docker//r-zabbix-agent-ct03-dca-1-ac96fcff/cpuacct.stat'
juin 20 23:28:18 rrrr zabbix_agentd[3546]: Cannot open metric file: '/sys/fs/cgroup/cpu,cpuacct/docker/dbbbfac6b6e41ebe180fbf3baa2309de5114c9d85c287d1296dab78a289e3a16/cpuacct.stat'
juin 20 23:28:18 rrrr zabbix_agentd[3546]: Cannot open metric file: '/sys/fs/cgroup/cpu,cpuacct/docker/dbbbfac6b6e41ebe180fbf3baa2309de5114c9d85c287d1296dab78a289e3a16/cpuacct.stat'
juin 20 23:28:18 rrrr zabbix_agentd[3546]: Cannot open metric file: '/sys/fs/cgroup/memory/docker/dbbbfac6b6e41ebe180fbf3baa2309de5114c9d85c287d1296dab78a289e3a16/memory.stat'
juin 20 23:28:19 rrrr zabbix_agentd[3545]: Cannot open metric file: '/sys/fs/cgroup/memory/docker/dbbbfac6b6e41ebe180fbf3baa2309de5114c9d85c287d1296dab78a289e3a16/memory.stat'
juin 20 23:28:29 rrrr zabbix_agentd[3545]: Cannot open metric file: '/sys/fs/cgroup/memory/docker//r-zabbix-agent-ct03-dca-1-ac96fcff/memory.stat'
juin 20 23:28:30 rrrr zabbix_agentd[3545]: Cannot open metric file: '/sys/fs/cgroup/memory/docker//r-zabbix-agent-ct03-dca-1-ac96fcff/memory.stat'
juin 20 23:28:30 rrrr zabbix_agentd[3545]: Cannot open metric file: '/sys/fs/cgroup/cpu,cpuacct/docker//r-zabbix-agent-ct03-dca-1-ac96fcff/cpuacct.stat'
juin 20 23:28:30 rrrr zabbix_agentd[3545]: Cannot open metric file: '/sys/fs/cgroup/cpu,cpuacct/docker//r-zabbix-agent-ct03-dca-1-ac96fcff/cpuacct.stat'
juin 20 23:31:37 rrrr zabbix_agentd[6745]: Cannot open metric file: '/sys/fs/cgroup/memory/docker//r-zabbix-agent-ct03-dca-1-ac96fcff/memory.stat'
juin 20 23:31:37 rrrr zabbix_agentd[6746]: Cannot open metric file: '/sys/fs/cgroup/cpu,cpuacct/docker//r-zabbix-agent-ct03-dca-1-ac96fcff/cpuacct.stat', container doesn't run
juin 20 23:32:18 rrrr zabbix_agentd[6747]: Cannot open metric file: '/sys/fs/cgroup/cpu,cpuacct/docker/dbbbfac6b6e41ebe180fbf3baa2309de5114c9d85c287d1296dab78a289e3a16/cpuacct.stat'
juin 20 23:32:18 rrrr zabbix_agentd[6745]: Cannot open metric file: '/sys/fs/cgroup/memory/docker/dbbbfac6b6e41ebe180fbf3baa2309de5114c9d85c287d1296dab78a289e3a16/memory.stat'
juin 20 23:32:48 rrrr zabbix_agentd[6747]: Cannot open metric file: '/sys/fs/cgroup/cpu,cpuacct/docker/dbbbfac6b6e41ebe180fbf3baa2309de5114c9d85c287d1296dab78a289e3a16/cpuacct.stat', container doesn't run
juin 20 23:35:48 rrrr zabbix_agentd[8197]: Cannot open metric file: '/sys/fs/cgroup/memory/docker/dbbbfac6b6e41ebe180fbf3baa2309de5114c9d85c287d1296dab78a289e3a16/memory.stat'
juin 20 23:36:07 rrrr zabbix_agentd[8195]: Cannot open metric file: '/sys/fs/cgroup/cpu,cpuacct/docker//r-zabbix-agent-ct03-dca-1-ac96fcff/cpuacct.stat'
juin 20 23:36:07 rrrr zabbix_agentd[8197]: Cannot open metric file: '/sys/fs/cgroup/cpu,cpuacct/docker//r-zabbix-agent-ct03-dca-1-ac96fcff/cpuacct.stat', container doesn't run
juin 20 23:40:52 rrrr zabbix_agentd[11699]: Cannot open metric file: '/sys/fs/cgroup/memory/docker/dbbbfac6b6e41ebe180fbf3baa2309de5114c9d85c287d1296dab78a289e3a16/memory.stat'
juin 20 23:40:52 rrrr zabbix_agentd[11699]: Cannot open metric file: '/sys/fs/cgroup/cpu,cpuacct/docker//r-zabbix-agent-ct03-dca-1-ac96fcff/cpuacct.stat', container doesn't run
juin 20 23:40:52 rrrr zabbix_agentd[11698]: Cannot open metric file: '/sys/fs/cgroup/cpu,cpuacct/docker/dbbbfac6b6e41ebe180fbf3baa2309de5114c9d85c287d1296dab78a289e3a16/cpuacct.stat', container doesn't run
juin 20 23:41:23 rrrr zabbix_agentd[11698]: Cannot open metric file: '/sys/fs/cgroup/cpu,cpuacct/docker//r-zabbix-agent-ct03-dca-1-ac96fcff/cpuacct.stat', container doesn't run
juin 20 23:45:16 rrrr zabbix_agentd[14744]: Cannot open metric file: '/sys/fs/cgroup/memory/docker//r-zabbix-agent-ct03-dca-1-ac96fcff/memory.stat'
juin 20 23:45:17 rrrr zabbix_agentd[14744]: Cannot open metric file: '/sys/fs/cgroup/cpu,cpuacct/docker//r-zabbix-agent-ct03-dca-1-ac96fcff/cpuacct.stat'