[ZBX-25300] Container restart counter is adding up separate containers restart counters together when containers are in pods, that are the same name but in different namespaces. Created: 2024 Sep 26  Updated: 2024 Oct 03  Resolved: 2024 Oct 03

Status: Closed
Project: ZABBIX BUGS AND ISSUES
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Problem report Priority: Trivial
Reporter: Visa P Assignee: Kamil Florowski (Inactive)
Resolution: Fixed Votes: 0
Labels: kubernetes
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

AWS EKS
Using https://www.zabbix.com/integrations/kubernetes

Attachments: PNG File image-2024-09-26-14-38-50-748.png     PNG File zabbix_restart_item_configuration.png     PNG File zabbix_restart_item_preprocessing.png    

 Description   

Steps to reproduce:

  1. Have a cluster with multiple namespaces.
  2. Have pods in them with same names as in other namespaces

Result:

In reality, only one of the namespaces pods container had restarted. Other pods in question have "Restart Count:  0". This can be verified with kubectl in CLI.

There is another example, where we have 1 restart in one namespace and 1 restart in another one. Zabbix reports 2 restart count for all of those pods.
Expected:
To see the correct restart count in Zabbix. Each item should report only that specific namespaces restart count and not combine them.

 Comments   
Comment by Alexey Pustovalov [ 2024 Sep 27 ]

please share additional information:
1. Pod inspect with visible number of restarts
2. "Container restarts" item configuration form + preprocessing tab.

Comment by Visa P [ 2024 Sep 27 ]

k describe pod community-0 -n<ns1>

Name:             community-0
Namespace:        <ns1>
Priority:         0
Service Account:  default
Node:             .
Start Time:       Tue, 24 Sep 2024 13:06:40 +0300
Labels:           .
Annotations:      <none>
Status:           Running
IP:               .
IPs:
  IP:           .
Controlled By:  StatefulSet/community
Containers:
...
  community:
    Container ID:  containerd://e30318940b9e94f2de3f42662a9c062d3295e5a29c0e7c2b2e754d070acf9bf1
    Image:         .
    Image ID:      .
    Port:          <none>
    Host Port:     <none>
    Command:
      .
    State:          Running
      Started:      Thu, 26 Sep 2024 22:52:50 +0300
    Last State:     Terminated
      .
    Ready:          True
    Restart Count:  6
    Limits:
      .
    Requests:
      .
    Environment:
      .
    Mounts:
      .
Conditions:
  Type                        Status
  PodReadyToStartContainers   True
  Initialized                 True
  Ready                       True
  ContainersReady             True
  PodScheduled                True
Volumes:
  .

 

k describe pod community-0 -n<ns2>

Name:             community-0
Namespace:        <ns2>
Priority:         0
Service Account:  default
Node:             .
Start Time:       Thu, 26 Sep 2024 21:54:32 +0300
Labels:           .
Annotations:      <none>
Status:           Running
IP:               .
IPs:
  IP:           .
Controlled By:  StatefulSet/community
Containers:
...
  community:
    Container ID:  containerd://4cbbffbc9c2882438ea2c5f99e4f9d070f17f01fe90205006d779a3f4434ba9b
    Image:         .
    Image ID:      .
    Port:          <none>
    Host Port:     <none>
    Command:
      .
    State:          Running
      Started:      Thu, 26 Sep 2024 21:54:34 +0300
    Ready:          True
    Restart Count:  0
    Limits:
      .
    Requests:
      .
    Environment:
      .
    Mounts:
      .
Conditions:
  Type                        Status
  PodReadyToStartContainers   True
  Initialized                 True
  Ready                       True
  ContainersReady             True
  PodScheduled                True
Volumes:
  .

k describe pod community-0 -n<ns3>

Name:             community-0
Namespace:        <ns3>
Priority:         0
Service Account:  default
Node:             .
Start Time:       Thu, 26 Sep 2024 18:37:52 +0300
Labels:           .
Annotations:      <none>
Status:           Running
IP:               .
IPs:
  IP:           .
Controlled By:  StatefulSet/community
Containers:
...
  community:
    Container ID:  containerd://54a27851d56a081d11ab376dfbb1bb847033ffbc2aae1fec3c23d735fcb8e93f
    Image:         .
    Image ID:      .
    Port:          <none>
    Host Port:     <none>
    Command:
      .
    State:          Running
      Started:      Thu, 26 Sep 2024 18:37:53 +0300
    Ready:          True
    Restart Count:  0
    Limits:
      .
    Requests:
      .
    Environment:
      .
    Mounts:
      .
Conditions:
  Type                        Status
  PodReadyToStartContainers   True
  Initialized                 True
  Ready                       True
  ContainersReady             True
  PodScheduled                True
Volumes:
  .

k describe pod community-0 -n<ns4>

Name:             community-0
Namespace:        <ns4>
Priority:         0
Service Account:  default
Node:             .
Start Time:       Thu, 26 Sep 2024 18:35:20 +0300
Labels:           .
Annotations:      <none>
Status:           Running
IP:               .
IPs:
  IP:           .
Controlled By:  StatefulSet/community
Containers:
...
  community:
    Container ID:  containerd://554f74e521703701a4fed9ace507d598b501052069c906ca012fd749f5b611ad
    Image:         .
    Image ID:      .
    Port:          <none>
    Host Port:     <none>
    Command:
      .
    State:          Running
      Started:      Thu, 26 Sep 2024 18:35:22 +0300
    Ready:          True
    Restart Count:  0
    Limits:
      .
    Requests:
      .
    Environment:
      .
    Mounts:
      .
Conditions:
  Type                        Status
  PodReadyToStartContainers   True
  Initialized                 True
  Ready                       True
  ContainersReady             True
  PodScheduled                True
Volumes:
  .

Some information redacted. Pods are identical in configuration, they just are in different namespaces

Comment by Visa P [ 2024 Sep 27 ]

Attached screenshots of requested item configurations. 
Prometheus parameter field value is: 

kube_pod_container_status_restarts_total{pod="community-0"}
Comment by Kamil Florowski (Inactive) [ 2024 Oct 01 ]

Hi,

thank you for providing the above details. How your preprocessing parameter looks like (not visible on the screenshot)? Have you tried to use namespace parameter?
For example:

kube_pod_container_status_restarts_total{pod="community-0", namespace="<your-namespace>"}
Comment by Visa P [ 2024 Oct 02 ]

The parameter is set as:
kube_pod_container_status_restarts_total

{pod="community-0"}

I tested by adding namespace to it and it did the trick. Restarts were correctly reported for each pod.

A simple solution.

Does this affect all the Pod Discovery items though? I think it does as they all have just the pod name defined and not namespace.

Comment by Kamil Florowski (Inactive) [ 2024 Oct 03 ]

Does this affect all the Pod Discovery items though?

  • Yes

Closing the ticket once the issue got resolved.

Comment by Visa P [ 2024 Oct 03 ]

I hope the fix will be included in a future version as well.

Generated at Wed Jun 25 07:21:20 EEST 2025 using Jira 9.12.4#9120004-sha1:625303b708afdb767e17cb2838290c41888e9ff0.