AWS ECS templates: CPU/memory/disk utilization items become permanently unsupported when Container Insights is disabled (NaN serialized as null in aws.ecs.get_metrics)

XMLWordPrintable

    • Type: Problem report
    • Resolution: Unresolved
    • Priority: Trivial
    • None
    • Affects Version/s: 7.0.28, 7.4.12, 8.0.0beta2, 8.0.0rc1 (master)
    • Component/s: Templates (T)
    • Environment:

      Steps to reproduce:

      1. Configure the official "AWS by HTTP" template with working credentials in an AWS account that has an ECS cluster with Container Insights disabled (this is the AWS default - Container Insights is opt-in and billed).
      2. Let "ECS clusters discovery" run. The LLD override links "AWS ECS Serverless Cluster by HTTP" (or the EC2 variant) to the discovered cluster host automatically.
      3. Wait for the master script item aws.ecs.get_metrics to run.
      4. Check the dependent items aws.ecs.cpu_utilization, aws.ecs.memory_utilization (and aws.ecs.disk.utilization on the Serverless variant).

      Result:
      The three dependent items become permanently unsupported with a misleading type-conversion error, reproduced on every poll:

      Value of type "string" is not suitable for value type "Numeric (float)". Value "null"
      

      All sibling dependent items of the same master (aws.ecs.task_count, aws.ecs.network.rx, ...) handle the same situation gracefully (no data, state OK), because they read the CloudWatch Values array via JSONPath ...Values.first().first() with a "discard value" error handler.

      Expected:
      When the utilization metrics are not available (Container Insights disabled, or no datapoints in the window), the computed keys should be omitted - or the items should otherwise end up with "no data", exactly like the sibling items - not permanently unsupported with a type-conversion error.

      Root cause:
      In getMetricsData() the script stores the raw CloudWatch Values array and then does arithmetic on it:

      obj[metrics[i]] = AWS.getField(id, 'Values');   // <-- array
      ...
      CPUUtilization = Math.abs(obj.CpuUtilized * 100 / obj.CpuReserved);      // array math
      MemoryUtilization = Math.abs(obj.MemoryUtilized * 100 / obj.MemoryReserved);
      DiskUtilization = Math.abs(obj.EphemeralStorageUtilized * 100 / obj.EphemeralStorageReserved);
      return ({ MetricData, MemoryUtilization, CPUUtilization, DiskUtilization });
      

      array * 100 / array only works through JS type coercion when each array has exactly one element. These metrics belong to the ECS/ContainerInsights namespace, so with Container Insights disabled GetMetricData returns Values: [] for all of them, and:

      • [] * 100 / [] -> 0 / 0 -> NaN
      • JSON.stringify(NaN) -> null
      • the dependent item's JSONPath $.CPUUtilization extracts the JSON null successfully (so the step's error handler never triggers), producing the string "null"
      • the final conversion to Numeric (float) fails -> item unsupported, forever.

      Note on the happy path: the request uses StartTime = now - 600s with Period = 600; CloudWatch anchors the bucket at StartTime and returns at most one datapoint, so the coercion works when Container Insights is enabled. The failure mode is specifically the empty-array case - which is the out-of-the-box state of any ECS cluster, and nothing in the template README requires Container Insights to be enabled.

      Additional issue in the same function ("AWS ECS Cluster by HTTP" variant only): DiskUtilization is referenced in the return statement but never computed (the variant computes only CPU and memory), so it is undefined and silently dropped by JSON.stringify - it only works by accident.

      Affected code (identical in release/7.0, release/7.2, release/7.4 and master):

      Suggested fix (validated in production on 7.0.25 against a live Fargate cluster; with one datapoint it is numerically identical to the current coercion behaviour, so nothing changes for working setups):

      // Values is an array (ScanBy=TimestampDescending -> [0] = most recent datapoint).
      // Only compute the ratio when both sides are present and reserved > 0; otherwise
      // OMIT the key so the dependent item's JSONPath falls into its existing
      // "discard value" error handler instead of receiving the string "null".
      var latest = function (a) { return (Array.isArray(a) && a.length) ? a[0] : null; };
      var ratio = function (u, r) {
          u = latest(u); r = latest(r);
          return (u === null || r === null || Number(r) == 0) ? null : Math.abs(u * 100 / r);
      };
      var out = {
          MetricData: MetricData,
          CPUUtilization: ratio(obj.CpuUtilized, obj.CpuReserved),
          MemoryUtilization: ratio(obj.MemoryUtilized, obj.MemoryReserved),
          DiskUtilization: ratio(obj.EphemeralStorageUtilized, obj.EphemeralStorageReserved)
      };
      Object.keys(out).forEach(function (k) { if (out[k] === null || out[k] === undefined) delete out[k]; });
      return (out);
      

      Operational caveat worth mentioning in the fix notes: items that already became unsupported before the fix will not self-recover, because a preprocessing chain that ends in "discard value" does not clear the unsupported state - only an accepted value does.

            Assignee:
            Zabbix Support Team
            Reporter:
            Rafael Souza da Silveira
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

              Created:
              Updated: