containerd/containerd · error

failed decoding metrics for sandbox %s: %w

Error message

failed decoding metrics for sandbox %s: %w

What it means

decodeSandboxCgroupMetrics failed to parse the metrics returned by the controller into the expected cgroupMetrics structure. The controller responded, but the payload was empty, malformed, or in an unexpected schema version.

Source

Thrown at internal/cri/server/sandbox_stats_linux.go:56

	ctx context.Context,
	sandbox sandboxstore.Sandbox) (*runtime.PodSandboxStats, error) {
	meta := sandbox.Metadata

	if sandbox.Status.Get().State != sandboxstore.StateReady {
		return nil, fmt.Errorf("failed to get pod sandbox stats since sandbox container %q is not in ready state: %w", meta.ID, errdefs.ErrUnavailable)
	}

	ctrl, err := c.sandboxService.SandboxController(sandbox.Sandboxer)
	if err != nil {
		return nil, fmt.Errorf("failed to get controller for sandbox %s: %w", sandbox.ID, err)
	}
	metric, err := ctrl.Metrics(ctx, sandbox.ID)
	if err != nil {
		return nil, fmt.Errorf("failed getting metrics for sandbox %s: %w", sandbox.ID, err)
	}
	stats, err := decodeSandboxCgroupMetrics(metric)
	if err != nil {
		return nil, fmt.Errorf("failed decoding metrics for sandbox %s: %w", sandbox.ID, err)
	}

	podSandboxStats := &runtime.PodSandboxStats{
		Linux: &runtime.LinuxPodSandboxStats{},
		Attributes: &runtime.PodSandboxAttributes{
			Id:          meta.ID,
			Metadata:    meta.Config.GetMetadata(),
			Labels:      meta.Config.GetLabels(),
			Annotations: meta.Config.GetAnnotations(),
		},
	}

	timestamp := time.Now()

	cpuStats, err := c.cpuContainerStats(*stats, timestamp)
	if err != nil {
		return nil, fmt.Errorf("failed to obtain cpu stats: %w", err)
	}

View on GitHub (pinned to 4246446a2b)

Solutions

1. Align containerd, shim, and runtime versions so the metrics schema matches what decodeSandboxCgroupMetrics expects
2. Log/dump the raw metric payload on failure to spot nil or unexpected fields
3. Check cgroup version (v1 vs v2) support on the host matches what the decoder anticipates
4. If the sandbox just started, retry once the cgroup is fully populated

Example fix

// before: decoding immediately after start
metric, _ := ctrl.Metrics(ctx, id)
stats, err := decodeSandboxCgroupMetrics(metric) // fails on fresh sandbox
// after: guard empty payload
if metric == nil || metric.Data == nil { return nil, errdefs.ErrUnavailable }
stats, err := decodeSandboxCgroupMetrics(metric)

// guard payload shape before decoding
if metric == nil || metric.GetData() == nil {
    return errdefs.ErrUnavailable // nothing to decode
}

func decodableMetric(m *types.Metric) bool {
    return m != nil && m.Data != nil && len(m.Data) > 0
}

stats, err := decodeSandboxCgroupMetrics(metric)
if err != nil {
    log.Printf("sandbox %s metrics undecodable (schema/version mismatch): %v", sb.ID, err)
    return nil // skip this sample rather than failing the pipeline
}

When it happens

Trigger: decodeSandboxCgroupMetrics receives a *types.Metric that cannot be unmarshaled/mapped into cgroupMetrics (nil fields, cgroup v1 vs v2 field mismatches, truncated metrics blob).

Common situations: Mismatched containerd/shim versions producing differing metrics schemas; sandbox metrics returned before the cgroup is populated (all zeros/nil); custom shims emitting nonstandard metrics.

Related errors

• failed to decode pod sandbox metrics %s: %w
• windows stats is empty
• failed to apply OCI options: %w
• failed to fetch metrics for task: %w
• unexpected metrics response: %+v

AI-assisted analysis of containerd/containerd@4246446a2b (2026-09-02). Data as JSON: /api/errors/dd943668802438ec. Report an issue: GitHub.