containerd/containerd · error

failed to fetch metrics for task: %w

Error message

failed to fetch metrics for task: %w

What it means

After finding the container, ContainerStats queries containerd's TaskService Metrics endpoint for that container's task. If the task-service call fails (task does not exist because the container never started or already exited, or the metrics service errored), the error is wrapped as 'failed to fetch metrics for task'.

Source

Thrown at internal/cri/server/container_stats.go:37

import (
	"context"
	"fmt"

	"github.com/containerd/containerd/api/services/tasks/v1"
	runtime "k8s.io/cri-api/pkg/apis/runtime/v1"
)

// ContainerStats returns stats of the container. If the container does not
// exist, the call returns an error.
func (c *criService) ContainerStats(ctx context.Context, in *runtime.ContainerStatsRequest) (*runtime.ContainerStatsResponse, error) {
	cntr, err := c.containerStore.Get(in.GetContainerId())
	if err != nil {
		return nil, fmt.Errorf("failed to find container: %w", err)
	}
	request := &tasks.MetricsRequest{Filters: []string{"id==" + cntr.ID}}
	resp, err := c.client.TaskService().Metrics(ctx, request)
	if err != nil {
		return nil, fmt.Errorf("failed to fetch metrics for task: %w", err)
	}
	if len(resp.Metrics) != 1 {
		return nil, fmt.Errorf("unexpected metrics response: %+v", resp.Metrics)
	}

	handler, err := c.getMetricsHandler(ctx, cntr.SandboxID)
	if err != nil {
		return nil, err
	}

	cs, err := handler(cntr.Metadata, resp.Metrics[0])
	if err != nil {
		return nil, fmt.Errorf("failed to decode container metrics: %w", err)
	}
	return &runtime.ContainerStatsResponse{Stats: cs.stats}, nil
}

View on GitHub (pinned to 4246446a2b)

Solutions

1. Check container state first: only request stats for containers in CONTAINER_RUNNING state
2. Handle the wrapped 'task does not exist' error as a terminal 'no stats available' condition
3. Retry transient gRPC errors with backoff, but not NotFound-class errors
4. Check containerd health/logs if Metrics fails across many containers

Example fix

// before: fetching stats for any container regardless of state
stats, err := runtimeService.ContainerStats(id)
// after: only fetch for running containers
status, err := runtimeService.ContainerStatus(id)
if err != nil { return err }
if status.Status.State() != runtime.ContainerState_CONTAINER_RUNNING {
    return nil // no task metrics for non-running containers
}
stats, err := runtimeService.ContainerStats(id)

status, err := runtimeService.ContainerStatus(id)
if err != nil { return err }
if status.Status.State() != runtime.ContainerState_CONTAINER_RUNNING {
    return nil // no live task; metrics unavailable by design
}

func hasLiveTask(status *runtime.ContainerStatusResponse) bool {
    return status.Status.State() == runtime.ContainerState_CONTAINER_RUNNING
}

stats, err := runtimeService.ContainerStats(id)
if err != nil {
    if isNotFound(err) || strings.Contains(err.Error(), "task does not exist") {
        return nil, nil // task gone: skip sample
    }
    if isTransient(err) { return retryWithBackoff() }
    return err
}

When it happens

Trigger: Calling ContainerStats for a container with no running task (CREATED but never started, already exited, or task removed by the runtime), or containerd's metrics RPC failing transiently.

Common situations: Polling stats for stopped containers after they exited; containerd under load timing out on Metrics; task killed by OOM so the task no longer exists; monitoring loops racing container shutdown.

Related errors

• unexpected metrics response: %+v
• failed to decode container metrics: %w
• failed to fetch metrics for tasks: %w
• failed to decode container metrics for %q: %w
• failed to cleanup container %s in task-service: %w

AI-assisted analysis of containerd/containerd@4246446a2b (2026-09-02). Data as JSON: /api/errors/0ec3bce490f2d5e9. Report an issue: GitHub.