{"record":{"id":"0ec3bce490f2d5e9","repo":"containerd/containerd","slug":"failed-to-fetch-metrics-for-task-w","errorCode":null,"errorMessage":"failed to fetch metrics for task: %w","messagePattern":"failed to fetch metrics for task: %w","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"internal/cri/server/container_stats.go","lineNumber":37,"sourceCode":"import (\n\t\"context\"\n\t\"fmt\"\n\n\t\"github.com/containerd/containerd/api/services/tasks/v1\"\n\truntime \"k8s.io/cri-api/pkg/apis/runtime/v1\"\n)\n\n// ContainerStats returns stats of the container. If the container does not\n// exist, the call returns an error.\nfunc (c *criService) ContainerStats(ctx context.Context, in *runtime.ContainerStatsRequest) (*runtime.ContainerStatsResponse, error) {\n\tcntr, err := c.containerStore.Get(in.GetContainerId())\n\tif err != nil {\n\t\treturn nil, fmt.Errorf(\"failed to find container: %w\", err)\n\t}\n\trequest := &tasks.MetricsRequest{Filters: []string{\"id==\" + cntr.ID}}\n\tresp, err := c.client.TaskService().Metrics(ctx, request)\n\tif err != nil {\n\t\treturn nil, fmt.Errorf(\"failed to fetch metrics for task: %w\", err)\n\t}\n\tif len(resp.Metrics) != 1 {\n\t\treturn nil, fmt.Errorf(\"unexpected metrics response: %+v\", resp.Metrics)\n\t}\n\n\thandler, err := c.getMetricsHandler(ctx, cntr.SandboxID)\n\tif err != nil {\n\t\treturn nil, err\n\t}\n\n\tcs, err := handler(cntr.Metadata, resp.Metrics[0])\n\tif err != nil {\n\t\treturn nil, fmt.Errorf(\"failed to decode container metrics: %w\", err)\n\t}\n\treturn &runtime.ContainerStatsResponse{Stats: cs.stats}, nil\n}\n","sourceCodeStart":19,"sourceCodeEnd":54,"githubUrl":"https://github.com/containerd/containerd/blob/4246446a2bf7d03837b0244118d858799393bd80/internal/cri/server/container_stats.go#L19-L54","documentation":"After finding the container, ContainerStats queries containerd's TaskService Metrics endpoint for that container's task. If the task-service call fails (task does not exist because the container never started or already exited, or the metrics service errored), the error is wrapped as 'failed to fetch metrics for task'.","triggerScenarios":"Calling ContainerStats for a container with no running task (CREATED but never started, already exited, or task removed by the runtime), or containerd's metrics RPC failing transiently.","commonSituations":"Polling stats for stopped containers after they exited; containerd under load timing out on Metrics; task killed by OOM so the task no longer exists; monitoring loops racing container shutdown.","solutions":["Check container state first: only request stats for containers in CONTAINER_RUNNING state","Handle the wrapped 'task does not exist' error as a terminal 'no stats available' condition","Retry transient gRPC errors with backoff, but not NotFound-class errors","Check containerd health/logs if Metrics fails across many containers"],"exampleFix":"// before: fetching stats for any container regardless of state\nstats, err := runtimeService.ContainerStats(id)\n// after: only fetch for running containers\nstatus, err := runtimeService.ContainerStatus(id)\nif err != nil { return err }\nif status.Status.State() != runtime.ContainerState_CONTAINER_RUNNING {\n    return nil // no task metrics for non-running containers\n}\nstats, err := runtimeService.ContainerStats(id)","handlingStrategy":"validation","validationCode":"status, err := runtimeService.ContainerStatus(id)\nif err != nil { return err }\nif status.Status.State() != runtime.ContainerState_CONTAINER_RUNNING {\n    return nil // no live task; metrics unavailable by design\n}","typeGuard":"func hasLiveTask(status *runtime.ContainerStatusResponse) bool {\n    return status.Status.State() == runtime.ContainerState_CONTAINER_RUNNING\n}","tryCatchPattern":"stats, err := runtimeService.ContainerStats(id)\nif err != nil {\n    if isNotFound(err) || strings.Contains(err.Error(), \"task does not exist\") {\n        return nil, nil // task gone: skip sample\n    }\n    if isTransient(err) { return retryWithBackoff() }\n    return err\n}","preventionTips":["Only collect stats for RUNNING containers","Classify wrapped errors: NotFound = skip, transient = retry with backoff","Watch containerd health if Metrics RPC failures cluster across containers"],"tags":["containerd","metrics","task-service","cri"],"backgroundTag":"task-metrics-unavailable","analyzedSha":"4246446a2bf7d03837b0244118d858799393bd80","analyzedAt":"2026-09-02T00:14:43.053Z","contentChangedAt":null,"schemaVersion":2},"datasetVersion":"2026-09-09T06:17:21.866Z"}