{"record":{"id":"6cc27f9b6a1910df","repo":"containerd/containerd","slug":"an-error-occurred-when-try-to-find-container-q-6cc27f","errorCode":null,"errorMessage":"an error occurred when try to find container %q: %w","messagePattern":"an error occurred when try to find container %q: %w","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"internal/cri/server/container_remove.go","lineNumber":41,"sourceCode":"\t\"time\"\n\n\tcontainerd \"github.com/containerd/containerd/v2/client\"\n\tcontainerstore \"github.com/containerd/containerd/v2/internal/cri/store/container\"\n\t\"github.com/containerd/containerd/v2/pkg/tracing\"\n\t\"github.com/containerd/errdefs\"\n\t\"github.com/containerd/log\"\n\truntime \"k8s.io/cri-api/pkg/apis/runtime/v1\"\n)\n\n// RemoveContainer removes the container.\nfunc (c *criService) RemoveContainer(ctx context.Context, r *runtime.RemoveContainerRequest) (_ *runtime.RemoveContainerResponse, retErr error) {\n\tspan := tracing.SpanFromContext(ctx)\n\tstart := time.Now()\n\tctrID := r.GetContainerId()\n\tcontainer, err := c.containerStore.Get(ctrID)\n\tif err != nil {\n\t\tif !errdefs.IsNotFound(err) {\n\t\t\treturn nil, fmt.Errorf(\"an error occurred when try to find container %q: %w\", ctrID, err)\n\t\t}\n\t\t// Do not return error if container metadata doesn't exist.\n\t\tlog.G(ctx).Tracef(\"RemoveContainer called for container %q that does not exist\", ctrID)\n\t\treturn &runtime.RemoveContainerResponse{}, nil\n\t}\n\n\tdefer c.nri.BlockPluginSync().Unblock()\n\n\tid := container.ID\n\tspan.SetAttributes(tracing.Attribute(\"container.id\", id))\n\ti, err := container.Container.Info(ctx)\n\tif err != nil {\n\t\tif !errdefs.IsNotFound(err) {\n\t\t\treturn nil, fmt.Errorf(\"get container info: %w\", err)\n\t\t}\n\t\t// Since containerd doesn't see the container and criservice's content store does,\n\t\t// we should try to recover from this state by removing entry for this container\n\t\t// from the container store as well and return successfully.","sourceCodeStart":23,"sourceCodeEnd":59,"githubUrl":"https://github.com/containerd/containerd/blob/4246446a2bf7d03837b0244118d858799393bd80/internal/cri/server/container_remove.go#L23-L59","documentation":"This error wraps any non-NotFound failure returned by the containerd metadata store (containerStore.Get) while looking up a container during RemoveContainer. It means the CRI plugin could not even consult its local metadata cache for the container ID — something failed below the 'not found' level, such as a broken boltDB metadata store or an internal decode error. It deliberately does NOT fire for unknown IDs; those return success silently.","triggerScenarios":"containerStore.Get returns an error that is not errdefs.IsNotFound — e.g. the containerd metadata (bolt) DB is corrupted, the container record fails to deserialize, or the underlying store returns an internal error while reading the container by ID.","commonSituations":"Disk corruption or truncation of containerd's metadata.db after a crash/power loss; containerd version downgrade leaving unreadable records; filesystem I/O errors (ENOSPC, EIO) on the state directory; concurrent restart of containerd while the CRI call is in flight.","solutions":["Inspect containerd logs for the wrapped root error to identify whether it is a boltDB/I/O failure","Verify filesystem health of /var/lib/containerd (mount rw, not full): check df, dmesg, and disk errors","Restart containerd (systemctl restart containerd / crictl info) to clear transient store issues","If metadata.db is corrupted, stop containerd, back up and remove/repair metadata.db, then restart (containers may need recreation)","Upgrade containerd to a recent release if deserialization errors follow a version change"],"exampleFix":null,"handlingStrategy":"try-catch","validationCode":"// Go: check container exists and log the ID before removing\nid := r.GetContainerId()\nif id == \"\" { return status.Error(codes.InvalidArgument, \"container id required\") }\nif _, err := crictlInspectContainer(id); err != nil && !isNotFound(err) {\n    log.Printf(\"container %s store lookup degraded: %v\", id, err)\n}","typeGuard":"func isRealStoreFailure(err error) bool {\n    return err != nil && !errdefs.IsNotFound(err)\n}","tryCatchPattern":"resp, err := client.RemoveContainer(ctx, &runtime.RemoveContainerRequest{ContainerId: id})\nif err != nil {\n    if st, ok := status.FromError(err); ok && st.Code() == codes.NotFound {\n        return nil // already gone, treat as success\n    }\n    return fmt.Errorf(\"remove container %s: metadata store read failed: %w\", id, err)\n}","preventionTips":["Monitor disk health and free space on /var/lib/containerd","Avoid unclean shutdowns; use UPS/journaling and fsync-safe storage for metadata.db","Keep containerd version current to avoid deserialization bugs","Alert on containerd restarts and metadata bolt errors in logs"],"tags":["containerd","cri","metadata-store","storage"],"backgroundTag":"containerd-metadata-store-read-failed","analyzedSha":"4246446a2bf7d03837b0244118d858799393bd80","analyzedAt":"2026-09-02T00:14:43.053Z","contentChangedAt":null,"schemaVersion":2},"datasetVersion":"2026-09-09T06:17:21.866Z"}