containerd/containerd · error
failed to retrieve checkpoint index blob from content store:
Error message
failed to retrieve checkpoint index blob from content store: %w
What it means
After a successful Checkpoint, containerd returns an image whose index manifest blob must be read back from the local content store via content.ReadBlob. This error means the blob referenced by the checkpoint image's target descriptor could not be fetched from the content store. It usually indicates content-store corruption or concurrent garbage collection of the just-created blob.
Source
Thrown at internal/cri/server/container_checkpoint_linux.go:234
if err != nil {
return nil, fmt.Errorf("checkpointing container %q failed: %w", container.ID, err)
}
// the checkpoint image has been provided as an index with manifests representing the tar of criu data, the rw layer, and the config
var (
index v1.Index
rawIndex []byte
targetDesc = img.Target()
contentStore = img.ContentStore()
)
// Once all content from the checkpoint image has been saved, the
// checkpoint image can be remove from the local image store.
defer c.client.ImageService().Delete(ctx, img.Metadata().Name)
rawIndex, err = content.ReadBlob(ctx, contentStore, targetDesc)
if err != nil {
return nil, fmt.Errorf("failed to retrieve checkpoint index blob from content store: %w", err)
}
if err = json.Unmarshal(rawIndex, &index); err != nil {
return nil, fmt.Errorf("failed to unmarshall blob into checkpoint data OCI index: %w", err)
}
// This internal containerd file is used by checkpointctl for checkpoint archive
// analysis. It lives in the container state dir, which can hold files from a
// prior checkpoint operation, so it is read without following symlinks.
if err := copyNoFollow(
filepath.Join(c.getContainerRootDir(container.ID), crmetadata.StatusFile),
filepath.Join(cpPath, crmetadata.StatusFile),
0o600,
); err != nil {
return nil, err
}
// dump.log and stats-dump are written directly into cpPath by CRIU via its
// work directory (see withCheckpointOpts above), so they are already presentView on GitHub (pinned to 4246446a2b)
Solutions
- Retry the checkpoint — GC races are transient
- Run containerd's content-store consistency checks / ctr content ls and remove orphaned blobs, or restart containerd
- Disable/relax content GC or protect checkpoint blobs during the operation window
- Check disk health and free space under /var/lib/containerd/io.containerd.content.v1.content
Defensive patterns
Strategy: retry
Validate before calling
// pre-check disk and store health on the node: // df -h /var/lib/containerd && ctr content ls | head
Try / catch
_, err := s.CheckpointContainer(ctx, req)
if err != nil && strings.Contains(err.Error(), "checkpoint index blob from content store") {
// transient GC/corruption: restart containerd, then retry once
} Prevention
- Avoid aggressive content GC settings
- Shut down containerd cleanly to prevent store corruption
- Monitor disk health on /var/lib/containerd
When it happens
Trigger: Content store GC ran between task.Checkpoint returning and ReadBlob; content store database corruption after an unclean containerd shutdown; disk I/O errors reading the content-store files.
Common situations: Nodes with aggressive GC settings; containerd restarted uncleanly (power loss) leaving dangling references; full or failing disks on /var/lib/containerd.
Related errors
- invalid runtime v2 checkpoint options format
- writer has been reset
- unable to discard to offset
- failed to get reader from content store: %w
- unable to create manifests file: %w
AI-assisted analysis of containerd/containerd@4246446a2b (2026-09-02).
Data as JSON: /api/errors/16581912d333bd8b.
Report an issue: GitHub.