containerd/containerd · error

failed to retrieve checkpoint index blob from content store:

Error message

failed to retrieve checkpoint index blob from content store: %w

What it means

After a successful Checkpoint, containerd returns an image whose index manifest blob must be read back from the local content store via content.ReadBlob. This error means the blob referenced by the checkpoint image's target descriptor could not be fetched from the content store. It usually indicates content-store corruption or concurrent garbage collection of the just-created blob.

Source

Thrown at internal/cri/server/container_checkpoint_linux.go:234

	if err != nil {
		return nil, fmt.Errorf("checkpointing container %q failed: %w", container.ID, err)
	}

	// the checkpoint image has been provided as an index with manifests representing the tar of criu data, the rw layer, and the config
	var (
		index        v1.Index
		rawIndex     []byte
		targetDesc   = img.Target()
		contentStore = img.ContentStore()
	)

	// Once all content from the checkpoint image has been saved, the
	// checkpoint image can be remove from the local image store.
	defer c.client.ImageService().Delete(ctx, img.Metadata().Name)

	rawIndex, err = content.ReadBlob(ctx, contentStore, targetDesc)
	if err != nil {
		return nil, fmt.Errorf("failed to retrieve checkpoint index blob from content store: %w", err)
	}
	if err = json.Unmarshal(rawIndex, &index); err != nil {
		return nil, fmt.Errorf("failed to unmarshall blob into checkpoint data OCI index: %w", err)
	}

	// This internal containerd file is used by checkpointctl for checkpoint archive
	// analysis. It lives in the container state dir, which can hold files from a
	// prior checkpoint operation, so it is read without following symlinks.
	if err := copyNoFollow(
		filepath.Join(c.getContainerRootDir(container.ID), crmetadata.StatusFile),
		filepath.Join(cpPath, crmetadata.StatusFile),
		0o600,
	); err != nil {
		return nil, err
	}

	// dump.log and stats-dump are written directly into cpPath by CRIU via its
	// work directory (see withCheckpointOpts above), so they are already present

View on GitHub (pinned to 4246446a2b)

Solutions

  1. Retry the checkpoint — GC races are transient
  2. Run containerd's content-store consistency checks / ctr content ls and remove orphaned blobs, or restart containerd
  3. Disable/relax content GC or protect checkpoint blobs during the operation window
  4. Check disk health and free space under /var/lib/containerd/io.containerd.content.v1.content
Defensive patterns

Strategy: retry

Validate before calling

// pre-check disk and store health on the node:
// df -h /var/lib/containerd && ctr content ls | head

Try / catch

_, err := s.CheckpointContainer(ctx, req)
if err != nil && strings.Contains(err.Error(), "checkpoint index blob from content store") {
    // transient GC/corruption: restart containerd, then retry once
}

Prevention

When it happens

Trigger: Content store GC ran between task.Checkpoint returning and ReadBlob; content store database corruption after an unclean containerd shutdown; disk I/O errors reading the content-store files.

Common situations: Nodes with aggressive GC settings; containerd restarted uncleanly (power loss) leaving dangling references; full or failing disks on /var/lib/containerd.

Related errors


AI-assisted analysis of containerd/containerd@4246446a2b (2026-09-02). Data as JSON: /api/errors/16581912d333bd8b. Report an issue: GitHub.