What causes "fetching blob %s: %w" in docker/compose?

Re-push the artifact so all layers exist in the repository: docker compose push.

fetching blob %s: %w

Error message

fetching blob %s: %w

What it means

After a Fetcher is obtained, GetBlob calls fetcher.Fetch on the blob descriptor (a GET against the registry's blobs endpoint). A failure here is wrapped with the blob digest: typical causes are 404 (blob missing from the repository, e.g. after registry garbage collection), 401/403 (blob-specific auth), or the descriptor referencing a blob that was never pushed.

Source

Thrown at internal/oci/resolver.go:110

	content, err := io.ReadAll(fetch)
	if err != nil {
		return spec.Descriptor{}, nil, err
	}
	return descriptor, content, nil
}

// GetBlob retrieves the content of a blob descriptor (e.g. an artifact layer)
// from the repository ref belongs to. Unlike Get it doesn't Resolve the
// digest, as the registry manifests endpoint only serves actual manifests;
// blob content must be fetched directly from the blobs endpoint.
func GetBlob(ctx context.Context, resolver remotes.Resolver, ref reference.Named, descriptor spec.Descriptor) ([]byte, error) {
	fetcher, err := resolver.Fetcher(ctx, ref.String())
	if err != nil {
		return nil, fmt.Errorf("creating fetcher for %s: %w", ref, err)
	}
	fetch, err := fetcher.Fetch(ctx, descriptor)
	if err != nil {
		return nil, fmt.Errorf("fetching blob %s: %w", descriptor.Digest, err)
	}
	defer func() { _ = fetch.Close() }()
	// bound the read by the declared size so a rogue registry can't cause
	// unbounded allocation; the extra byte detects oversized responses.
	content, err := io.ReadAll(io.LimitReader(fetch, descriptor.Size+1))
	if err != nil {
		return nil, fmt.Errorf("reading blob %s: %w", descriptor.Digest, err)
	}
	if int64(len(content)) != descriptor.Size {
		return nil, fmt.Errorf("blob %s size mismatch: expected %d bytes, got %d", descriptor.Digest, descriptor.Size, len(content))
	}
	// GetBlob bypasses containerd's content store, so integrity must be
	// checked here before callers write the bytes to disk.
	if err := descriptor.Digest.Validate(); err != nil {
		return nil, fmt.Errorf("invalid digest %s: %w", descriptor.Digest, err)
	}
	if actual := descriptor.Digest.Algorithm().FromBytes(content); actual != descriptor.Digest {
		return nil, fmt.Errorf("blob digest mismatch: expected %s, got %s", descriptor.Digest, actual)

View on GitHub (pinned to ddc4b044b6)

Solutions

Re-push the artifact so all layers exist in the repository: docker compose push.
If the registry ran GC, restore/repair the artifact by pushing it again from the source.
Check auth scopes cover blob pulls (pull scope for the target repo).
Verify the digest manually: GET /v2/<repo>/blobs/<digest> with curl to confirm 200 vs 404.

Example fix

# verify blob presence
$ curl -H "Authorization: Bearer $TOKEN" \
    https://registry.example.com/v2/my/repo/blobs/sha256:<digest> -o /dev/null -w '%{http_code}\n'
# 404 => re-push the artifact

Defensive patterns

Strategy: retry

Validate before calling

// optional preflight blob existence check
req, _ := http.NewRequestWithContext(ctx, http.MethodHead, blobURL(ref, digest), nil)
if resp, err := http.DefaultClient.Do(req); err == nil && resp.StatusCode == http.StatusNotFound {
    return fmt.Errorf("blob %s missing; re-push the artifact", digest)
}

Try / catch

fetch, err := fetcher.Fetch(ctx, descriptor)
if err != nil {
    if isRetryable(err) { // network/5xx
        time.Sleep(backoff)
        fetch, err = fetcher.Fetch(ctx, descriptor)
    }
    if err != nil {
        return fmt.Errorf("fetching blob %s: %w", descriptor.Digest, err)
    }
}

Prevention

Re-push artifacts after registry garbage collection.
Ensure auth scopes cover blob access, not just manifests.
Verify blob presence with a HEAD request when diagnosing.

When it happens

Trigger: Fetching a layer blob whose digest is absent from the repository (deleted by GC, cross-repo mount not completed, partial push), or auth scopes that cover the manifest but not blob access.

Common situations: Pulling a compose OCI artifact after the registry garbage-collected unreferenced layers; registries with per-blob token scopes (authenticating for repo A, blob lives in repo B after cross-repo mount); interrupted pushes that uploaded the manifest without all blobs.

Related errors

AI-assisted analysis of docker/compose@ddc4b044b6 (2026-08-15). Data as JSON: /api/errors/5b2b28b7be1259da. Report an issue: GitHub.