kubernetes/kops · error

fetching %q: %v

Error message

fetching %q: %v

What it means

remote.Get failed while fetching the source image's manifest descriptor from its registry. ParseReference succeeded, so the reference is syntactically fine; the failure is at the network/registry layer: DNS, connection, TLS, authentication (401/403), or the manifest simply does not exist (404 NAME_UNKNOWN / MANIFEST_UNKNOWN). kOps wraps it as "fetching %q: %v" using authn.DefaultKeychain for credentials.

Source

Thrown at pkg/assets/assetcopy/copyimage.go:55

func (e *CopyImage) Run() error {
	source := e.SourceImage
	target := e.TargetImage

	sourceRef, err := name.ParseReference(source)
	if err != nil {
		return fmt.Errorf("parsing reference %q: %v", source, err)
	}

	targetRef, err := name.ParseReference(target)
	if err != nil {
		return fmt.Errorf("parsing reference for %q: %v", target, err)
	}

	options := []remote.Option{remote.WithAuthFromKeychain(authn.DefaultKeychain)}

	desc, err := remote.Get(sourceRef, options...)
	if err != nil {
		return fmt.Errorf("fetching %q: %v", source, err)
	}

	targetDesc, err := remote.Get(targetRef, options...)
	if err == nil && desc.Digest.String() == targetDesc.Digest.String() {
		klog.Infof("no need to copy image from %v to %v", sourceRef, targetRef)
		return nil
	}

	switch desc.MediaType {
	case types.OCIImageIndex, types.DockerManifestList:
		// Handle indexes separately.
		if err := copyIndex(desc, sourceRef, targetRef, options...); err != nil {
			return fmt.Errorf("failed to copy index: %v", err)
		}
	default:
		// Assume anything else is an image, since some registries don't set mediaTypes properly.
		if err := copyImage(desc, sourceRef, targetRef, options...); err != nil {
			return fmt.Errorf("failed to copy image: %v", err)

View on GitHub (pinned to 4c8573c808)

Solutions

1. Verify the image exists: `crane manifest <source>` or `docker manifest inspect <source>`; fix the tag/digest in the cluster spec if it 404s.
2. Ensure registry credentials are available to authn.DefaultKeychain (`docker login <registry>`) and the credential helper works.
3. Check network/DNS/proxy connectivity to the registry host from the machine running kops.
4. Retry later if the registry is rate-limiting (e.g. Docker Hub 429).

Example fix

// before (no credentials, private source)
# kops set cluster spec.assets... ; remote.Get -> 401
// after
$ docker login registry.k8s.io   # or your private registry
$ kops update cluster ...        # re-run the copy

if out, err := exec.Command("crane", "digest", sourceImage).CombinedOutput(); err != nil {
	return fmt.Errorf("source image %q unreachable before copy: %v: %s", sourceImage, err, out)
}

if err := e.Run(); err != nil {
	if strings.Contains(err.Error(), "fetching") {
		var terr *net.OpError
		if errors.As(err, &terr) {
			// network-level: check DNS/proxy/firewall, then retry with backoff
		}
		if strings.Contains(err.Error(), "401") || strings.Contains(err.Error(), "403") {
			// re-authenticate: docker login <registry>, refresh credential helper
		}
		if strings.Contains(err.Error(), "404") || strings.Contains(err.Error(), "not found") {
			// fix the image tag/digest in the cluster spec
		}
	}
	return err
}

When it happens

Trigger: CopyImage.Run calls remote.Get(sourceRef) and the registry returns an error: image/tag/digest not present in the source repo, no network route, registry requires auth and no keychain credentials are configured (docker config / credential helper), rate limiting, or TLS problems with a private registry.

Common situations: Typos in image name or tag in the kops cluster spec (image never existed at that tag); running kops from a machine without internet or registry access; unauthenticated pulls of a private image (missing `docker login` / invalid ~/.docker/config.json); image moved or deleted upstream (e.g. registry.k8s.io restructuring); corporate proxy blocking registry.k8s.io.

Related errors

• failed to copy image: %v
• failed to copy index: %v
• failed to SSH to %q (with user %q): %w
• unable to SSH to %q: %v
• failed to list vpcs: %v

AI-assisted analysis of kubernetes/kops@4c8573c808 (2026-09-05). Data as JSON: /api/errors/178064558f255fdb. Report an issue: GitHub.