What causes "failed to copy index: %v" in kubernetes/kops?

Verify push credentials for the TARGET registry (`docker login `, check keychain/credential helper).

failed to copy index: %v

Error message

failed to copy index: %v

What it means

The source descriptor's media type was an OCI image index or Docker manifest list, so Run called copyIndex, which fetches the index via desc.ImageIndex() and pushes it with remote.WriteIndex; any failure there is wrapped as "failed to copy index: %v". Typical causes are errors reading child manifests from the source or errors writing the index (and its children) to the target registry.

Source

Thrown at pkg/assets/assetcopy/copyimage.go:68

	options := []remote.Option{remote.WithAuthFromKeychain(authn.DefaultKeychain)}

	desc, err := remote.Get(sourceRef, options...)
	if err != nil {
		return fmt.Errorf("fetching %q: %v", source, err)
	}

	targetDesc, err := remote.Get(targetRef, options...)
	if err == nil && desc.Digest.String() == targetDesc.Digest.String() {
		klog.Infof("no need to copy image from %v to %v", sourceRef, targetRef)
		return nil
	}

	switch desc.MediaType {
	case types.OCIImageIndex, types.DockerManifestList:
		// Handle indexes separately.
		if err := copyIndex(desc, sourceRef, targetRef, options...); err != nil {
			return fmt.Errorf("failed to copy index: %v", err)
		}
	default:
		// Assume anything else is an image, since some registries don't set mediaTypes properly.
		if err := copyImage(desc, sourceRef, targetRef, options...); err != nil {
			return fmt.Errorf("failed to copy image: %v", err)
		}
	}

	return nil
}

func copyImage(desc *remote.Descriptor, sourceRef name.Reference, targetRef name.Reference, options ...remote.Option) error {
	klog.Infof("copying image from %v to %v", sourceRef, targetRef)

	img, err := desc.Image()
	if err != nil {
		return err
	}

View on GitHub (pinned to 4c8573c808)

Solutions

Verify push credentials for the TARGET registry (`docker login <target-registry>`, check keychain/credential helper).
Retry the copy; remote.WriteIndex retries internally per-blob but transient network failures still surface — re-running resumes.
Check the target registry's media-type/size limits and that the repository is allowed to be created.
Inspect the inner error (%v) for the specific blob/manifest that failed and verify it exists on the source with `crane manifest`.

Example fix

// before
$ kops ... # failed to copy index: writing manifest: unauthorized: authentication required
// after
$ docker login myregistry.example.com
$ kops replace -f cluster.yaml && kops update cluster ... # re-run asset copy

Defensive patterns

Strategy: retry

Validate before calling

if out, err := exec.Command("crane", "validate", "--remote", sourceImage).CombinedOutput(); err != nil {
	return fmt.Errorf("multi-arch source %q not fully pullable: %v: %s", sourceImage, err, out)
}
if err := checkPushAccess(targetRegistry); err != nil {
	return err
}

Try / catch

err := e.Run()
for i := 0; i < 3 && err != nil; i++ {
	if !strings.Contains(err.Error(), "failed to copy index") {
		break
	}
	time.Sleep(time.Duration(1<<i) * time.Second)
	if err = e.Run(); err == nil {
		break
	}
}

Prevention

Confirm push credentials for the target registry before bulk multi-arch copies.
Pre-create the target repository if your registry requires it (Harbor/ECR policies).
Ensure stable network to the target registry; large index copies are long and interruption-prone.
Check target registry media-type and size limits for manifest lists beforehand.

When it happens

Trigger: The source image is a multi-arch manifest list (e.g. official kubernetes images); desc.ImageIndex() fails to fetch child manifests from the source, or remote.WriteIndex fails pushing to the target because of auth (401/403), missing repository (need to create it), quota, unsupported media types, or network interruption mid-push.

Common situations: Copying multi-arch images into a private registry whose credentials are missing (unauthorized push); target registry rejects a child manifest's media type or size; interrupted push leaves the index incomplete and subsequent runs fail; target registry requires the repository to exist first (e.g. some Harbor setups); disk/quota limits on the private registry.

Related errors

AI-assisted analysis of kubernetes/kops@4c8573c808 (2026-09-05). Data as JSON: /api/errors/ec81cd9a82517d8a. Report an issue: GitHub.