What causes "error reading secret %q for mirror: %v" in kubernetes/kops?

Inspect the wrapped cause: if it's 'parsing secret', the file is corrupt — re-create the secret with `kops replace secret` or delete it

kubernetes/kops · error

error reading secret %q for mirror: %v

Error message

error reading secret %q for mirror: %v

What it means

During MirrorTo, after listing secret names, each name is loaded with FindSecret(). If FindSecret returns a non-nil error (e.g. the file exists in the listing but its contents cannot be read, or the JSON is corrupt), the error is wrapped as 'error reading secret %q for mirror'. It indicates the store is inconsistent or unreadable mid-mirror.

Source

Thrown at upup/pkg/fi/secrets/vfs_secretstore.go:65

	return c
}

func (c *VFSSecretStore) MirrorTo(ctx context.Context, basedir vfs.Path) error {
	if basedir.Path() == c.basedir.Path() {
		klog.V(2).Infof("Skipping mirror of secret store from %q to %q (same path)", c.basedir, basedir)
		return nil
	}
	klog.V(2).Infof("Mirroring secret store from %q to %q", c.basedir, basedir)

	secrets, err := c.ListSecrets()
	if err != nil {
		return fmt.Errorf("error listing secrets for mirror: %v", err)
	}

	for _, name := range secrets {
		secret, err := c.FindSecret(name)
		if err != nil {
			return fmt.Errorf("error reading secret %q for mirror: %v", name, err)
		}

		if secret == nil {
			return fmt.Errorf("unable to find secret %q for mirror", name)
		}

		p := BuildVfsSecretPath(basedir, name)

		acl, err := acls.GetACL(ctx, p, c.cluster)
		if err != nil {
			return fmt.Errorf("error building acl for secret %q for mirror: %v", name, err)
		}

		klog.Infof("mirroring secret %s -> %s", name, p)

		err = createSecret(ctx, secret, p, acl, true)
		if err != nil {
			return fmt.Errorf("error writing secret %q for mirror: %v", name, err)

View on GitHub (pinned to 4c8573c808)

Solutions

Inspect the wrapped cause: if it's 'parsing secret', the file is corrupt — re-create the secret with `kops replace secret` or delete it
Check read permissions on the specific secret object in the state store
Re-run the mirror once network/concurrent-modification issues are resolved
Back up the secrets prefix before any manual repair

Defensive patterns

Strategy: try-catch

Validate before calling

// Validate each listed secret parses before mirroring
for _, id := range mustList(store) {
    if _, err := store.FindSecret(id); err != nil {
        return fmt.Errorf("secret %s unreadable: %v", id, err)
    }
}

Try / catch

if err := store.MirrorTo(ctx, target); err != nil {
    var name string
    if _, scanErr := fmt.Sscanf(err.Error(), "error reading secret %q for mirror", &name); scanErr == nil {
        return fmt.Errorf("repair or delete corrupt secret %q, then retry", name)
    }
    return err
}

Prevention

Avoid running multiple kops processes against one state store concurrently
Validate secret JSON after any manual edit of the state store
Back up the secrets prefix before pruning operations
Monitor state store integrity (e.g. s3 inventory checksums)

When it happens

Trigger: Listing returned a secret name but reading <basedir>/<name> fails: network/permission error on the object, or the object content is not valid JSON (loadSecret only swallows IsNotExist; corrupt files produce 'parsing secret from ...' which is wrapped here).

Common situations: Partially deleted or truncated secret files in the state store, concurrent modification during mirror (kops CLI run twice), corrupted uploads to the secrets prefix.

Related errors

AI-assisted analysis of kubernetes/kops@4c8573c808 (2026-09-05). Data as JSON: /api/errors/84b11f7a1f0fbec1. Report an issue: GitHub.