What causes "error writing secret %q for mirror: %v" in kubernetes/kops?

Test write access to the destination: `aws s3 cp` a dummy file into the prefix

error writing secret %q for mirror: %v

Error message

error writing secret %q for mirror: %v

What it means

The final step of mirroring each secret is createSecret(..., replace=true), which JSON-serializes the secret and calls p.WriteFile on the destination path. If the write fails (storage error, permission denied, ACL rejected, or — less commonly — serialization failure inside createSecret), MirrorTo wraps it as 'error writing secret %q for mirror'.

Source

Thrown at upup/pkg/fi/secrets/vfs_secretstore.go:83

			return fmt.Errorf("error reading secret %q for mirror: %v", name, err)
		}

		if secret == nil {
			return fmt.Errorf("unable to find secret %q for mirror", name)
		}

		p := BuildVfsSecretPath(basedir, name)

		acl, err := acls.GetACL(ctx, p, c.cluster)
		if err != nil {
			return fmt.Errorf("error building acl for secret %q for mirror: %v", name, err)
		}

		klog.Infof("mirroring secret %s -> %s", name, p)

		err = createSecret(ctx, secret, p, acl, true)
		if err != nil {
			return fmt.Errorf("error writing secret %q for mirror: %v", name, err)
		}
	}

	return nil
}

// DeleteSecret implements fi.SecretStore DeleteSecret
func (c *VFSSecretStore) DeleteSecret(name string) error {
	ctx := context.TODO()

	p := c.buildSecretPath(name)
	return p.Remove(ctx)
}

func (c *VFSSecretStore) ListSecrets() ([]string, error) {
	files, err := c.basedir.ReadDir()
	var ids []string
	if os.IsNotExist(err) {

View on GitHub (pinned to 4c8573c808)

Solutions

Test write access to the destination: `aws s3 cp` a dummy file into the prefix
Grant s3:PutObject (or GCS equivalent) on the destination bucket/prefix to your principal
Remove conflicting objects at the destination path
Check the wrapped %v cause — if it says 'serializing', the secret itself cannot be JSON-marshaled (rare; usually means corrupted in-memory state)

Example fix

// before: destination bucket policy denies writes
store.MirrorTo(ctx, target) // error writing secret "dockerconfig" for mirror: AccessDenied
// after: grant put permission first
// aws s3api put-bucket-policy --bucket dest-bucket --policy file://allow-put.json
err := store.MirrorTo(ctx, target)

Defensive patterns

Strategy: try-catch

Validate before calling

// Probe write access before mirroring
probe := secrets.BuildVfsSecretPath(target, ".write-probe")
if err := probe.WriteFile(ctx, strings.NewReader("probe"), nil); err != nil {
    return fmt.Errorf("no write access to destination: %v", err)
}
_ = probe.Remove(ctx)

Try / catch

if err := store.MirrorTo(ctx, target); err != nil {
    if strings.Contains(err.Error(), "error writing secret") {
        return fmt.Errorf("check PutObject on destination bucket: %v", err)
    }
    return err
}

Prevention

Grant write (PutObject/storage.objects.create) on the destination prefix
Test with a dummy object upload before bulk mirroring
Watch for bucket policies/object-lock that block overwrites
Check quotas on the destination storage

When it happens

Trigger: Writing <target-basdir>/<name> fails: no write access to destination bucket/prefix, object-lock/versioning conflicts, quota exceeded, or the destination is a directory-like key that cannot be overwritten.

Common situations: Mirroring secrets into a read-only or other account's S3 bucket, S3 bucket policy denying s3:PutObject, running out of GCS quota, or the destination path already holding an undeletable object.

Related errors

AI-assisted analysis of kubernetes/kops@4c8573c808 (2026-09-05). Data as JSON: /api/errors/e9b4fac1d1d5040e. Report an issue: GitHub.