kubernetes/kops · error

unable to load secret immediately after creation %v: %v

Error message

unable to load secret immediately after creation %v: %v

What it means

After ReplaceSecret writes the secret, it immediately reads it back via loadSecret to confirm the write round-trips. If that verification read fails (the file is not there yet, or parses as invalid JSON), the error is wrapped as 'unable to load secret immediately after creation %v: %v'. It signals an eventual-consistency window, a concurrent overwrite, or a corrupt write.

Source

Thrown at upup/pkg/fi/secrets/vfs_secretstore.go:174

func (c *VFSSecretStore) ReplaceSecret(id string, secret *fi.Secret) (*fi.Secret, error) {
	ctx := context.TODO()

	p := c.buildSecretPath(id)

	acl, err := acls.GetACL(ctx, p, c.cluster)
	if err != nil {
		return nil, err
	}

	err = createSecret(ctx, secret, p, acl, true)
	if err != nil {
		return nil, fmt.Errorf("unable to write secret: %v", err)
	}

	// Confirm the secret exists
	s, err := c.loadSecret(ctx, p)
	if err != nil {
		return nil, fmt.Errorf("unable to load secret immediately after creation %v: %v", p, err)
	}
	return s, nil
}

// createSecret will create the Secret, overwriting an existing secret if replace is true
func createSecret(ctx context.Context, s *fi.Secret, p vfs.Path, acl vfs.ACL, replace bool) error {
	data, err := json.Marshal(s)
	if err != nil {
		return fmt.Errorf("error serializing secret: %v", err)
	}

	rs := bytes.NewReader(data)
	if replace {
		return p.WriteFile(ctx, rs, acl)
	}
	return p.CreateFile(ctx, rs, acl)
}

View on GitHub (pinned to 4c8573c808)

Solutions

1. Re-run the replace; verify with `kops get secret <id>`
2. Check the wrapped cause: if 'parsing secret', inspect the stored object for corruption and re-create it
3. Ensure no other process is concurrently modifying the secrets store during rotation
4. Use a storage backend with strong read-after-write consistency

s, err := store.ReplaceSecret(id, sec)
if err != nil && strings.Contains(err.Error(), "unable to load secret immediately after creation") {
    // read-back raced or store is inconsistent; retry with backoff
    time.Sleep(time.Second)
    s, err = store.ReplaceSecret(id, sec)
}

When it happens

Trigger: loadSecret on <secrets-basdir>/<id> fails right after a successful WriteFile: eventual consistency on the storage backend (rare on modern S3 but possible with custom VFS backends), another process deleting/overwriting the key in between, or the written bytes not parsing as fi.Secret JSON.

Common situations: Custom/legacy object stores with read-after-write inconsistency, aggressive caching layers in front of the state store, concurrent kops runs during secret rotation.

Related errors

• unable to write secret: %v
• error loading secret %q: %w
• cannot parse SecretStore %q: %w
• secrets path is not cluster readable: %v
• error writing secret to %q: %v

AI-assisted analysis of kubernetes/kops@4c8573c808 (2026-09-05). Data as JSON: /api/errors/58f81fcb0ceed391. Report an issue: GitHub.