k3s-io/k3s · critical

found multiple bootstrap keys in storage

Error message

found multiple bootstrap keys in storage

What it means

bootstrapKeyData found more than one key under the '/bootstrap' prefix, but its contract allows exactly one. Multiple keys arise when bootstrap data was written under several token-derived key names (empty-string hash from legacy versions, old token format, and the current normalized token) and were never collapsed.

Source

Thrown at pkg/cluster/storage.go:138

		}
		return err
	}

	return nil
}

// bootstrapKeyData lists keys stored in the datastore with the prefix "/bootstrap", and
// will return the first such key. It will return an error if not exactly one key is found.
func bootstrapKeyData(ctx context.Context, storageClient store.ReadCloser) (*mvccpb.KeyValue, error) {
	bootstrapList, err := storageClient.List(ctx, "/bootstrap", 0)
	if err != nil {
		return nil, err
	}
	if len(bootstrapList) == 0 {
		return nil, errors.New("no bootstrap data found")
	}
	if len(bootstrapList) > 1 {
		return nil, errors.New("found multiple bootstrap keys in storage")
	}
	return &bootstrapList[0], nil
}

// storageBootstrap loads data from the datastore's bootstrap key into the
// ControlRuntimeBootstrap struct. The storage key and encryption passphrase are both derived
// from the join token. If no bootstrap key exists, indicating that data needs to be written
// back to the datastore, this function will set c.saveBootstrap to true and create an empty
// bootstrap key as a lock. This function will not return successfully until either the
// bootstrap key has been locked, or data is read into the struct.
func (c *Cluster) storageBootstrap(ctx context.Context) error {
	if c.config.KineTLS {
		bootstrapCtx, cancel := context.WithCancel(ctx)
		defer func() {
			time.Sleep(time.Second)
			cancel()
		}()

View on GitHub (pinned to 6ba341e396)

Solutions

1. Upgrade all servers to a consistent k3s version so migrateTokens (which rewrites empty-string-key and old-token-key data onto the normalized token key) runs and removes duplicates.
2. Manually inspect etcdctl get /bootstrap/ --prefix, identify the key equal to ShortHash(currentNormalizedToken), back up and delete the others.
3. If identity is unclear, export the bootstrap from a healthy server (decrypt with the known token), delete all /bootstrap keys, and let the healthy server re-save.

Example fix

# identify current key
TOKEN='K10...::server:...'; NORM=$(echo -n "$TOKEN" | cut -d: -f1)
# delete stale keys, keep /bootstrap/<shorthash-of-$NORM>
etcdctl del /bootstrap/<stale-key>

// Require exactly one bootstrap key before proceeding (mirror bootstrapKeyData):
list, err := storageClient.List(ctx, "/bootstrap", 0)
switch {
case len(list) == 0: return errors.New("no bootstrap data")
case len(list) > 1:  return fmt.Errorf("multiple bootstrap keys (%d): run token migration or clean stale keys", len(list))
}

if err := bootstrapKeyData(ctx, sc); err != nil {
	if strings.Contains(err.Error(), "multiple bootstrap keys") {
		// export+re-save bootstrap under current token, delete stale keys, retry
	}
}

When it happens

Trigger: Cluster history includes token changes or upgrades across k3s versions with different key-derivation formats; migrateTokens failed or was skipped on a path that still calls bootstrapKeyData; two servers each wrote bootstrap under their own token.

Common situations: Upgrading very old k3s to newer token normalization without completing migration; rotating the cluster token while peers held old tokens; manual key insertion.

Related errors

• no bootstrap data found in datastore - check server token va
• no bootstrap data found
• bootstrap data already found and encrypted with different to
• etcd disabled
• failed to migrate content from sqlite to etcd: %w

AI-assisted analysis of k3s-io/k3s@6ba341e396 (2026-08-15). Data as JSON: /api/errors/e7934c050996546e. Report an issue: GitHub.