{"record":{"id":"e7934c050996546e","repo":"k3s-io/k3s","slug":"found-multiple-bootstrap-keys-in-storage","errorCode":null,"errorMessage":"found multiple bootstrap keys in storage","messagePattern":"found multiple bootstrap keys in storage","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"critical","filePath":"pkg/cluster/storage.go","lineNumber":138,"sourceCode":"\t\t}\n\t\treturn err\n\t}\n\n\treturn nil\n}\n\n// bootstrapKeyData lists keys stored in the datastore with the prefix \"/bootstrap\", and\n// will return the first such key. It will return an error if not exactly one key is found.\nfunc bootstrapKeyData(ctx context.Context, storageClient store.ReadCloser) (*mvccpb.KeyValue, error) {\n\tbootstrapList, err := storageClient.List(ctx, \"/bootstrap\", 0)\n\tif err != nil {\n\t\treturn nil, err\n\t}\n\tif len(bootstrapList) == 0 {\n\t\treturn nil, errors.New(\"no bootstrap data found\")\n\t}\n\tif len(bootstrapList) > 1 {\n\t\treturn nil, errors.New(\"found multiple bootstrap keys in storage\")\n\t}\n\treturn &bootstrapList[0], nil\n}\n\n// storageBootstrap loads data from the datastore's bootstrap key into the\n// ControlRuntimeBootstrap struct. The storage key and encryption passphrase are both derived\n// from the join token. If no bootstrap key exists, indicating that data needs to be written\n// back to the datastore, this function will set c.saveBootstrap to true and create an empty\n// bootstrap key as a lock. This function will not return successfully until either the\n// bootstrap key has been locked, or data is read into the struct.\nfunc (c *Cluster) storageBootstrap(ctx context.Context) error {\n\tif c.config.KineTLS {\n\t\tbootstrapCtx, cancel := context.WithCancel(ctx)\n\t\tdefer func() {\n\t\t\ttime.Sleep(time.Second)\n\t\t\tcancel()\n\t\t}()\n","sourceCodeStart":120,"sourceCodeEnd":156,"githubUrl":"https://github.com/k3s-io/k3s/blob/6ba341e396edc16b8dcae978a7c5e3ac7ee5606e/pkg/cluster/storage.go#L120-L156","documentation":"bootstrapKeyData found more than one key under the '/bootstrap' prefix, but its contract allows exactly one. Multiple keys arise when bootstrap data was written under several token-derived key names (empty-string hash from legacy versions, old token format, and the current normalized token) and were never collapsed.","triggerScenarios":"Cluster history includes token changes or upgrades across k3s versions with different key-derivation formats; migrateTokens failed or was skipped on a path that still calls bootstrapKeyData; two servers each wrote bootstrap under their own token.","commonSituations":"Upgrading very old k3s to newer token normalization without completing migration; rotating the cluster token while peers held old tokens; manual key insertion.","solutions":["Upgrade all servers to a consistent k3s version so migrateTokens (which rewrites empty-string-key and old-token-key data onto the normalized token key) runs and removes duplicates.","Manually inspect etcdctl get /bootstrap/ --prefix, identify the key equal to ShortHash(currentNormalizedToken), back up and delete the others.","If identity is unclear, export the bootstrap from a healthy server (decrypt with the known token), delete all /bootstrap keys, and let the healthy server re-save."],"exampleFix":"# identify current key\nTOKEN='K10...::server:...'; NORM=$(echo -n \"$TOKEN\" | cut -d: -f1)\n# delete stale keys, keep /bootstrap/<shorthash-of-$NORM>\netcdctl del /bootstrap/<stale-key>","handlingStrategy":"validation","validationCode":"// Require exactly one bootstrap key before proceeding (mirror bootstrapKeyData):\nlist, err := storageClient.List(ctx, \"/bootstrap\", 0)\nswitch {\ncase len(list) == 0: return errors.New(\"no bootstrap data\")\ncase len(list) > 1:  return fmt.Errorf(\"multiple bootstrap keys (%d): run token migration or clean stale keys\", len(list))\n}","typeGuard":null,"tryCatchPattern":"if err := bootstrapKeyData(ctx, sc); err != nil {\n\tif strings.Contains(err.Error(), \"multiple bootstrap keys\") {\n\t\t// export+re-save bootstrap under current token, delete stale keys, retry\n\t}\n}","preventionTips":["Upgrade all servers together so token-normalization migration completes on every datastore.","Never rotate the cluster token casually; each rotation risks a second /bootstrap key.","Periodically audit etcdctl get /bootstrap/ --prefix --keys-only; exactly one key is the healthy invariant."],"tags":["k3s","bootstrap","etcd","token","migration"],"backgroundTag":null,"analyzedSha":"6ba341e396edc16b8dcae978a7c5e3ac7ee5606e","analyzedAt":"2026-08-15T16:27:54.286Z","schemaVersion":2},"datasetVersion":"2026-08-15T17:31:12.345Z"}