{"record":{"id":"84998891b72002a2","repo":"k3s-io/k3s","slug":"failed-to-start-etcd-client","errorCode":null,"errorMessage":"failed to start etcd client","messagePattern":"failed to start etcd client","errorType":"panic","errorClass":null,"httpStatus":null,"severity":"critical","filePath":"pkg/etcd/etcd.go","lineNumber":664,"sourceCode":"\t})\n}\n\n// Register adds db info routes for the http request handler, and registers cluster controller callbacks\nfunc (e *ETCD) Register(handler http.Handler) (http.Handler, error) {\n\te.config.Runtime.ClusterControllerStarts[\"etcd-node-metadata\"] = func(ctx context.Context) {\n\t\tregisterMetadataHandlers(ctx, e)\n\t}\n\n\t// The apiserver endpoint controller needs to run on a node with a local apiserver,\n\t// in order to successfully seed etcd with the endpoint list. The member removal controller\n\t// also needs to run on a non-etcd node as to avoid disruption if running on the node that\n\t// is being removed from the cluster.\n\tif !e.config.DisableAPIServer {\n\t\te.config.Runtime.LeaderElectedClusterControllerStarts[version.Program+\"-etcd\"] = func(ctx context.Context) {\n\t\t\t// ensure client is started, as etcd startup may not have handled this if this is a control-plane-only node\n\t\t\tif e.client == nil {\n\t\t\t\tif err := e.startClient(ctx); err != nil {\n\t\t\t\t\tpanic(errors.WithMessage(err, \"failed to start etcd client\"))\n\t\t\t\t}\n\t\t\t}\n\n\t\t\tregisterEndpointsHandlers(ctx, e)\n\t\t\tregisterMemberHandlers(ctx, e)\n\t\t\tregisterSnapshotHandlers(ctx, e)\n\n\t\t\t// Re-run informer factory startup after core and leader-elected controllers have started.\n\t\t\t// Additional caches may need to start for the newly added OnChange/OnRemove callbacks.\n\t\t\tif err := start.All(ctx, 5, e.config.Runtime.K3s, e.config.Runtime.Core); err != nil {\n\t\t\t\tpanic(errors.WithMessage(err, \"failed to start wrangler controllers\"))\n\t\t\t}\n\t\t}\n\t}\n\n\t// Tombstone file checking is unnecessary if we're not running etcd.\n\tif !e.config.DisableETCD {\n\t\ttombstoneFile := filepath.Join(dbDir(e.config), \"tombstone\")","sourceCodeStart":646,"sourceCodeEnd":682,"githubUrl":"https://github.com/k3s-io/k3s/blob/6ba341e396edc16b8dcae978a7c5e3ac7ee5606e/pkg/etcd/etcd.go#L646-L682","documentation":"k3s etcd package (pkg/etcd/etcd.go): the leader-elected '-etcd' controller startup asserts that an etcd client exists; on a control-plane node where normal etcd startup did not create one, it calls startClient, which dials the local etcd with client TLS certificates. A panic with this message means the embedded etcd could not be reached or the client could not be constructed (TLS material, connectivity, or etcd not serving yet).","triggerScenarios":"Leader election won on a control-plane node whose local etcd is down or still starting (no listener on 127.0.0.1:2379); missing/expired/mismatched etcd client certs in ${datadir}/server/tls (client-ca, etcd-client); etcd data dir corruption or permission problems; cluster restored from snapshot with stale certificates.","commonSituations":"Restarting k3s immediately after a crash where etcd is slow to gain quorum; certificate rotation or restore-from-backup leaving etcd client certs inconsistent; disk-full preventing etcd from serving; mixed k3s versions on control-plane nodes during upgrades.","solutions":["Check embedded etcd health on the node: k3s etcd-snapshot... or curl the local endpoint and inspect k3s logs for etcd startup errors before this panic","Verify etcd client TLS files exist and match the CA in ${datadir}/server/tls (etcd/client-ca.crt, etcd/server-client.crt/key) and regenerate if a restore rotated material","If etcd is wedged after a failed restore, follow the tombstone/backup guidance: back up and remove ${datadir}/server/db or rejoin with --server","Ensure quorum: on multi-node control planes confirm a majority of etcd members are up; on single-node just free resources (disk, memory) and restart k3s"],"exampleFix":"// before\nif e.client == nil {\n    if err := e.startClient(ctx); err != nil {\n        panic(errors.WithMessage(err, \"failed to start etcd client\"))\n    }\n}\n\n// after (fail with context instead of panic when etcd is not yet reachable)\nif e.client == nil {\n    if err := e.startClient(ctx); err != nil {\n        return fmt.Errorf(\"failed to start etcd client (is embedded etcd running on this node?): %w\", err)\n    }\n}","handlingStrategy":"validation","validationCode":"// before the leader-elected etcd controller runs, confirm the local etcd answers with the client certs\nexec.Command(\"etcdctl\", \"--endpoints=https://127.0.0.1:2379\",\n    \"--cacert=${datadir}/server/tls/etcd/server-ca.crt\",\n    \"--cert=${datadir}/server/tls/etcd/server-client.crt\",\n    \"--key=${datadir}/server/tls/etcd/server-client.key\",\n    \"endpoint\", \"health\").Run()","typeGuard":null,"tryCatchPattern":"// recover pattern for a panic thrown inside the leader-elected controller start func\nfunc safeStart(ctx context.Context, name string, start func(context.Context)) {\n    defer func() {\n        if r := recover(); r != nil {\n            logrus.WithField(\"stack\", string(debug.Stack())).Fatalf(\"%s controller panic: %v\", name, r)\n        }\n    }()\n    start(ctx)\n}","preventionTips":["Verify embedded etcd health and the ${datadir}/server/tls/etcd client certificate set after every restore or cert rotation","Keep control-plane resources (disk, memory) within bounds so etcd can serve during controller startup","On multi-node control planes, ensure an etcd quorum exists before restarting nodes so startClient does not race a down datastore","Watch k3s logs for the tombstone-file guidance and follow it rather than forcing restarts on a removed member"],"tags":["etcd","tls","panic","control-plane","leader-election","k3s"],"backgroundTag":null,"analyzedSha":"6ba341e396edc16b8dcae978a7c5e3ac7ee5606e","analyzedAt":"2026-08-15T16:27:54.286Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}