k3s-io/k3s · critical

failed to start etcd client

Error message

failed to start etcd client

What it means

k3s etcd package (pkg/etcd/etcd.go): the leader-elected '-etcd' controller startup asserts that an etcd client exists; on a control-plane node where normal etcd startup did not create one, it calls startClient, which dials the local etcd with client TLS certificates. A panic with this message means the embedded etcd could not be reached or the client could not be constructed (TLS material, connectivity, or etcd not serving yet).

Source

Thrown at pkg/etcd/etcd.go:664

	})
}

// Register adds db info routes for the http request handler, and registers cluster controller callbacks
func (e *ETCD) Register(handler http.Handler) (http.Handler, error) {
	e.config.Runtime.ClusterControllerStarts["etcd-node-metadata"] = func(ctx context.Context) {
		registerMetadataHandlers(ctx, e)
	}

	// The apiserver endpoint controller needs to run on a node with a local apiserver,
	// in order to successfully seed etcd with the endpoint list. The member removal controller
	// also needs to run on a non-etcd node as to avoid disruption if running on the node that
	// is being removed from the cluster.
	if !e.config.DisableAPIServer {
		e.config.Runtime.LeaderElectedClusterControllerStarts[version.Program+"-etcd"] = func(ctx context.Context) {
			// ensure client is started, as etcd startup may not have handled this if this is a control-plane-only node
			if e.client == nil {
				if err := e.startClient(ctx); err != nil {
					panic(errors.WithMessage(err, "failed to start etcd client"))
				}
			}

			registerEndpointsHandlers(ctx, e)
			registerMemberHandlers(ctx, e)
			registerSnapshotHandlers(ctx, e)

			// Re-run informer factory startup after core and leader-elected controllers have started.
			// Additional caches may need to start for the newly added OnChange/OnRemove callbacks.
			if err := start.All(ctx, 5, e.config.Runtime.K3s, e.config.Runtime.Core); err != nil {
				panic(errors.WithMessage(err, "failed to start wrangler controllers"))
			}
		}
	}

	// Tombstone file checking is unnecessary if we're not running etcd.
	if !e.config.DisableETCD {
		tombstoneFile := filepath.Join(dbDir(e.config), "tombstone")

View on GitHub (pinned to 6ba341e396)

Solutions

1. Check embedded etcd health on the node: k3s etcd-snapshot... or curl the local endpoint and inspect k3s logs for etcd startup errors before this panic
2. Verify etcd client TLS files exist and match the CA in ${datadir}/server/tls (etcd/client-ca.crt, etcd/server-client.crt/key) and regenerate if a restore rotated material
3. If etcd is wedged after a failed restore, follow the tombstone/backup guidance: back up and remove ${datadir}/server/db or rejoin with --server
4. Ensure quorum: on multi-node control planes confirm a majority of etcd members are up; on single-node just free resources (disk, memory) and restart k3s

Example fix

// before
if e.client == nil {
    if err := e.startClient(ctx); err != nil {
        panic(errors.WithMessage(err, "failed to start etcd client"))
    }
}

// after (fail with context instead of panic when etcd is not yet reachable)
if e.client == nil {
    if err := e.startClient(ctx); err != nil {
        return fmt.Errorf("failed to start etcd client (is embedded etcd running on this node?): %w", err)
    }
}

// before the leader-elected etcd controller runs, confirm the local etcd answers with the client certs
exec.Command("etcdctl", "--endpoints=https://127.0.0.1:2379",
    "--cacert=${datadir}/server/tls/etcd/server-ca.crt",
    "--cert=${datadir}/server/tls/etcd/server-client.crt",
    "--key=${datadir}/server/tls/etcd/server-client.key",
    "endpoint", "health").Run()

// recover pattern for a panic thrown inside the leader-elected controller start func
func safeStart(ctx context.Context, name string, start func(context.Context)) {
    defer func() {
        if r := recover(); r != nil {
            logrus.WithField("stack", string(debug.Stack())).Fatalf("%s controller panic: %v", name, r)
        }
    }()
    start(ctx)
}

When it happens

Trigger: Leader election won on a control-plane node whose local etcd is down or still starting (no listener on 127.0.0.1:2379); missing/expired/mismatched etcd client certs in ${datadir}/server/tls (client-ca, etcd-client); etcd data dir corruption or permission problems; cluster restored from snapshot with stale certificates.

Common situations: Restarting k3s immediately after a crash where etcd is slow to gain quorum; certificate rotation or restore-from-backup leaving etcd client certs inconsistent; disk-full preventing etcd from serving; mixed k3s versions on control-plane nodes during upgrades.

Related errors

• failed to start wrangler controllers
• failed to start %s leader controller
• no bootstrap data found in datastore - check server token va
• etcd datastore disabled
• no bootstrap data found

AI-assisted analysis of k3s-io/k3s@6ba341e396 (2026-08-15). Data as JSON: /api/errors/84998891b72002a2. Report an issue: GitHub.