k3s-io/k3s · critical

failed to start wrangler controllers

Error message

failed to start wrangler controllers

What it means

k3s etcd package (pkg/etcd/etcd.go): after the leader-elected etcd controllers (endpoints, member, snapshot) register their OnChange/OnRemove callbacks, start.All re-runs the wrangler informer factory startup for the K3s and Core runtimes so new caches can begin syncing. This panic means one of those shared informer factories failed to start, almost always because the local apiserver was unreachable or returned errors during cache startup.

Source

Thrown at pkg/etcd/etcd.go:675

	// also needs to run on a non-etcd node as to avoid disruption if running on the node that
	// is being removed from the cluster.
	if !e.config.DisableAPIServer {
		e.config.Runtime.LeaderElectedClusterControllerStarts[version.Program+"-etcd"] = func(ctx context.Context) {
			// ensure client is started, as etcd startup may not have handled this if this is a control-plane-only node
			if e.client == nil {
				if err := e.startClient(ctx); err != nil {
					panic(errors.WithMessage(err, "failed to start etcd client"))
				}
			}

			registerEndpointsHandlers(ctx, e)
			registerMemberHandlers(ctx, e)
			registerSnapshotHandlers(ctx, e)

			// Re-run informer factory startup after core and leader-elected controllers have started.
			// Additional caches may need to start for the newly added OnChange/OnRemove callbacks.
			if err := start.All(ctx, 5, e.config.Runtime.K3s, e.config.Runtime.Core); err != nil {
				panic(errors.WithMessage(err, "failed to start wrangler controllers"))
			}
		}
	}

	// Tombstone file checking is unnecessary if we're not running etcd.
	if !e.config.DisableETCD {
		tombstoneFile := filepath.Join(dbDir(e.config), "tombstone")
		if _, err := os.Stat(tombstoneFile); err == nil {
			if e.config.JoinURL == "" {
				return nil, errors.New("tombstone file has been detected but --server is empty: backup and delete ${datadir}/server/db to create a new cluster, or set --server to rejoin the cluster")
			}
			logrus.Infof("tombstone file has been detected, removing ${datadir}/server/db to rejoin the cluster")
			if _, err := backupDirWithRetention(dbDir(e.config), maxBackupRetention); err != nil {
				return nil, err
			}
		}

		if err := e.setName(false); err != nil {

View on GitHub (pinned to 6ba341e396)

Solutions

1. Check apiserver health on the node (k3s kubectl get --raw /readyz) and the k3s log for apiserver startup errors preceding this panic
2. Free apiserver-side constraints: disk space, memory, and etcd health, since watch startup fails when the datastore behind the apiserver is slow or erroring
3. Restart k3s (systemctl restart k3s) once the apiserver and etcd report healthy so leader-elected controllers re-run their startup
4. If it recurs on upgrades, align k3s versions across control-plane nodes and check release notes for wrangler/informer changes

Example fix

// before
if err := start.All(ctx, 5, e.config.Runtime.K3s, e.config.Runtime.Core); err != nil {
    panic(errors.WithMessage(err, "failed to start wrangler controllers"))
}

// after (retry briefly to tolerate apiserver warmup instead of panicking)
if err := wait.PollUntilContextTimeout(ctx, 5*time.Second, 2*time.Minute, true,
    func(ctx context.Context) (bool, error) {
        return start.All(ctx, 5, e.config.Runtime.K3s, e.config.Runtime.Core) == nil, nil
    }); err != nil {
    panic(errors.WithMessage(err, "failed to start wrangler controllers"))
}

// operator-level pre-check: local apiserver must be ready before k3s (re)starts leader-elected controllers
if !apiReady("https://127.0.0.1:6443/readyz", serverCA, clientCert, clientKey, 2*time.Minute) {
    return fmt.Errorf("apiserver not ready; fix apiserver/etcd health before restarting k3s")
}

When it happens

Trigger: Leader-elected etcd controller start racing a local apiserver that is not yet listening or is crash-looping; apiserver rejecting informer watches (authn/authz misconfiguration, overloaded apiserver); control-plane node where etcd (the apiserver's backend) is unhealthy, making watch establishment fail.

Common situations: Control-plane restarts where leader election fires before the apiserver is fully ready; resource-starved nodes where the apiserver is too slow to serve initial watches; partially upgraded clusters; snapshot-restore bootstraps with slow datastore startup.

Related errors

• failed to start wranger controllers
• failed to unmarshal apiserver addresses from etcd: %v
• failed to start etcd client
• no bootstrap data found in datastore - check server token va
• etcd datastore disabled

AI-assisted analysis of k3s-io/k3s@6ba341e396 (2026-08-15). Data as JSON: /api/errors/6b2770126405702d. Report an issue: GitHub.