k3s-io/k3s · critical

failed to start %s leader controller

Error message

failed to start %s leader controller

What it means

k3s server (pkg/server/server.go, apiserverControllers): after core controllers start, k3s iterates config.LeaderControllers and starts each leader-elected controller; if any returns an error it panics with 'failed to start %s leader controller' where %s is the controller's Go function name (via util.GetFunctionName). This is a fail-fast wrapper: the true cause is always in the wrapped error from the named controller (etcd, coredns, metrics-server, etc.).

Source

Thrown at pkg/server/server.go:186

		}
	} else {
		for name, cb := range controlConfig.Runtime.LeaderElectedClusterControllerStarts {
			go leader.RunOrDie(ctx, "", name, sc.K8s, cb)
		}
	}

	return nil
}

// apiserverControllers starts the core controllers, as well as the leader-elected controllers
// that should only run on a control-plane node.
func apiserverControllers(ctx context.Context, sc *Context, config *Config) {
	if err := coreControllers(ctx, sc, config); err != nil {
		panic(err)
	}
	for _, controller := range config.LeaderControllers {
		if err := controller(ctx, sc); err != nil {
			panic(errors.WithMessagef(err, "failed to start %s leader controller", util.GetFunctionName(controller)))
		}
	}

	// Re-run informer factory startup after core and leader-elected controllers have started.
	// Additional caches may need to start for the newly added OnChange/OnRemove callbacks.
	if err := sc.Start(ctx); err != nil {
		panic(errors.WithMessage(err, "failed to start wranger controllers"))
	}
}

// runOrDie is similar to leader.RunOrDie, except that it runs the callback
// immediately, without performing leader election.
func runOrDie(ctx context.Context, name string, cb leader.Callback) {
	defer func() {
		if err := recover(); err != nil {
			logrus.WithField("stack", string(debug.Stack())).Fatalf("%s controller panic: %v", name, err)
		}
	}()

View on GitHub (pinned to 6ba341e396)

Solutions

1. Read the wrapped error and stack in the k3s log: the named function (e.g. 'etcd-...') plus the underlying message identifies the failing controller
2. Fix the named controller's dependency: for etcd see etcd/client connectivity, for coredns/metrics-server check their manifests and disabled-component flags
3. Validate cluster configuration (CIDRs, flags like --disable-*, --flannel-backend) on all control-plane nodes and restart k3s
4. If caused by a startup race, restarting k3s once dependencies are healthy usually clears it

Example fix

// before
for _, controller := range config.LeaderControllers {
    if err := controller(ctx, sc); err != nil {
        panic(errors.WithMessagef(err, "failed to start %s leader controller", util.GetFunctionName(controller)))
    }
}

// after (surface the failing controller as a returned error for cleaner crash reporting)
for _, controller := range config.LeaderControllers {
    if err := controller(ctx, sc); err != nil {
        err = errors.WithMessagef(err, "failed to start %s leader controller", util.GetFunctionName(controller))
        logrus.WithField("stack", string(debug.Stack())).Fatal(err)
    }
}

// mirror the runOrDie recover pattern from the same file when starting leader controllers
func startLeaderControllers(ctx context.Context, sc *server.Context, config *server.Config) (err error) {
    defer func() {
        if r := recover(); r != nil {
            err = fmt.Errorf("leader controller startup panic: %v", r)
        }
    }()
    for _, controller := range config.LeaderControllers {
        if e := controller(ctx, sc); e != nil {
            return errors.WithMessagef(e, "failed to start %s leader controller", util.GetFunctionName(controller))
        }
    }
    return nil
}

When it happens

Trigger: Any single leader-elected controller failing during startup: the '-etcd' controller failing to create its client, cloud-controller-manager or coredns controllers failing on bad config, storage/metrics controllers failing when their dependencies (CRDs, apiserver resources) are unavailable at start time.

Common situations: Control-plane nodes with partially applied custom configuration (bad cluster-cidr/service-cidr, disabled components whose controllers still start); nodes that gain leader election while a dependency (etcd, apiserver extension layer) is unhealthy; version mismatches after partial upgrades.

Related errors

• failed to start etcd client
• failed to start wranger controllers
• no bootstrap data is available to reconcile against
• server node name not set
• not authorized

AI-assisted analysis of k3s-io/k3s@6ba341e396 (2026-08-15). Data as JSON: /api/errors/d514e81dc183e5ee. Report an issue: GitHub.