{"record":{"id":"d514e81dc183e5ee","repo":"k3s-io/k3s","slug":"failed-to-start-s-leader-controller","errorCode":null,"errorMessage":"failed to start %s leader controller","messagePattern":"failed to start (.+?) leader controller","errorType":"panic","errorClass":null,"httpStatus":null,"severity":"critical","filePath":"pkg/server/server.go","lineNumber":186,"sourceCode":"\t\t}\n\t} else {\n\t\tfor name, cb := range controlConfig.Runtime.LeaderElectedClusterControllerStarts {\n\t\t\tgo leader.RunOrDie(ctx, \"\", name, sc.K8s, cb)\n\t\t}\n\t}\n\n\treturn nil\n}\n\n// apiserverControllers starts the core controllers, as well as the leader-elected controllers\n// that should only run on a control-plane node.\nfunc apiserverControllers(ctx context.Context, sc *Context, config *Config) {\n\tif err := coreControllers(ctx, sc, config); err != nil {\n\t\tpanic(err)\n\t}\n\tfor _, controller := range config.LeaderControllers {\n\t\tif err := controller(ctx, sc); err != nil {\n\t\t\tpanic(errors.WithMessagef(err, \"failed to start %s leader controller\", util.GetFunctionName(controller)))\n\t\t}\n\t}\n\n\t// Re-run informer factory startup after core and leader-elected controllers have started.\n\t// Additional caches may need to start for the newly added OnChange/OnRemove callbacks.\n\tif err := sc.Start(ctx); err != nil {\n\t\tpanic(errors.WithMessage(err, \"failed to start wranger controllers\"))\n\t}\n}\n\n// runOrDie is similar to leader.RunOrDie, except that it runs the callback\n// immediately, without performing leader election.\nfunc runOrDie(ctx context.Context, name string, cb leader.Callback) {\n\tdefer func() {\n\t\tif err := recover(); err != nil {\n\t\t\tlogrus.WithField(\"stack\", string(debug.Stack())).Fatalf(\"%s controller panic: %v\", name, err)\n\t\t}\n\t}()","sourceCodeStart":168,"sourceCodeEnd":204,"githubUrl":"https://github.com/k3s-io/k3s/blob/6ba341e396edc16b8dcae978a7c5e3ac7ee5606e/pkg/server/server.go#L168-L204","documentation":"k3s server (pkg/server/server.go, apiserverControllers): after core controllers start, k3s iterates config.LeaderControllers and starts each leader-elected controller; if any returns an error it panics with 'failed to start %s leader controller' where %s is the controller's Go function name (via util.GetFunctionName). This is a fail-fast wrapper: the true cause is always in the wrapped error from the named controller (etcd, coredns, metrics-server, etc.).","triggerScenarios":"Any single leader-elected controller failing during startup: the '-etcd' controller failing to create its client, cloud-controller-manager or coredns controllers failing on bad config, storage/metrics controllers failing when their dependencies (CRDs, apiserver resources) are unavailable at start time.","commonSituations":"Control-plane nodes with partially applied custom configuration (bad cluster-cidr/service-cidr, disabled components whose controllers still start); nodes that gain leader election while a dependency (etcd, apiserver extension layer) is unhealthy; version mismatches after partial upgrades.","solutions":["Read the wrapped error and stack in the k3s log: the named function (e.g. 'etcd-...') plus the underlying message identifies the failing controller","Fix the named controller's dependency: for etcd see etcd/client connectivity, for coredns/metrics-server check their manifests and disabled-component flags","Validate cluster configuration (CIDRs, flags like --disable-*, --flannel-backend) on all control-plane nodes and restart k3s","If caused by a startup race, restarting k3s once dependencies are healthy usually clears it"],"exampleFix":"// before\nfor _, controller := range config.LeaderControllers {\n    if err := controller(ctx, sc); err != nil {\n        panic(errors.WithMessagef(err, \"failed to start %s leader controller\", util.GetFunctionName(controller)))\n    }\n}\n\n// after (surface the failing controller as a returned error for cleaner crash reporting)\nfor _, controller := range config.LeaderControllers {\n    if err := controller(ctx, sc); err != nil {\n        err = errors.WithMessagef(err, \"failed to start %s leader controller\", util.GetFunctionName(controller))\n        logrus.WithField(\"stack\", string(debug.Stack())).Fatal(err)\n    }\n}","handlingStrategy":"try-catch","validationCode":null,"typeGuard":null,"tryCatchPattern":"// mirror the runOrDie recover pattern from the same file when starting leader controllers\nfunc startLeaderControllers(ctx context.Context, sc *server.Context, config *server.Config) (err error) {\n    defer func() {\n        if r := recover(); r != nil {\n            err = fmt.Errorf(\"leader controller startup panic: %v\", r)\n        }\n    }()\n    for _, controller := range config.LeaderControllers {\n        if e := controller(ctx, sc); e != nil {\n            return errors.WithMessagef(e, \"failed to start %s leader controller\", util.GetFunctionName(controller))\n        }\n    }\n    return nil\n}","preventionTips":["Always read the wrapped error and function name in the panic message first; it names the exact failing controller","Validate cluster-wide flags (CIDRs, --disable-*, backends) before restarting control-plane nodes","Ensure dependencies of each leader controller (etcd, CRDs, extension apiserver) are healthy before leader election fires","During upgrades, upgrade and verify one control-plane node at a time so a bad controller start does not repeat cluster-wide"],"tags":["controllers","leader-election","panic","startup","k3s"],"backgroundTag":null,"analyzedSha":"6ba341e396edc16b8dcae978a7c5e3ac7ee5606e","analyzedAt":"2026-08-15T16:27:54.286Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}