{"record":{"id":"6b2770126405702d","repo":"k3s-io/k3s","slug":"failed-to-start-wrangler-controllers","errorCode":null,"errorMessage":"failed to start wrangler controllers","messagePattern":"failed to start wrangler controllers","errorType":"panic","errorClass":null,"httpStatus":null,"severity":"critical","filePath":"pkg/etcd/etcd.go","lineNumber":675,"sourceCode":"\t// also needs to run on a non-etcd node as to avoid disruption if running on the node that\n\t// is being removed from the cluster.\n\tif !e.config.DisableAPIServer {\n\t\te.config.Runtime.LeaderElectedClusterControllerStarts[version.Program+\"-etcd\"] = func(ctx context.Context) {\n\t\t\t// ensure client is started, as etcd startup may not have handled this if this is a control-plane-only node\n\t\t\tif e.client == nil {\n\t\t\t\tif err := e.startClient(ctx); err != nil {\n\t\t\t\t\tpanic(errors.WithMessage(err, \"failed to start etcd client\"))\n\t\t\t\t}\n\t\t\t}\n\n\t\t\tregisterEndpointsHandlers(ctx, e)\n\t\t\tregisterMemberHandlers(ctx, e)\n\t\t\tregisterSnapshotHandlers(ctx, e)\n\n\t\t\t// Re-run informer factory startup after core and leader-elected controllers have started.\n\t\t\t// Additional caches may need to start for the newly added OnChange/OnRemove callbacks.\n\t\t\tif err := start.All(ctx, 5, e.config.Runtime.K3s, e.config.Runtime.Core); err != nil {\n\t\t\t\tpanic(errors.WithMessage(err, \"failed to start wrangler controllers\"))\n\t\t\t}\n\t\t}\n\t}\n\n\t// Tombstone file checking is unnecessary if we're not running etcd.\n\tif !e.config.DisableETCD {\n\t\ttombstoneFile := filepath.Join(dbDir(e.config), \"tombstone\")\n\t\tif _, err := os.Stat(tombstoneFile); err == nil {\n\t\t\tif e.config.JoinURL == \"\" {\n\t\t\t\treturn nil, errors.New(\"tombstone file has been detected but --server is empty: backup and delete ${datadir}/server/db to create a new cluster, or set --server to rejoin the cluster\")\n\t\t\t}\n\t\t\tlogrus.Infof(\"tombstone file has been detected, removing ${datadir}/server/db to rejoin the cluster\")\n\t\t\tif _, err := backupDirWithRetention(dbDir(e.config), maxBackupRetention); err != nil {\n\t\t\t\treturn nil, err\n\t\t\t}\n\t\t}\n\n\t\tif err := e.setName(false); err != nil {","sourceCodeStart":657,"sourceCodeEnd":693,"githubUrl":"https://github.com/k3s-io/k3s/blob/6ba341e396edc16b8dcae978a7c5e3ac7ee5606e/pkg/etcd/etcd.go#L657-L693","documentation":"k3s etcd package (pkg/etcd/etcd.go): after the leader-elected etcd controllers (endpoints, member, snapshot) register their OnChange/OnRemove callbacks, start.All re-runs the wrangler informer factory startup for the K3s and Core runtimes so new caches can begin syncing. This panic means one of those shared informer factories failed to start, almost always because the local apiserver was unreachable or returned errors during cache startup.","triggerScenarios":"Leader-elected etcd controller start racing a local apiserver that is not yet listening or is crash-looping; apiserver rejecting informer watches (authn/authz misconfiguration, overloaded apiserver); control-plane node where etcd (the apiserver's backend) is unhealthy, making watch establishment fail.","commonSituations":"Control-plane restarts where leader election fires before the apiserver is fully ready; resource-starved nodes where the apiserver is too slow to serve initial watches; partially upgraded clusters; snapshot-restore bootstraps with slow datastore startup.","solutions":["Check apiserver health on the node (k3s kubectl get --raw /readyz) and the k3s log for apiserver startup errors preceding this panic","Free apiserver-side constraints: disk space, memory, and etcd health, since watch startup fails when the datastore behind the apiserver is slow or erroring","Restart k3s (systemctl restart k3s) once the apiserver and etcd report healthy so leader-elected controllers re-run their startup","If it recurs on upgrades, align k3s versions across control-plane nodes and check release notes for wrangler/informer changes"],"exampleFix":"// before\nif err := start.All(ctx, 5, e.config.Runtime.K3s, e.config.Runtime.Core); err != nil {\n    panic(errors.WithMessage(err, \"failed to start wrangler controllers\"))\n}\n\n// after (retry briefly to tolerate apiserver warmup instead of panicking)\nif err := wait.PollUntilContextTimeout(ctx, 5*time.Second, 2*time.Minute, true,\n    func(ctx context.Context) (bool, error) {\n        return start.All(ctx, 5, e.config.Runtime.K3s, e.config.Runtime.Core) == nil, nil\n    }); err != nil {\n    panic(errors.WithMessage(err, \"failed to start wrangler controllers\"))\n}","handlingStrategy":"retry","validationCode":"// operator-level pre-check: local apiserver must be ready before k3s (re)starts leader-elected controllers\nif !apiReady(\"https://127.0.0.1:6443/readyz\", serverCA, clientCert, clientKey, 2*time.Minute) {\n    return fmt.Errorf(\"apiserver not ready; fix apiserver/etcd health before restarting k3s\")\n}","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Restart k3s only after the local apiserver reports ready, so informer factory startup has a healthy endpoint","Keep the datastore behind the apiserver (embedded etcd or sqlite) healthy and resource-sufficient; slow watches are the usual trigger","Align k3s versions across control-plane nodes during upgrades to avoid wrangler/informer contract drift","Monitor for repeated panics in logs and treat them as a signal to investigate apiserver latency rather than looping restarts"],"tags":["wrangler","informer","apiserver","etcd","panic","k3s"],"backgroundTag":null,"analyzedSha":"6ba341e396edc16b8dcae978a7c5e3ac7ee5606e","analyzedAt":"2026-08-15T16:27:54.286Z","schemaVersion":2},"datasetVersion":"2026-08-15T17:31:12.345Z"}