{"record":{"id":"39a87b29a698de1a","repo":"k3s-io/k3s","slug":"all-servers-failed","errorCode":null,"errorMessage":"all servers failed","messagePattern":"all servers failed","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"critical","filePath":"pkg/agent/loadbalancer/servers.go","lineNumber":500,"sourceCode":"\t\t}\n\t}, time.Second, ctx.Done())\n\tlogrus.Debugf(\"Stopped health checking for load balancer %s\", serviceName)\n}\n\n// dialContext attemps to dial a connection to a server from the server list.\n// Success or failure is recorded to ensure that server state is updated appropriately.\nfunc (sl *serverList) dialContext(ctx context.Context, network, _ string) (net.Conn, error) {\n\tfor _, s := range sl.getServers() {\n\t\tdialTime := time.Now()\n\t\tconn, err := s.dialContext(ctx, network)\n\t\tif err == nil {\n\t\t\tsl.recordSuccess(s, reasonDial)\n\t\t\treturn conn, nil\n\t\t}\n\t\tlogrus.Debugf(\"Dial error from server %s after %s: %s\", s, time.Now().Sub(dialTime), err)\n\t\tsl.recordFailure(s, reasonDial)\n\t}\n\treturn nil, errors.New(\"all servers failed\")\n}\n\n// compareServers is a comparison function that can be used to sort the server list\n// so that servers with a more preferred state, or higher number of connections, are ordered first.\nfunc compareServers(a, b *server) int {\n\tc := cmp.Compare(b.state, a.state)\n\tif c == 0 {\n\t\treturn cmp.Compare(len(b.connections), len(a.connections))\n\t}\n\treturn c\n}\n","sourceCodeStart":482,"sourceCodeEnd":512,"githubUrl":"https://github.com/k3s-io/k3s/blob/6ba341e396edc16b8dcae978a7c5e3ac7ee5606e/pkg/agent/loadbalancer/servers.go#L482-L512","documentation":"The k3s agent runs an embedded load balancer in front of the supervisor/apiserver; dialContext iterates the current server list and tries each endpoint. If every server's dial fails (connection refused/timeout/unreachable), the LB records a failure for each and returns this aggregate error. It means the agent-side proxy has no reachable control-plane endpoint at that moment.","triggerScenarios":"Agent bootstrapping or steady-state operation where all configured --server / load-balancer endpoints fail TCP dial: server down, wrong host/port, firewall blocking 6443, DNS resolving to a dead VIP, or all servers paused for etcd maintenance/cluster-reset.","commonSituations":"Typo'd or stale --server URL; security-group/firewall blocking the supervisor port; DNS entry for the cluster VIP not updated after server replacement; agents left running while all servers were stopped simultaneously.","solutions":["Verify at least one server is up and its supervisor port answers: curl -k https://<server>:6443/readyz from the agent host","Check network path: security groups, firewalld/iptables, and DNS/VIP resolution for the --server address","If all servers were intentionally stopped (e.g. cluster-reset), bring one back before restarting agents","For HA, keep multiple server entries reachable so the LB can fail over instead of exhausting the list"],"exampleFix":"# before\nk3s agent --server https://typo.example.com:6443 --token ...\n# -> all servers failed\n\n# after\nk3s agent --server https://10.0.0.10:6443 --token ... # verified reachable: curl -k https://10.0.0.10:6443/readyz","handlingStrategy":"retry","validationCode":"// Preflight from the agent host: at least one supervisor endpoint must answer\nfor _, ep := range serverEndpoints {\n    resp, err := http.Get(ep + \"/readyz\") // kubeconfig-authenticated client in practice\n    if err == nil && resp.StatusCode/100 == 2 { return nil }\n}\nreturn errors.New(\"no reachable server endpoint; check firewall/DNS before starting agent\")","typeGuard":null,"tryCatchPattern":"for attempt := 0; attempt < maxAttempts; attempt++ {\n    conn, err := lb.DialContext(ctx, \"tcp\", \"\")\n    if err == nil { break }\n    if strings.Contains(err.Error(), \"all servers failed\") {\n        time.Sleep(backoff(attempt)) // LB keeps probing; retry until a server recovers\n        continue\n    }\n    return err\n}","preventionTips":["Run >= 2 servers in HA so the LB has failover targets","Pre-check reachability of the --server endpoint (curl -k https://host:6443/readyz) before starting agents","Keep firewall rules and DNS/VIP records for the supervisor port (6443) correct during maintenance","Alert on the LB's failure counters rather than waiting for total outage"],"tags":["load-balancer","connectivity","agent","supervisor","networking"],"backgroundTag":null,"analyzedSha":"6ba341e396edc16b8dcae978a7c5e3ac7ee5606e","analyzedAt":"2026-08-15T16:27:54.286Z","schemaVersion":2},"datasetVersion":"2026-08-15T17:31:12.345Z"}