k3s-io/k3s · critical

all servers failed

Error message

all servers failed

What it means

The k3s agent runs an embedded load balancer in front of the supervisor/apiserver; dialContext iterates the current server list and tries each endpoint. If every server's dial fails (connection refused/timeout/unreachable), the LB records a failure for each and returns this aggregate error. It means the agent-side proxy has no reachable control-plane endpoint at that moment.

Source

Thrown at pkg/agent/loadbalancer/servers.go:500

		}
	}, time.Second, ctx.Done())
	logrus.Debugf("Stopped health checking for load balancer %s", serviceName)
}

// dialContext attemps to dial a connection to a server from the server list.
// Success or failure is recorded to ensure that server state is updated appropriately.
func (sl *serverList) dialContext(ctx context.Context, network, _ string) (net.Conn, error) {
	for _, s := range sl.getServers() {
		dialTime := time.Now()
		conn, err := s.dialContext(ctx, network)
		if err == nil {
			sl.recordSuccess(s, reasonDial)
			return conn, nil
		}
		logrus.Debugf("Dial error from server %s after %s: %s", s, time.Now().Sub(dialTime), err)
		sl.recordFailure(s, reasonDial)
	}
	return nil, errors.New("all servers failed")
}

// compareServers is a comparison function that can be used to sort the server list
// so that servers with a more preferred state, or higher number of connections, are ordered first.
func compareServers(a, b *server) int {
	c := cmp.Compare(b.state, a.state)
	if c == 0 {
		return cmp.Compare(len(b.connections), len(a.connections))
	}
	return c
}

View on GitHub (pinned to 6ba341e396)

Solutions

1. Verify at least one server is up and its supervisor port answers: curl -k https://<server>:6443/readyz from the agent host
2. Check network path: security groups, firewalld/iptables, and DNS/VIP resolution for the --server address
3. If all servers were intentionally stopped (e.g. cluster-reset), bring one back before restarting agents
4. For HA, keep multiple server entries reachable so the LB can fail over instead of exhausting the list

Example fix

# before
k3s agent --server https://typo.example.com:6443 --token ...
# -> all servers failed

# after
k3s agent --server https://10.0.0.10:6443 --token ... # verified reachable: curl -k https://10.0.0.10:6443/readyz

// Preflight from the agent host: at least one supervisor endpoint must answer
for _, ep := range serverEndpoints {
    resp, err := http.Get(ep + "/readyz") // kubeconfig-authenticated client in practice
    if err == nil && resp.StatusCode/100 == 2 { return nil }
}
return errors.New("no reachable server endpoint; check firewall/DNS before starting agent")

for attempt := 0; attempt < maxAttempts; attempt++ {
    conn, err := lb.DialContext(ctx, "tcp", "")
    if err == nil { break }
    if strings.Contains(err.Error(), "all servers failed") {
        time.Sleep(backoff(attempt)) // LB keeps probing; retry until a server recovers
        continue
    }
    return err
}

When it happens

Trigger: Agent bootstrapping or steady-state operation where all configured --server / load-balancer endpoints fail TCP dial: server down, wrong host/port, firewall blocking 6443, DNS resolving to a dead VIP, or all servers paused for etcd maintenance/cluster-reset.

Common situations: Typo'd or stale --server URL; security-group/firewall blocking the supervisor port; DNS entry for the cluster VIP not updated after server replacement; agents left running while all servers were stopped simultaneously.

Related errors

• ipv4 mode requested but no ipv4 network provided
• incorrect netMode for flannel tailscale backend
• Initial server URL host is not defined for load balancer
• invalid username/password combination
• --token is required

AI-assisted analysis of k3s-io/k3s@6ba341e396 (2026-08-15). Data as JSON: /api/errors/39a87b29a698de1a. Report an issue: GitHub.