k3s-io/k3s · warning

no peer addresses available

Error message

no peer addresses available

What it means

k3s embedded spegel (pkg/spegel/spegel.go, peerInfo handler): the internal HTTP endpoint other nodes contact for p2p registry peers queries the bootstrapper (Bootstrapper.Get) for peer info and formats each peer's multiaddresses as '/p2p/<id>' entries. If peers were returned but none expose any addresses, addrs stays empty and the handler answers HTTP 503 with 'no peer addresses available'. It is a transient availability signal, not a crash: clients retry and succeed once peers register routable addresses.

Source

Thrown at pkg/spegel/spegel.go:339

// peerInfo sends a peer address retrieved from the bootstrapper via HTTP
func (c *Config) peerInfo() http.HandlerFunc {
	return http.HandlerFunc(func(resp http.ResponseWriter, req *http.Request) {
		info, err := c.Bootstrapper.Get(req.Context())
		if err != nil {
			http.Error(resp, err.Error(), http.StatusInternalServerError)
			return
		}

		addrs := []string{}
		for _, ai := range info {
			for _, ma := range ai.Addrs {
				addrs = append(addrs, fmt.Sprintf("%s/p2p/%s", ma, ai.ID))
			}
		}

		if len(addrs) == 0 {
			http.Error(resp, "no peer addresses available", http.StatusServiceUnavailable)
			return
		}

		client, _, _ := net.SplitHostPort(req.RemoteAddr)
		if req.Header.Get("Accept") == "application/json" {
			b, err := json.Marshal(addrs)
			if err != nil {
				http.Error(resp, err.Error(), http.StatusInternalServerError)
				return
			}
			logrus.Debugf("Serving p2p peer addrs %v to client at %s", addrs, client)
			resp.Header().Set("Content-Type", "application/json")
			resp.WriteHeader(http.StatusOK)
			resp.Write(b)
			return
		}

		logrus.Debugf("Serving p2p peer addr %v to client at %s", addrs[0], client)

View on GitHub (pinned to 6ba341e396)

Solutions

1. Treat it as transient: wait for peer bootstrap (seconds to a couple of minutes) and let the mirror client retry; images then pull from peers or fall back to the upstream registry
2. Verify peer discovery: check that other nodes' spegel state is healthy and that the discovery backend (headless service / bootstrap list) resolves them with addresses
3. Confirm CNI/flannel is up on all nodes so peers advertise routable multiaddresses
4. On clusters where p2p mirroring is not wanted, disable spegel (unsetting the embedded registry mirror envs) so the endpoint is not consulted

Example fix

// before
resp, err := http.Get(peerInfoURL)
if err != nil {
    return err
}
addrs := parsePeers(resp)

// after (client side: honor 503 with backoff until peers register)
var addrs []string
err := wait.PollUntilContextTimeout(ctx, 2*time.Second, 2*time.Minute, true,
    func(ctx context.Context) (bool, error) {
        resp, err := http.Get(peerInfoURL)
        if err != nil {
            return false, nil
        }
        defer resp.Body.Close()
        if resp.StatusCode == http.StatusServiceUnavailable {
            return false, nil // no peer addresses available yet
        }
        addrs = parsePeers(resp)
        return len(addrs) > 0, nil
    })
if err != nil {
    return fmt.Errorf("no peer addresses available after retry window: %w", err)
}

// caller-side probe: only use the p2p mirror once the peer endpoint serves addresses
resp, err := http.Head(peerInfoURL)
if err == nil && resp.StatusCode == http.StatusOK {
    // peers available; safe to configure the registry mirror with p2p endpoints
}

// 503 means 'not yet'; retry with backoff instead of failing the pull
for attempt := 0; attempt < 5; attempt++ {
    resp, err := http.Get(peerInfoURL)
    if err == nil {
        defer resp.Body.Close()
        if resp.StatusCode == http.StatusOK {
            return parsePeers(resp)
        }
        if resp.StatusCode != http.StatusServiceUnavailable {
            return nil, fmt.Errorf("peer info endpoint: %s", resp.Status)
        }
    }
    time.Sleep(time.Duration(attempt+1) * 2 * time.Second)
}
return nil, errors.New("no peer addresses available after retries")

When it happens

Trigger: A node's registry mirror requesting peer addresses right at cluster bootstrap before any spegel peers have registered with the bootstrapper (or before headless-service-based discovery resolves); peers present in the list but with empty Addrs because their overlay/flannel addresses are not yet assigned; effectively single-peer clusters where the only peer is the caller itself.

Common situations: Fresh k3s clusters pulling images in the first minutes after boot; nodes joining while CNI/flannel is still converging so peer multiaddresses are missing; clusters where most nodes are drained or cordoned; spegel disabled-by-detection on tiny clusters leaving no peers to serve.

Related errors

• failed to unmarshal apiserver addresses from etcd: %v
• insufficient PSK bytes
• ipv4 mode requested but no ipv4 network provided
• incorrect netMode for flannel tailscale backend
• all servers failed

AI-assisted analysis of k3s-io/k3s@6ba341e396 (2026-08-15). Data as JSON: /api/errors/ae14e2093ec60b61. Report an issue: GitHub.