k3s-io/k3s · warning

server %s is stopping

Error message

server %s is stopping

What it means

server.dialContext refuses new connections once the entry's state is stateInvalid — the entry is being torn down after the load balancer's server list was updated. The error tells consumers to stop using the cached server object and dial through the balancer again (serverList.dialContext picks a live entry).

Source

Thrown at pkg/agent/loadbalancer/servers.go:421

		state:          state,
		lastTransition: time.Now(),
		healthCheck:    func() HealthCheckResult { return HealthCheckResultUnknown },
		connections:    make(map[net.Conn]struct{}),
	}
}

func (s *server) String() string {
	format := "%s@%s"
	if s.isDefault {
		format += "*"
	}
	return fmt.Sprintf(format, s.address, s.state)
}

// dialContext dials a new connection to the server using the environment's proxy settings, and adds its wrapped connection to the map
func (s *server) dialContext(ctx context.Context, network string) (net.Conn, error) {
	if s.state == stateInvalid {
		return nil, fmt.Errorf("server %s is stopping", s.address)
	}

	conn, err := defaultDialer.Dial(network, s.address)
	if err != nil {
		return nil, err
	}

	// Wrap the connection and add it to the server's connection map
	s.mutex.Lock()
	defer s.mutex.Unlock()

	wrappedConn := &serverConn{server: s, Conn: conn}
	s.connections[wrappedConn] = struct{}{}
	return wrappedConn, nil
}

// closeAll closes all connections to the server, and removes their entries from the map
func (s *server) closeAll() {

View on GitHub (pinned to 6ba341e396)

Solutions

1. Dial through the load balancer's dialer instead of a cached *server — it selects an entry that is not stopping
2. Drop cached server references after address-list updates
3. Retry the operation once after the list settles

Example fix

// before (stale reference)
conn, err := srv.dialContext(ctx, "tcp") // srv captured before an lb.Update

// after (balancer re-selects a valid server)
conn, err := lb.dialContext(ctx, "tcp", lb.ServerAddresses()[0])

conn, err := dial() // may fail with "server X is stopping"
if err != nil && strings.HasSuffix(err.Error(), "is stopping") {
    // entry was invalidated by an address-list update:
    // re-dial through the load balancer (it skips stateInvalid servers) instead of this server
    conn, err = lbDial()
}

When it happens

Trigger: A dial racing a server removal: lb.Update marked the server invalid while a consumer still holding the *server called dialContext.

Common situations: HA setups with rapid server-list churn; connections opened during component shutdown; consumers caching server references across reconfigurations.

Related errors

• unsupported proxy scheme: %s
• this server has not yet been promoted from learner to voting
• apiserver not ready
• no IPv4 CIDRs found
• no IPv4 address found

AI-assisted analysis of k3s-io/k3s@6ba341e396 (2026-08-15). Data as JSON: /api/errors/9a017970b9e73fea. Report an issue: GitHub.