{"record":{"id":"9a017970b9e73fea","repo":"k3s-io/k3s","slug":"server-s-is-stopping","errorCode":null,"errorMessage":"server %s is stopping","messagePattern":"server (.+?) is stopping","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"warning","filePath":"pkg/agent/loadbalancer/servers.go","lineNumber":421,"sourceCode":"\t\tstate:          state,\n\t\tlastTransition: time.Now(),\n\t\thealthCheck:    func() HealthCheckResult { return HealthCheckResultUnknown },\n\t\tconnections:    make(map[net.Conn]struct{}),\n\t}\n}\n\nfunc (s *server) String() string {\n\tformat := \"%s@%s\"\n\tif s.isDefault {\n\t\tformat += \"*\"\n\t}\n\treturn fmt.Sprintf(format, s.address, s.state)\n}\n\n// dialContext dials a new connection to the server using the environment's proxy settings, and adds its wrapped connection to the map\nfunc (s *server) dialContext(ctx context.Context, network string) (net.Conn, error) {\n\tif s.state == stateInvalid {\n\t\treturn nil, fmt.Errorf(\"server %s is stopping\", s.address)\n\t}\n\n\tconn, err := defaultDialer.Dial(network, s.address)\n\tif err != nil {\n\t\treturn nil, err\n\t}\n\n\t// Wrap the connection and add it to the server's connection map\n\ts.mutex.Lock()\n\tdefer s.mutex.Unlock()\n\n\twrappedConn := &serverConn{server: s, Conn: conn}\n\ts.connections[wrappedConn] = struct{}{}\n\treturn wrappedConn, nil\n}\n\n// closeAll closes all connections to the server, and removes their entries from the map\nfunc (s *server) closeAll() {","sourceCodeStart":403,"sourceCodeEnd":439,"githubUrl":"https://github.com/k3s-io/k3s/blob/6ba341e396edc16b8dcae978a7c5e3ac7ee5606e/pkg/agent/loadbalancer/servers.go#L403-L439","documentation":"server.dialContext refuses new connections once the entry's state is stateInvalid — the entry is being torn down after the load balancer's server list was updated. The error tells consumers to stop using the cached server object and dial through the balancer again (serverList.dialContext picks a live entry).","triggerScenarios":"A dial racing a server removal: lb.Update marked the server invalid while a consumer still holding the *server called dialContext.","commonSituations":"HA setups with rapid server-list churn; connections opened during component shutdown; consumers caching server references across reconfigurations.","solutions":["Dial through the load balancer's dialer instead of a cached *server — it selects an entry that is not stopping","Drop cached server references after address-list updates","Retry the operation once after the list settles"],"exampleFix":"// before (stale reference)\nconn, err := srv.dialContext(ctx, \"tcp\") // srv captured before an lb.Update\n\n// after (balancer re-selects a valid server)\nconn, err := lb.dialContext(ctx, \"tcp\", lb.ServerAddresses()[0])","handlingStrategy":"retry","validationCode":null,"typeGuard":null,"tryCatchPattern":"conn, err := dial() // may fail with \"server X is stopping\"\nif err != nil && strings.HasSuffix(err.Error(), \"is stopping\") {\n    // entry was invalidated by an address-list update:\n    // re-dial through the load balancer (it skips stateInvalid servers) instead of this server\n    conn, err = lbDial()\n}","preventionTips":["Dial via the load balancer rather than cached *server references","After lb.Update, drop previously obtained server entries","Treat 'is stopping' as transient: one retry through the balancer is the expected recovery"],"tags":["loadbalancer","lifecycle","transient","network","retry"],"backgroundTag":null,"analyzedSha":"6ba341e396edc16b8dcae978a7c5e3ac7ee5606e","analyzedAt":"2026-08-15T16:27:54.286Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}