{"record":{"id":"5c17564718978503","repo":"kubernetes/kops","slug":"found-server-s-d-with-unexpected-state-q","errorCode":null,"errorMessage":"found server %s(%d) with unexpected state: %q","messagePattern":"found server (.+?)\\((.+?)\\) with unexpected state: %q","errorType":"validation","errorClass":null,"httpStatus":null,"severity":"warning","filePath":"pkg/nodeidentity/hetzner/identify.go","lineNumber":97,"sourceCode":"\n\t// If caching is enabled try pulling nodeidentity.Info from cache before doing a Hetzner Cloud API call.\n\tif i.cacheEnabled {\n\t\tobj, exists, err := i.cache.GetByKey(serverID)\n\t\tif err != nil {\n\t\t\tklog.Warningf(\"Nodeidentity info cache lookup failure: %v\", err)\n\t\t}\n\t\tif exists {\n\t\t\treturn obj.(*nodeidentity.Info), nil\n\t\t}\n\t}\n\n\tserver, err := i.getServer(serverID)\n\tif err != nil {\n\t\treturn nil, err\n\t}\n\n\tif server.Status != hcloud.ServerStatusRunning && server.Status != hcloud.ServerStatusStarting {\n\t\treturn nil, fmt.Errorf(\"found server %s(%d) with unexpected state: %q\", server.Name, server.ID, server.Status)\n\t}\n\n\tlabels := map[string]string{}\n\tfor key, value := range server.Labels {\n\t\tswitch {\n\t\tcase key == hetzner.TagKubernetesInstanceRole:\n\t\t\tswitch kops.InstanceGroupRole(value) {\n\t\t\tcase kops.InstanceGroupRoleControlPlane:\n\t\t\t\tlabels[nodelabels.RoleLabelControlPlane20] = \"\"\n\t\t\tcase kops.InstanceGroupRoleNode:\n\t\t\t\tlabels[nodelabels.RoleLabelNode16] = \"\"\n\t\t\tcase kops.InstanceGroupRoleAPIServer:\n\t\t\t\tlabels[nodelabels.RoleLabelAPIServer16] = \"\"\n\t\t\tdefault:\n\t\t\t\tklog.Warningf(\"Unknown node role %q for server %s(%d)\", value, server.Name, server.ID)\n\t\t\t}\n\t\tcase strings.HasPrefix(key, hetzner.TagKubernetesNodeLabelPrefix):\n\t\t\tlabels[strings.TrimPrefix(key, hetzner.TagKubernetesNodeLabelPrefix)] = value","sourceCodeStart":79,"sourceCodeEnd":115,"githubUrl":"https://github.com/kubernetes/kops/blob/4c8573c808a73d578c5eadc86d410646ea0b0d73/pkg/nodeidentity/hetzner/identify.go#L79-L115","documentation":"The Hetzner server was found, but its status is neither 'running' nor 'starting', so kops refuses to derive identity information from it. States like 'off', 'stopping', 'migrating', or 'deleting' mean the node's identity data is not trustworthy or the node is going away.","triggerScenarios":"server.Status is any value other than hcloud.ServerStatusRunning or hcloud.ServerStatusStarting — e.g. server stopped via Hetzner console, being deleted, or in migration at the moment of the lookup.","commonSituations":"Server manually powered off in the Hetzner Cloud console; node being deleted during a rolling update while its identity is still queried; Hetzner host migration in progress.","solutions":["Start the server (hcloud server poweron <name>) so it returns to 'running'","If the node is being decommissioned, delete the Node object and ignore this error","Wait and retry if the state is transient (starting/migrating)","Check Hetzner Cloud console / robot for automated shutdowns or billing suspensions"],"exampleFix":"// before\nif server.Status != hcloud.ServerStatusRunning && server.Status != hcloud.ServerStatusStarting {\n  return nil, fmt.Errorf(\"found server %s(%d) with unexpected state: %q\", ...)\n}\n// after (caller tolerates teardown states)\ninfo, err := id.IdentifyNode(ctx, node)\nif err != nil && strings.Contains(err.Error(), \"unexpected state\") {\n  return nil, nil // node is stopping/deleting; skip\n}","handlingStrategy":"fallback","validationCode":"srv, _, _ := client.Server.GetByID(ctx, id)\nif srv != nil && srv.Status != \"running\" && srv.Status != \"starting\" { /* skip node */ }","typeGuard":"func serverQueryable(s *hcloud.Server) bool {\n  return s != nil && (s.Status == hcloud.ServerStatusRunning || s.Status == hcloud.ServerStatusStarting)\n}","tryCatchPattern":"info, err := IdentifyNode(ctx, node)\nif err != nil && strings.Contains(err.Error(), \"unexpected state\") {\n  // node powering off/deleting: tolerate and requeue\n}","preventionTips":["Avoid stopping servers via the Hetzner console while in a cluster","Coordinate node shutdowns with kops rolling-update","Check Hetzner console for suspensions/migrations affecting servers"],"tags":["hetzner","server-state","node-lifecycle"],"backgroundTag":"unexpected-server-state","analyzedSha":"4c8573c808a73d578c5eadc86d410646ea0b0d73","analyzedAt":"2026-09-05T04:13:19.212Z","contentChangedAt":"2026-09-05T04:13:19.212Z","schemaVersion":2},"datasetVersion":"2026-09-12T07:17:12.445Z"}