kubernetes/kops · warning

found server %s(%d) with unexpected state: %q

Error message

found server %s(%d) with unexpected state: %q

What it means

The Hetzner server was found, but its status is neither 'running' nor 'starting', so kops refuses to derive identity information from it. States like 'off', 'stopping', 'migrating', or 'deleting' mean the node's identity data is not trustworthy or the node is going away.

Source

Thrown at pkg/nodeidentity/hetzner/identify.go:97

	// If caching is enabled try pulling nodeidentity.Info from cache before doing a Hetzner Cloud API call.
	if i.cacheEnabled {
		obj, exists, err := i.cache.GetByKey(serverID)
		if err != nil {
			klog.Warningf("Nodeidentity info cache lookup failure: %v", err)
		}
		if exists {
			return obj.(*nodeidentity.Info), nil
		}
	}

	server, err := i.getServer(serverID)
	if err != nil {
		return nil, err
	}

	if server.Status != hcloud.ServerStatusRunning && server.Status != hcloud.ServerStatusStarting {
		return nil, fmt.Errorf("found server %s(%d) with unexpected state: %q", server.Name, server.ID, server.Status)
	}

	labels := map[string]string{}
	for key, value := range server.Labels {
		switch {
		case key == hetzner.TagKubernetesInstanceRole:
			switch kops.InstanceGroupRole(value) {
			case kops.InstanceGroupRoleControlPlane:
				labels[nodelabels.RoleLabelControlPlane20] = ""
			case kops.InstanceGroupRoleNode:
				labels[nodelabels.RoleLabelNode16] = ""
			case kops.InstanceGroupRoleAPIServer:
				labels[nodelabels.RoleLabelAPIServer16] = ""
			default:
				klog.Warningf("Unknown node role %q for server %s(%d)", value, server.Name, server.ID)
			}
		case strings.HasPrefix(key, hetzner.TagKubernetesNodeLabelPrefix):
			labels[strings.TrimPrefix(key, hetzner.TagKubernetesNodeLabelPrefix)] = value

View on GitHub (pinned to 4c8573c808)

Solutions

1. Start the server (hcloud server poweron <name>) so it returns to 'running'
2. If the node is being decommissioned, delete the Node object and ignore this error
3. Wait and retry if the state is transient (starting/migrating)
4. Check Hetzner Cloud console / robot for automated shutdowns or billing suspensions

Example fix

// before
if server.Status != hcloud.ServerStatusRunning && server.Status != hcloud.ServerStatusStarting {
  return nil, fmt.Errorf("found server %s(%d) with unexpected state: %q", ...)
}
// after (caller tolerates teardown states)
info, err := id.IdentifyNode(ctx, node)
if err != nil && strings.Contains(err.Error(), "unexpected state") {
  return nil, nil // node is stopping/deleting; skip
}

srv, _, _ := client.Server.GetByID(ctx, id)
if srv != nil && srv.Status != "running" && srv.Status != "starting" { /* skip node */ }

func serverQueryable(s *hcloud.Server) bool {
  return s != nil && (s.Status == hcloud.ServerStatusRunning || s.Status == hcloud.ServerStatusStarting)
}

info, err := IdentifyNode(ctx, node)
if err != nil && strings.Contains(err.Error(), "unexpected state") {
  // node powering off/deleting: tolerate and requeue
}

When it happens

Trigger: server.Status is any value other than hcloud.ServerStatusRunning or hcloud.ServerStatusStarting — e.g. server stopped via Hetzner console, being deleted, or in migration at the moment of the lookup.

Common situations: Server manually powered off in the Hetzner Cloud console; node being deleted during a rolling update while its identity is still queried; Hetzner host migration in progress.

Related errors

• failed to get private networks from hetzner cloud metadata:
• failed to convert private networks to object: %w
• instance %v not managed by mig %s
• %s is required
• providerID not set for node %s

AI-assisted analysis of kubernetes/kops@4c8573c808 (2026-09-05). Data as JSON: /api/errors/5c17564718978503. Report an issue: GitHub.