kubernetes/kops · error

found server %s (%s) with unexpected state: %q

Error message

found server %s (%s) with unexpected state: %q

What it means

IdentifyNode resolves a Scaleway instance into kOps node identity labels. After fetching the server, it requires the instance to be in Running or Starting state; otherwise node identity cannot be reliably determined. This error means the node being identified is in a transitional or abnormal lifecycle state.

Source

Thrown at pkg/nodeidentity/scaleway/identify.go:100

	// If caching is enabled try pulling nodeidentity.Info from cache before doing a Scaleway API call.
	if i.cacheEnabled {
		obj, exists, err := i.cache.GetByKey(serverID)
		if err != nil {
			klog.Warningf("Nodeidentity info cache lookup failure: %v", err)
		}
		if exists {
			return obj.(*nodeidentity.Info), nil
		}
	}

	server, err := i.getServer(ctx, serverID)
	if err != nil {
		return nil, err
	}

	if server.State != instance.ServerStateRunning && server.State != instance.ServerStateStarting {
		return nil, fmt.Errorf("found server %s (%s) with unexpected state: %q", server.Name, server.ID, server.State)
	}

	labels := map[string]string{}
	role := scaleway.InstanceRoleFromTags(server.Tags)
	switch kops.InstanceGroupRole(role) {
	case kops.InstanceGroupRoleControlPlane:
		labels[nodelabels.RoleLabelControlPlane20] = ""
	case kops.InstanceGroupRoleNode:
		labels[nodelabels.RoleLabelNode16] = ""
	case kops.InstanceGroupRoleAPIServer:
		labels[nodelabels.RoleLabelAPIServer16] = ""
	default:
		klog.Warningf("Unknown node role %q for server %s(%s)", role, server.Name, server.ID)
	}

	info := &nodeidentity.Info{
		InstanceID: serverID,
		Labels:     labels,

View on GitHub (pinned to 4c8573c808)

Solutions

1. Wait for the instance to reach Running state and let kubelet retry node registration
2. Check the server state in the Scaleway console/API (scaleway instance get) and start it if stopped
3. Investigate why the instance is not running: billing, quota, failed boot, or manual shutdown
4. If the instance should never register, remove it from the cluster / delete the machine

Example fix

// before: server is stopped
server.State = "archived"
// after: ensure the server is running before node registration
scw instance server-action action=start server-id=<id>
# or wait: until server.State == "running"

// Go: pre-check server state via Scaleway API
api := instance.NewAPI(client)
srv, err := api.GetServer(&instance.GetServerRequest{ServerID: id, Zone: zone}, scw.WithContext(ctx))
if err == nil && srv != nil && srv.Server.State != instance.ServerStateRunning && srv.Server.State != instance.ServerStateStarting {
    return fmt.Errorf("defer registration: server %s in state %s", id, srv.Server.State)
}

func serverIsRunnable(s *instance.Server) bool {
    return s != nil && (s.State == instance.ServerStateRunning || s.State == instance.ServerStateStarting)
}

When it happens

Trigger: A Scaleway server whose State is not Running/Starting (e.g. stopped, stopping, error, archived) calls IdentifyNode during node registration.

Common situations: Node rebooting or shutting down during cluster bring-up; a stopped/terminated instance whose cloud provider metadata still routes to the identifier; Scaleway API eventual consistency returning a server still provisioning; instances killed by spot/preemptible reclaim.

Related errors

• found instance %q, but status is %q
• creating client for Scaleway NodeIdentifier: %w
• providerID not set for node %s
• providerID %q not recognized for node %s
• unexpected format for server id %s

AI-assisted analysis of kubernetes/kops@4c8573c808 (2026-09-05). Data as JSON: /api/errors/4a171c756b36966e. Report an issue: GitHub.