kubernetes/kops · error

failed to detach instance: %v

Error message

failed to detach instance: %v

What it means

UpdateSingleInstance detaches the instance from its cloud autoscaling/instance group before replacing it (surge updates). When c.detachInstance fails, the error is wrapped as 'failed to detach instance'. Control-plane instances are skipped by design and Karpenter-managed groups are skipped entirely.

Source

Thrown at pkg/instancegroups/instancegroups.go:772

		if apierrors.IsNotFound(err) {
			return nil
		}

		return fmt.Errorf("error deleting node: %v", err)
	}

	return nil
}

// UpdateSingleInstance performs a rolling update on a single instance
func (c *RollingUpdateCluster) UpdateSingleInstance(ctx context.Context, cloudMember *cloudinstances.CloudInstance, detach bool) error {
	if detach {
		if cloudMember.CloudInstanceGroup.InstanceGroup.IsControlPlane() {
			klog.Warning("cannot detach control-plane instances. Assuming --surge=false")
		} else if cloudMember.CloudInstanceGroup.InstanceGroup.Spec.Manager != api.InstanceManagerKarpenter {
			err := c.detachInstance(cloudMember)
			if err != nil {
				return fmt.Errorf("failed to detach instance: %v", err)
			}
			if err := c.maybeValidate(" after detaching instance", c.ValidateCount, cloudMember.CloudInstanceGroup); err != nil {
				return err
			}
		}
	}

	return c.drainTerminateAndWait(ctx, cloudMember, 0)
}

View on GitHub (pinned to 4c8573c808)

Solutions

1. Confirm instance membership in the ASG (`aws autoscaling describe-auto-scaling-instances`) and retry once state is consistent
2. Grant IAM permission autoscaling:DetachInstances on the ASG
3. For Karpenter-managed groups, set spec.Manager: karpenter in the InstanceGroup so detach is skipped
4. Re-run `kops rolling-update cluster` — it re-syncs group membership before detaching

Example fix

// before
kind: InstanceGroup
metadata:
  name: nodes
// after
kind: InstanceGroup
metadata:
  name: nodes
spec:
  manager: karpenter

members, err := asgSvc.DescribeAutoScalingInstances(&autoscaling.DescribeAutoScalingInstancesInput{InstanceIds: []string{instanceID}})
if err != nil || len(members.AutoScalingInstances) == 0 {
    return fmt.Errorf("instance %s not in any ASG; detach will fail", instanceID)
}

if err := c.UpdateSingleInstance(ctx, cloudMember, detach, true); err != nil {
    if strings.Contains(err.Error(), "failed to detach instance") {
        klog.Warningf("detach failed: %v; will retry after resync", err)
        time.Sleep(backoff)
        return c.UpdateSingleInstance(ctx, cloudMember, detach, true)
    }
    return err
}

When it happens

Trigger: The provider detach call fails: AWS AutoScalingGroup DetachInstances errors (instance not in group, ASG not found), permission denied, or the group is managed by a controller that also mutates membership concurrently.

Common situations: Instance was already removed/replaced by the autoscaler so the ID is no longer a member; IAM missing autoscaling:DetachInstances; kOps state out of sync after a partially failed previous rolling update; using Karpenter-managed groups incorrectly flagged (Manager not set).

Related errors

• the image for the hook exec action not set
• errShutdown
• error detaching instance %q, node %q: %v
• DIGITALOCEAN_ACCESS_TOKEN is required
• IP version is incorrect

AI-assisted analysis of kubernetes/kops@4c8573c808 (2026-09-05). Data as JSON: /api/errors/ec696fafba096bb5. Report an issue: GitHub.