kubernetes/kops · error

error deleting instance %q: %v

Error message

error deleting instance %q: %v

What it means

The ec2.TerminateInstances call failed with an error other than InvalidInstanceID.NotFound (which kOps deliberately treats as already-deleted success). This means the instance could not be terminated for a real reason — permissions, invalid state, dependency, or API/network failure — and kOps surfaces the raw AWS error wrapped with the instance ID.

Source

Thrown at upup/pkg/fi/cloudup/awsup/aws_cloud.go:505

	return nil
}

func deleteInstance(ctx context.Context, c AWSCloud, i *cloudinstances.CloudInstance) error {
	id := i.ID
	if id == "" {
		return fmt.Errorf("id was not set on CloudInstance: %v", i)
	}

	request := &ec2.TerminateInstancesInput{
		InstanceIds: []string{id},
	}

	if _, err := c.EC2().TerminateInstances(ctx, request); err != nil {
		if AWSErrorCode(err) == "InvalidInstanceID.NotFound" {
			klog.V(2).Infof("Got InvalidInstanceID.NotFound error deleting instance %q; will treat as already-deleted", id)
		} else {
			return fmt.Errorf("error deleting instance %q: %v", id, err)
		}
	}

	klog.V(8).Infof("deleted aws ec2 instance %q", id)

	return nil
}

// deregisterInstance ensures that the instance is fully drained/removed from all associated loadBalancers and targetGroups before termination.
func deregisterInstance(ctx context.Context, c AWSCloud, i *cloudinstances.CloudInstance) error {
	asg := i.CloudInstanceGroup.Raw.(*autoscalingtypes.AutoScalingGroup)

	asgDetails, err := c.Autoscaling().DescribeAutoScalingGroups(ctx, &autoscaling.DescribeAutoScalingGroupsInput{
		AutoScalingGroupNames: []string{aws.ToString(asg.AutoScalingGroupName)},
	})
	if err != nil {
		return fmt.Errorf("error describing autoScalingGroups: %v", err)
	}

View on GitHub (pinned to 4c8573c808)

Solutions

1. Read the wrapped AWS error code and address it directly (AccessDenied → fix IAM; OperationNotPermitted → disable termination protection / scale-down protection)
2. Ensure the IAM role used by kOps has ec2:TerminateInstances on the relevant instances
3. Check for ASG instance protection or scale-in protection on the instance and remove it
4. If throttling/transient, retry; InvalidInstanceID.NotFound is already treated as success

Example fix

// before: instance protected
// error deleting instance "i-0abc...": OperationNotPermitted: The instance 'i-0abc' may not be terminated...
// after: disable protection then retry
aws autoscaling set-instance-protection --instance-ids i-0abc... --auto-scaling-group-name nodes --no-protected-from-scale-in
aws ec2 modify-instance-attribute --instance-id i-0abc... --no-disable-api-termination

// pre-check termination protection and ASG scale-in protection
out, _ := svc.DescribeInstanceAttribute(&ec2.DescribeInstanceAttributeInput{
	InstanceId: aws.String(inst.ID), Attribute: aws.String("disableApiTermination")})
if aws.ToBool(out.DisableApiTermination.Value) {
	return fmt.Errorf("instance %s has API termination enabled-protection; disable first", inst.ID)
}

func isAlreadyDeleted(err error) bool {
	var ae smithy.APIError
	return errors.As(err, &ae) && ae.ErrorCode() == "InvalidInstanceID.NotFound"
}

if err := cloud.DeleteInstance(inst); err != nil {
	var ae smithy.APIError
	if errors.As(err, &ae) {
		switch ae.ErrorCode() {
		case "OperationNotPermitted":
			// disable termination/scale-in protection then retry
		case "AccessDenied":
			// fix IAM ec2:TerminateInstances
		}
	}
	return err
}

When it happens

Trigger: AWSCloud.deleteInstance calls c.EC2().TerminateInstances and AWS returns e.g. AccessDenied, InvalidInstanceID.Malformed, OperationNotPermitted (instance protected against termination / attached EBS volume with termination protection), or a throttling/network error.

Common situations: IAM role missing ec2:TerminateInstances; instance has termination protection or is part of an ASG with instance protection; spot instance interruption conflicts; wrong region/credentials; malformed instance ID passed in.

Related errors

• id was not set on CloudInstance: %v
• DIGITALOCEAN_ACCESS_TOKEN is required
• timed out waiting for volume to detach
• unexpected number of network interfaces for instance %q: %v
• unexpected amount of ipv6 prefixes on interface %q: %v

AI-assisted analysis of kubernetes/kops@4c8573c808 (2026-09-05). Data as JSON: /api/errors/4f97b260d7fe9296. Report an issue: GitHub.