What causes "error deleting Instance %s: %w" in kubernetes/kops?

Check the wrapped googleapi error code: 409/operation-in-progress means retry once the concurrent operation completes

error deleting Instance %s: %w

Error message

error deleting Instance %s: %w

What it means

This error is returned by DeleteInstance when Instances().Delete(project, zone, name) fails with anything other than 404 Not Found. It wraps the googleapi error with the instance's selfLink. Not-found instances are treated as already deleted and return nil, so this error signals a real API failure: permissions, dependency (e.g. instance protected by deletion protection or still attached), or a transient API problem.

Source

Thrown at upup/pkg/fi/cloudup/gce/wrappers.go:101

	return c.WaitForOp(op)
}

// DeleteInstance deletes the specified instance (by URL) in GCE
func DeleteInstance(c GCECloud, instanceSelfLink string) error {
	klog.V(2).Infof("Deleting GCE Instance %s", instanceSelfLink)
	u, err := ParseGoogleCloudURL(instanceSelfLink)
	if err != nil {
		return err
	}

	op, err := c.Compute().Instances().Delete(u.Project, u.Zone, u.Name)
	if err != nil {
		if IsNotFound(err) {
			klog.Infof("Instance not found, assuming deleted: %q", instanceSelfLink)
			return nil
		}
		return fmt.Errorf("error deleting Instance %s: %w", instanceSelfLink, err)
	}

	return c.WaitForOp(op)
}

// ListManagedInstances lists the specified InstanceGroupManagers in GCE
func ListManagedInstances(c GCECloud, igm *compute.InstanceGroupManager) ([]*compute.ManagedInstance, error) {
	ctx := context.Background()
	project := c.Project()

	zoneName := LastComponent(igm.Zone)

	// TODO: Only select a subset of fields
	//	req.Fields(
	//		googleapi.Field("items/selfLink"),
	//		googleapi.Field("items/metadata/items[key='cluster-name']"),
	//		googleapi.Field("items/metadata/items[key='instance-template']"),
	//	)

View on GitHub (pinned to 4c8573c808)

Solutions

Check the wrapped googleapi error code: 409/operation-in-progress means retry once the concurrent operation completes
Disable deletion protection (`gcloud compute instances update NAME --no-deletion-protection`) if that flag is set
Confirm the selfLink parses to the right project/zone and the instance still exists with `gcloud compute instances describe`
Verify IAM (compute.instances.delete / compute.instanceAdmin.v1) for the service account
Retry — kOps treats 404 as success, so re-running the delete is safe and idempotent

Example fix

// before: single-shot delete, fails on protection flag
if err := DeleteInstance(cloud, instSelfLink); err != nil {
	return err
}
// after: clear deletion protection, then retry delete
if err := DeleteInstance(cloud, instSelfLink); err != nil {
	if strings.Contains(err.Error(), "deletionProtection") {
		if _, e := cloud.Compute().Instances().SetDeletionProtection(proj, zone, name, false); e != nil {
			return e
		}
		return DeleteInstance(cloud, instSelfLink)
	}
	return err
}

Defensive patterns

Strategy: try-catch

Validate before calling

// Confirm the instance exists and lacks deletion protection before deleting
inst, err := cloud.Compute().Instances().Get(project, zone, name)
if err != nil {
	if isNotFound(err) {
		return nil
	}
	return err
}
if inst.DeletionProtection {
	if _, err := cloud.Compute().Instances().SetDeletionProtection(project, zone, name, false); err != nil {
		return err
	}
}

Type guard

func isDeletionConflict(err error) bool {
	var gerr *googleapi.Error
	return errors.As(err, &gerr) && (gerr.Code == 409 || gerr.Code == 412)
}

Try / catch

err := gce.DeleteInstance(cloud, selfLink)
if err != nil {
	var gerr *googleapi.Error
	if errors.As(err, &gerr) {
		switch gerr.Code {
		case 404:
			return nil // already deleted
		case 409, 412:
			// concurrent operation; retry after backoff
		case 403:
			// check IAM and deletion protection
		}
	}
	return err
}

Prevention

Avoid setting deletionProtection on kOps-managed instances
Run teardown serially per instance to avoid 409 conflicts with in-flight deletes
Validate selfLinks parse to the expected project/zone before deleting
Grant compute.instanceAdmin.v1 to the operator service account

When it happens

Trigger: Calling DeleteInstance for a remaining GCE VM (orphaned instances after MIG teardown) where Instances().Delete returns a non-NotFound error: deletion protection enabled, instance already being deleted by another operation (conflict), IAM denial, wrong zone/project parsed from the selfLink, or API 5xx.

Common situations: Instances with deletionProtection=true left from custom provisioning; two teardown processes deleting the same instance concurrently causing OPERATION_NOT_DONE/409 conflicts; stale selfLinks after instances were rebuilt; service account lacking compute.instances.delete; zone misconfiguration causing 404-style failures surfaced as other errors.

Related errors

AI-assisted analysis of kubernetes/kops@4c8573c808 (2026-09-05). Data as JSON: /api/errors/66d61d5a100d6f07. Report an issue: GitHub.