kubernetes/kops · error

error deleting InstanceGroupManager %s: %w

Error message

error deleting InstanceGroupManager %s: %w

What it means

This error is returned by DeleteInstanceGroupManager in kOps' GCE cloud provider when the Compute API call InstanceGroupManagers().Delete fails with anything other than a 404 Not Found. It wraps the underlying googleapi error (quota, permission, dependency, or operation conflict) together with the MIG's selfLink so the failing resource is identifiable. A not-found MIG is deliberately treated as already-deleted success and does NOT produce this error.

Source

Thrown at upup/pkg/fi/cloudup/gce/wrappers.go:41

	compute "google.golang.org/api/compute/v1"
	"k8s.io/klog/v2"
)

// DeleteInstanceGroupManager deletes the specified InstanceGroupManager in GCE
func DeleteInstanceGroupManager(c GCECloud, t *compute.InstanceGroupManager) error {
	klog.V(2).Infof("Deleting GCE InstanceGroupManager %s", t.SelfLink)
	u, err := ParseGoogleCloudURL(t.SelfLink)
	if err != nil {
		return err
	}

	op, err := c.Compute().InstanceGroupManagers().Delete(u.Project, u.Zone, u.Name)
	if err != nil {
		if IsNotFound(err) {
			klog.Infof("InstanceGroupManager not found, assuming deleted: %q", t.SelfLink)
			return nil
		}
		return fmt.Errorf("error deleting InstanceGroupManager %s: %w", t.SelfLink, err)
	}

	return c.WaitForOp(op)
}

// DeleteMIGInstances deletes the instances in the MIG in GCE
func DeleteMIGInstances(c GCECloud, t *compute.InstanceGroupManager) error {
	klog.V(2).Infof("Deleting Instances in InstanceGroupManager %s", t.SelfLink)
	u, err := ParseGoogleCloudURL(t.SelfLink)
	if err != nil {
		return err
	}

	op, err := c.Compute().InstanceGroupManagers().Resize(u.Project, u.Zone, u.Name, 0)
	if err != nil {
		if IsNotFound(err) {
			klog.Infof("InstanceGroupManager not found, assuming deleted: %q", t.SelfLink)
			return nil

View on GitHub (pinned to 4c8573c808)

Solutions

1. Check the wrapped googleapi error (%v of the cause) for the exact GCE error code and failed operation name in the message
2. Ensure the service account has compute.instanceGroupManagerAdmin/compute.instanceAdmin IAM roles for the project
3. Retry the kOps delete after in-flight GCE operations on the MIG complete (kops delete cluster --yes is idempotent; already-deleted MIGs are skipped)
4. If the MIG is stuck, inspect it with `gcloud compute instance-groups managed describe` and delete any blocking operations, then re-run
5. For repeated 5xx/quota errors, back off and retry later or raise quota for the region

Example fix

// before: non-idempotent manual teardown that hard-fails on transient errors
if err := DeleteInstanceGroupManager(cloud, igm); err != nil {
	return err
}
// after: retry transient failures; 404 already treated as success
var lastErr error
for i := 0; i < 3; i++ {
	lastErr = DeleteInstanceGroupManager(cloud, igm)
	if lastErr == nil {
		return nil
	}
	if isRetryableGCEError(lastErr) {
		time.Sleep(time.Duration(1<<i) * time.Second)
		continue
	}
	return lastErr
}
return lastErr

// Before tearing down, check the MIG exists and no op is in flight
igm, err := cloud.Compute().InstanceGroupManagers().Get(project, zone, name)
if err != nil {
	if isNotFound(err) {
		return nil // already deleted, skip
	}
	return err
}
if igm.Status != nil && len(igm.Status.PendingActions) > 0 {
	return fmt.Errorf("MIG %s has pending actions; retry later", name)
}

// Unwrap and classify the googleapi error
func isGCEPreconditionOrConflict(err error) bool {
	var gerr *googleapi.Error
	if errors.As(err, &gerr) {
		return gerr.Code == 409 || gerr.Code == 412 || gerr.Code == 429
	}
	return false
}

// In Go, pattern-match the wrapped error rather than try/catch
err := gce.DeleteInstanceGroupManager(cloud, igm)
if err != nil {
	var gerr *googleapi.Error
	if errors.As(err, &gerr) && gerr.Code == 409 {
		// concurrent op; retry with backoff
	} else if !isNotFound(err) {
		return fmt.Errorf("deleting MIG %s: %w", igm.SelfLink, err)
	}
}

When it happens

Trigger: Calling DeleteInstanceGroupManager (via DeleteCloudGroup during cluster/instance-group teardown) where InstanceGroupManagers().Delete(project, zone, name) returns a non-NotFound error: e.g. the MIG still has instances being deleted by another operation, insufficient compute.instanceGroupManagerManager permissions, project/quota issues, or transient API 5xx errors.

Common situations: kOps cluster deletion racing with a GCE deletion operation already in flight on the same MIG ('Failed to delete instance group manager: operation already in progress'); service account lacking compute.instanceGroupManagerAdmin or compute.instanceAdmin roles; stale MIG selfLinks after a previous partial delete; regional-vs-zonal MIG mismatches; intermittent GCE API 500/503s during large cluster teardown.

Related errors

• error resizing InstanceGroupManager %s to 0: %w
• error deleting Instance %s: %w
• error listing ManagedInstances in %s: %w
• error deleting InstanceTemplate %s: %w
• error deleting Subnetwork %s: %v

AI-assisted analysis of kubernetes/kops@4c8573c808 (2026-09-05). Data as JSON: /api/errors/c1fdaa93a36ccbaf. Report an issue: GitHub.