kubernetes/kops · error

timed out waiting for instances in MIG %q to terminate

Error message

timed out waiting for instances in MIG %q to terminate

What it means

DeleteCloudInstanceGroup waits (up to a hard 10-minute deadline) for all VMs in a GCE managed instance group (MIG) to terminate after deletion. If instances still exist after the timeout, it returns this error rather than blocking forever, leaving the caller (DeleteGroup) with a MIG that did not fully drain.

Source

Thrown at upup/pkg/fi/cloudup/gce/instancegroups.go:57

// DeleteGroup deletes a cloud of instances controlled by an Instance Group Manager
func (c *gceCloudImplementation) DeleteGroup(g *cloudinstances.CloudInstanceGroup) error {
	return DeleteCloudInstanceGroup(c, g)
}

// DeleteCloudInstanceGroup deletes the InstanceGroupManager and current InstanceTemplate
func DeleteCloudInstanceGroup(c GCECloud, g *cloudinstances.CloudInstanceGroup) error {
	mig := g.Raw.(*compute.InstanceGroupManager)
	err := DeleteMIGInstances(c, mig)
	if err != nil {
		return err
	}

	timeout := time.Now().Add(10 * time.Minute)

	klog.Infof("Waiting for instances in MIG %q to terminate...", mig.Name)
	for {
		if time.Now().After(timeout) {
			return fmt.Errorf("timed out waiting for instances in MIG %q to terminate", mig.Name)
		}

		instances, err := ListManagedInstances(c, mig)
		if err != nil {
			return fmt.Errorf("error listing managed instances for %q: %v", mig.Name, err)
		}
		if len(instances) == 0 {
			klog.Infof("All instances in MIG %q terminated", mig.Name)
			break
		}
		klog.Infof("%d instance(s) remaining in MIG %q, waiting...", len(instances), mig.Name)
		time.Sleep(PollingInterval)
	}

	err = DeleteInstanceGroupManager(c, mig)
	if err != nil {
		return err
	}

View on GitHub (pinned to 4c8573c808)

Solutions

1. Inspect the MIG and its instances in the GCE console / `gcloud compute instance-groups managed list-instances` to see which instances remain and why.
2. Delete the stuck instances explicitly (`gcloud compute instances delete`), then retry DeleteGroup.
3. Check for hung shutdown scripts or disk detach operations; fix/kill them and wait for termination.
4. Retry the deletion once the transient GCE operation backlog clears.
5. As a last resort, abandon instances from the MIG and delete the MIG manually before re-running kops delete.

Example fix

// before: retry immediately and hit the same 10m timeout
err := cloud.DeleteCloudInstanceGroup(g)
// after: ensure instances are gone/stuck ones force-deleted first, then retry
// gcloud compute instances delete <stuck-instance> --zone <zone> --quiet
err := cloud.DeleteCloudInstanceGroup(g)

// check MIG instance states before attempting delete so you know what you're waiting on
mgr, _ := svc.InstanceGroupManagers.Get(project, zone, migName).Context(ctx).Do()
instances, _ := svc.InstanceGroupManagers.ListInstances(project, zone, migName, "").Context(ctx).Do()
if instances != nil && len(instances.Items) == 0 {
	// nothing to wait for
}

err := cloud.DeleteCloudInstanceGroup(mig)
if err != nil && strings.Contains(err.Error(), "timed out waiting for instances in MIG") {
	// inspect stuck instances, force-delete them, then retry once
	return retryAfterCleanup(mig)
}

When it happens

Trigger: Calling DeleteGroup on a GCE instance group where the MIG's instances remain in STOPPING/deleting state for more than 10 minutes, e.g. large instance count, instances stuck shutting down, or deletion requests not actually accepted by the API while ListManagedInstances keeps returning instances.

Common situations: kops delete cluster / delete instancegroup against a GCE cluster where VMs are slow to terminate, GCE operations are queued, or an instance is stuck in a stopping state (disk detach, shutdown script hanging).

Understand the failure class

• Timeouts: ETIMEDOUT, deadlines, and hung requests — what actually expires when a request times out.

Related errors

• error listing managed instances for %q: %v
• cannot find owner for instance %s
• instance %s did not have Version set
• ig name not set on instance template %s
• error fetching GCE managed instance group %q: %v

AI-assisted analysis of kubernetes/kops@4c8573c808 (2026-09-05). Data as JSON: /api/errors/f7df0b79eb108ed5. Report an issue: GitHub.