{"record":{"id":"f7df0b79eb108ed5","repo":"kubernetes/kops","slug":"timed-out-waiting-for-instances-in-mig-q-to-termi","errorCode":null,"errorMessage":"timed out waiting for instances in MIG %q to terminate","messagePattern":"timed out waiting for instances in MIG %q to terminate","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"upup/pkg/fi/cloudup/gce/instancegroups.go","lineNumber":57,"sourceCode":"// DeleteGroup deletes a cloud of instances controlled by an Instance Group Manager\nfunc (c *gceCloudImplementation) DeleteGroup(g *cloudinstances.CloudInstanceGroup) error {\n\treturn DeleteCloudInstanceGroup(c, g)\n}\n\n// DeleteCloudInstanceGroup deletes the InstanceGroupManager and current InstanceTemplate\nfunc DeleteCloudInstanceGroup(c GCECloud, g *cloudinstances.CloudInstanceGroup) error {\n\tmig := g.Raw.(*compute.InstanceGroupManager)\n\terr := DeleteMIGInstances(c, mig)\n\tif err != nil {\n\t\treturn err\n\t}\n\n\ttimeout := time.Now().Add(10 * time.Minute)\n\n\tklog.Infof(\"Waiting for instances in MIG %q to terminate...\", mig.Name)\n\tfor {\n\t\tif time.Now().After(timeout) {\n\t\t\treturn fmt.Errorf(\"timed out waiting for instances in MIG %q to terminate\", mig.Name)\n\t\t}\n\n\t\tinstances, err := ListManagedInstances(c, mig)\n\t\tif err != nil {\n\t\t\treturn fmt.Errorf(\"error listing managed instances for %q: %v\", mig.Name, err)\n\t\t}\n\t\tif len(instances) == 0 {\n\t\t\tklog.Infof(\"All instances in MIG %q terminated\", mig.Name)\n\t\t\tbreak\n\t\t}\n\t\tklog.Infof(\"%d instance(s) remaining in MIG %q, waiting...\", len(instances), mig.Name)\n\t\ttime.Sleep(PollingInterval)\n\t}\n\n\terr = DeleteInstanceGroupManager(c, mig)\n\tif err != nil {\n\t\treturn err\n\t}","sourceCodeStart":39,"sourceCodeEnd":75,"githubUrl":"https://github.com/kubernetes/kops/blob/4c8573c808a73d578c5eadc86d410646ea0b0d73/upup/pkg/fi/cloudup/gce/instancegroups.go#L39-L75","documentation":"DeleteCloudInstanceGroup waits (up to a hard 10-minute deadline) for all VMs in a GCE managed instance group (MIG) to terminate after deletion. If instances still exist after the timeout, it returns this error rather than blocking forever, leaving the caller (DeleteGroup) with a MIG that did not fully drain.","triggerScenarios":"Calling DeleteGroup on a GCE instance group where the MIG's instances remain in STOPPING/deleting state for more than 10 minutes, e.g. large instance count, instances stuck shutting down, or deletion requests not actually accepted by the API while ListManagedInstances keeps returning instances.","commonSituations":"kops delete cluster / delete instancegroup against a GCE cluster where VMs are slow to terminate, GCE operations are queued, or an instance is stuck in a stopping state (disk detach, shutdown script hanging).","solutions":["Inspect the MIG and its instances in the GCE console / `gcloud compute instance-groups managed list-instances` to see which instances remain and why.","Delete the stuck instances explicitly (`gcloud compute instances delete`), then retry DeleteGroup.","Check for hung shutdown scripts or disk detach operations; fix/kill them and wait for termination.","Retry the deletion once the transient GCE operation backlog clears.","As a last resort, abandon instances from the MIG and delete the MIG manually before re-running kops delete."],"exampleFix":"// before: retry immediately and hit the same 10m timeout\nerr := cloud.DeleteCloudInstanceGroup(g)\n// after: ensure instances are gone/stuck ones force-deleted first, then retry\n// gcloud compute instances delete <stuck-instance> --zone <zone> --quiet\nerr := cloud.DeleteCloudInstanceGroup(g)","handlingStrategy":"retry","validationCode":"// check MIG instance states before attempting delete so you know what you're waiting on\nmgr, _ := svc.InstanceGroupManagers.Get(project, zone, migName).Context(ctx).Do()\ninstances, _ := svc.InstanceGroupManagers.ListInstances(project, zone, migName, \"\").Context(ctx).Do()\nif instances != nil && len(instances.Items) == 0 {\n\t// nothing to wait for\n}","typeGuard":null,"tryCatchPattern":"err := cloud.DeleteCloudInstanceGroup(mig)\nif err != nil && strings.Contains(err.Error(), \"timed out waiting for instances in MIG\") {\n\t// inspect stuck instances, force-delete them, then retry once\n\treturn retryAfterCleanup(mig)\n}","preventionTips":["Drain and pre-delete unhealthy nodes before cluster teardown.","Avoid shutdown scripts that can hang instance termination.","Run `gcloud compute instance-groups managed list-instances` first to confirm the MIG is small enough to drain within the 10-minute window."],"tags":["gce","instance-groups","timeout","mig"],"backgroundTag":"deletion-timeout","analyzedSha":"4c8573c808a73d578c5eadc86d410646ea0b0d73","analyzedAt":"2026-09-05T04:13:19.212Z","contentChangedAt":"2026-09-05T04:13:19.212Z","schemaVersion":2},"datasetVersion":"2026-09-12T12:17:11.808Z"}