{"record":{"id":"61543900a9efdc33","repo":"kubernetes/kops","slug":"failed-to-delete-warm-pool-instance-q-w","errorCode":null,"errorMessage":"failed to delete warm pool instance %q: %w","messagePattern":"failed to delete warm pool instance %q: %w","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"pkg/instancegroups/instancegroups.go","lineNumber":161,"sourceCode":"\t} else if err = c.maybeValidate(\"\", 1, group); err != nil {\n\t\treturn err\n\t}\n\n\tif !c.CloudOnly {\n\t\terr = c.taintAllNeedUpdate(ctx, group, update)\n\t\tif err != nil {\n\t\t\treturn err\n\t\t}\n\t}\n\n\tnonWarmPool := []*cloudinstances.CloudInstance{}\n\t// Run through the warm pool and delete all instances directly\n\tfor _, instance := range update {\n\t\tif instance.State == cloudinstances.WarmPool {\n\t\t\tklog.Infof(\"deleting warm pool instance %q\", instance.ID)\n\t\t\terr := c.Cloud.DeleteInstance(instance)\n\t\t\tif err != nil {\n\t\t\t\treturn fmt.Errorf(\"failed to delete warm pool instance %q: %w\", instance.ID, err)\n\t\t\t}\n\t\t} else {\n\t\t\tnonWarmPool = append(nonWarmPool, instance)\n\t\t}\n\t}\n\tupdate = nonWarmPool\n\n\tsettings := resolveSettings(c.Cluster, group.InstanceGroup, numInstances)\n\n\trunningDrains := 0\n\tmaxSurge := settings.MaxSurge.IntValue()\n\n\tif maxSurge > len(update) {\n\t\tmaxSurge = len(update)\n\t}\n\n\tmaxConcurrency := maxSurge + settings.MaxUnavailable.IntValue()\n","sourceCodeStart":143,"sourceCodeEnd":179,"githubUrl":"https://github.com/kubernetes/kops/blob/4c8573c808a73d578c5eadc86d410646ea0b0d73/pkg/instancegroups/instancegroups.go#L143-L179","documentation":"During a rolling update, instances in the WarmPool state are deleted directly via Cloud.DeleteInstance before the normal cordon/drain flow. If the cloud provider rejects or fails the delete, the instance ID and underlying error are wrapped and the rolling update aborts for this group.","triggerScenarios":"An instance with State==WarmPool whose Cloud.DeleteInstance call fails — e.g. instance already terminated, IAM lacks ec2:TerminateInstances, or API throttling during `kops rolling-update cluster` with warm pools enabled.","commonSituations":"Warm pool instances terminated out-of-band, insufficient permissions on the kops IAM role, or AWS rate limiting while deleting many warm instances at once.","solutions":["Inspect the wrapped provider error; if the instance is already gone, retry — the state may reconcile","Grant the cloud IAM role terminate/delete instance permissions","Retry the rolling update after transient API errors subside"],"exampleFix":null,"handlingStrategy":"retry","validationCode":"// Pre-check warm pool instances are still live before rolling update\nfor _, inst := range group.NeedUpdate {\n\tif inst.State == cloudinstances.WarmPool {\n\t\tif _, err := cloud.DescribeInstance(inst.ID); err != nil {\n\t\t\tlog.Printf(\"warm pool instance %s may be gone: %v\", inst.ID, err)\n\t\t}\n\t}\n}","typeGuard":null,"tryCatchPattern":"err := c.RollingUpdate(ctx, groups, &k8sClients)\nif err != nil {\n\tvar wpe *WarmPoolError\n\tif strings.Contains(err.Error(), \"failed to delete warm pool instance\") && isTransient(err) {\n\t\ttime.Sleep(15 * time.Second)\n\t\terr = c.RollingUpdate(ctx, groups, &k8sClients)\n\t}\n\t_ = wpe\n\treturn err\n}","preventionTips":["Grant the IAM role ec2:TerminateInstances (or provider equivalent) for warm pool management","Reconcile warm pool state if instances were terminated out-of-band","Handle throttling with backoff since warm pool deletes can burst"],"tags":["rolling-update","warm-pool","cloud-api","instance-deletion"],"backgroundTag":"warm-pool-instance-delete-failed","analyzedSha":"4c8573c808a73d578c5eadc86d410646ea0b0d73","analyzedAt":"2026-09-05T04:13:19.212Z","contentChangedAt":"2026-09-05T04:13:19.212Z","schemaVersion":2},"datasetVersion":"2026-09-12T12:17:11.808Z"}