What causes "failed to delete warm pool instance %q: %w" in kubernetes/kops?

Inspect the wrapped provider error; if the instance is already gone, retry — the state may reconcile

failed to delete warm pool instance %q: %w

Error message

failed to delete warm pool instance %q: %w

What it means

During a rolling update, instances in the WarmPool state are deleted directly via Cloud.DeleteInstance before the normal cordon/drain flow. If the cloud provider rejects or fails the delete, the instance ID and underlying error are wrapped and the rolling update aborts for this group.

Source

Thrown at pkg/instancegroups/instancegroups.go:161

	} else if err = c.maybeValidate("", 1, group); err != nil {
		return err
	}

	if !c.CloudOnly {
		err = c.taintAllNeedUpdate(ctx, group, update)
		if err != nil {
			return err
		}
	}

	nonWarmPool := []*cloudinstances.CloudInstance{}
	// Run through the warm pool and delete all instances directly
	for _, instance := range update {
		if instance.State == cloudinstances.WarmPool {
			klog.Infof("deleting warm pool instance %q", instance.ID)
			err := c.Cloud.DeleteInstance(instance)
			if err != nil {
				return fmt.Errorf("failed to delete warm pool instance %q: %w", instance.ID, err)
			}
		} else {
			nonWarmPool = append(nonWarmPool, instance)
		}
	}
	update = nonWarmPool

	settings := resolveSettings(c.Cluster, group.InstanceGroup, numInstances)

	runningDrains := 0
	maxSurge := settings.MaxSurge.IntValue()

	if maxSurge > len(update) {
		maxSurge = len(update)
	}

	maxConcurrency := maxSurge + settings.MaxUnavailable.IntValue()

View on GitHub (pinned to 4c8573c808)

Solutions

Inspect the wrapped provider error; if the instance is already gone, retry — the state may reconcile
Grant the cloud IAM role terminate/delete instance permissions
Retry the rolling update after transient API errors subside

Defensive patterns

Strategy: retry

Validate before calling

// Pre-check warm pool instances are still live before rolling update
for _, inst := range group.NeedUpdate {
	if inst.State == cloudinstances.WarmPool {
		if _, err := cloud.DescribeInstance(inst.ID); err != nil {
			log.Printf("warm pool instance %s may be gone: %v", inst.ID, err)
		}
	}
}

Try / catch

err := c.RollingUpdate(ctx, groups, &k8sClients)
if err != nil {
	var wpe *WarmPoolError
	if strings.Contains(err.Error(), "failed to delete warm pool instance") && isTransient(err) {
		time.Sleep(15 * time.Second)
		err = c.RollingUpdate(ctx, groups, &k8sClients)
	}
	_ = wpe
	return err
}

Prevention

Grant the IAM role ec2:TerminateInstances (or provider equivalent) for warm pool management
Reconcile warm pool state if instances were terminated out-of-band
Handle throttling with backoff since warm pool deletes can burst

When it happens

Trigger: An instance with State==WarmPool whose Cloud.DeleteInstance call fails — e.g. instance already terminated, IAM lacks ec2:TerminateInstances, or API throttling during `kops rolling-update cluster` with warm pools enabled.

Common situations: Warm pool instances terminated out-of-band, insufficient permissions on the kops IAM role, or AWS rate limiting while deleting many warm instances at once.

Related errors

AI-assisted analysis of kubernetes/kops@4c8573c808 (2026-09-05). Data as JSON: /api/errors/61543900a9efdc33. Report an issue: GitHub.