kubernetes/kops · error

failed to drain node %q: %w

Error message

failed to drain node %q: %w

What it means

Before terminating an instance during a rolling update, kOps drains its Kubernetes node (evicting pods). This error wraps drainNode failure and is returned only when FailOnDrainError is true; otherwise it is logged and the roll proceeds to delete the instance anyway.

Source

Thrown at pkg/instancegroups/instancegroups.go:457

	nodeName := ""
	if u.Node != nil {
		nodeName = u.Node.Name
	}

	isBastion := u.CloudInstanceGroup.InstanceGroup.IsBastion()

	if isBastion {
		// We don't want to validate for bastions - they aren't part of the cluster
	} else if c.CloudOnly {
		klog.Warning("Not draining cluster nodes as 'cloudonly' flag is set.")
	} else {
		if u.Node != nil {
			klog.Infof("Draining the node: %q.", nodeName)

			if err := c.drainNode(ctx, u); err != nil {
				if c.FailOnDrainError {
					return fmt.Errorf("failed to drain node %q: %w", nodeName, err)
				}
				klog.Infof("Ignoring error draining node %q: %v", nodeName, err)
			}
		} else {
			klog.Warningf("Skipping drain of instance %q, because it is not registered in kubernetes", instanceID)
		}
	}

	// GCE often re-uses names, so we delete the node object to prevent the new instance from using the cordoned Node object
	// Scaleway has the same behavior
	if (c.Cluster.GetCloudProvider() == api.CloudProviderGCE || c.Cluster.GetCloudProvider() == api.CloudProviderScaleway) &&
		!isBastion && !c.CloudOnly {
		if u.Node == nil {
			klog.Warningf("no kubernetes Node associated with %s, skipping node deletion", instanceID)
		} else {
			klog.Infof("deleting node %q from kubernetes", nodeName)
			if err := c.deleteNode(ctx, u.Node); err != nil {
				return fmt.Errorf("error deleting node %q: %v", nodeName, err)

View on GitHub (pinned to 4c8573c808)

Solutions

1. Fix blocking PodDisruptionBudgets or temporarily allow evictions
2. Check node health (kubectl describe node) and evict stuck pods manually
3. Retry the rolling update after the API server is reachable
4. Only enable FailOnDrainError once drain reliability is confirmed; otherwise the error is logged and roll continues

Example fix

// before
kubectl create pdb --min-available=100% ...   # blocks all evictions
// after
kubectl patch pdb my-pdb -p '{"spec":{"minAvailable":1}}'  # allow one eviction during roll

// pre-flight: ensure evictable workloads
pdbs, _ := clientset.PolicyV1().PodDisruptionBudgets(ns).List(ctx, metav1.ListOptions{})
// verify disruptionsAllowed > 0 for PDBs targeting pods on the node

func canDrain(u *cloudinstances.CloudInstance) bool { return u != nil && u.Node != nil }

if err := c.drainNode(ctx, u); err != nil {
    if c.FailOnDrainError {
        return fmt.Errorf("failed to drain node %q: %w", nodeName, err)
    }
    klog.Infof("Ignoring error draining node %q: %v", nodeName, err)
}

When it happens

Trigger: c.drainNode(ctx, u) fails while FailOnDrainError is set: pod eviction blocked by PodDisruptionBudgets, node NotReady, API server errors, or K8sClient not configured (surfaces 'K8sClient not set').

Common situations: PDBs preventing eviction of critical pods; node already NotReady/deleted; long-running jobs without proper eviction grace; cluster unreachable from the kops host.

Related errors

• node name not set
• error adding needs-update label: %v
• error patching needs-update label: %v
• error listing nodes in cluster: %v
• failed to taint node %q: %v

AI-assisted analysis of kubernetes/kops@4c8573c808 (2026-09-05). Data as JSON: /api/errors/851b635ec820a729. Report an issue: GitHub.