What causes "error draining node: %v" in kubernetes/kops?

Inspect PDBs: `kubectl get pdb -A` and temporarily raise maxUnavailable or delete the blocking PDB

error draining node: %v

Error message

error draining node: %v

What it means

drainNode invokes k8s.io/kubectl drain.RunNodeDrain to evict pods; any non-NotFound error is wrapped as 'error draining node'. This typically means pod eviction requests failed — e.g. pods without a controller, unmanaged DaemonSet pods, or PodDisruptionBudgets preventing eviction within the timeout.

Source

Thrown at pkg/instancegroups/instancegroups.go:738

				klog.Infof("skipping deregistration of instance %q, as part of instancegroup with role %q", u.ID, role)
				shouldDeregister = false
			}
		}
	}

	if shouldDeregister {
		if err := c.Cloud.DeregisterInstance(u); err != nil {
			return &DeregisterError{
				err: fmt.Errorf("error deregistering instance %q, node %q: %w", u.ID, u.Node.Name, err),
			}
		}
	}

	if err := drain.RunNodeDrain(helper, u.Node.Name); err != nil {
		if apierrors.IsNotFound(err) {
			return nil
		}
		return fmt.Errorf("error draining node: %v", err)
	}

	if c.PostDrainDelay > 0 {
		klog.Infof("Waiting for %s for pods to stabilize after draining.", c.PostDrainDelay)
		time.Sleep(c.PostDrainDelay)
	}

	return nil
}

// deleteNode deletes a node from the k8s API.  It does not delete the underlying instance.
func (c *RollingUpdateCluster) deleteNode(ctx context.Context, node *corev1.Node) error {
	var options metav1.DeleteOptions
	err := c.K8sClient.CoreV1().Nodes().Delete(ctx, node.Name, options)
	if err != nil {
		if apierrors.IsNotFound(err) {
			return nil
		}

View on GitHub (pinned to 4c8573c808)

Solutions

Inspect PDBs: `kubectl get pdb -A` and temporarily raise maxUnavailable or delete the blocking PDB
Increase drain timeouts or add `--post-drain-delay` and retry the rolling update
Force with `kops rolling-update cluster --cloudonly` if eviction is not required and instances can be terminated directly
Fix the workload: ensure replicas > 1 and controllers own the pods so they reschedule on other nodes

Example fix

// before
maxUnavailable: 0
// after
maxUnavailable: 1

Defensive patterns

Strategy: try-catch

Validate before calling

pdbs, _ := client.PolicyV1().PodDisruptionBudgets("").List(ctx, metav1.ListOptions{})
for _, p := range pdbs.Items {
    if p.Status.DisruptionsAllowed < 1 {
        klog.Warningf("PDB %s/%s blocks drain", p.Namespace, p.Name)
    }
}

Try / catch

if err := c.drainNode(ctx, u); err != nil {
    var ev *evictionError
    if errors.As(err, &ev) {
        klog.Warningf("drain blocked by PDB: %v", ev)
        // relax PDB or use --cloudonly
    }
    return err
}

Prevention

Audit PDBs (kubectl get pdb -A) before rolling updates
Ensure all pods are owned by controllers with replicas >= 2
Add --post-drain-delay to give pods time to stabilize

When it happens

Trigger: RunNodeDrain fails because eviction API calls time out due to PDBs (allowed disruptions=0), pods with emptyDir local storage cannot be evicted, or pods are in CrashLoopBackOff and never terminate within drain timeout.

Common situations: Rolling update hangs/fails on clusters with strict PodDisruptionBudgets on system components (e.g. kube-dns, metrics-server); workloads with `podAntiAffinity` making rescheduling impossible; single-replica statefulsets.

Related errors

AI-assisted analysis of kubernetes/kops@4c8573c808 (2026-09-05). Data as JSON: /api/errors/0907fdcb21803c8a. Report an issue: GitHub.