{"record":{"id":"0907fdcb21803c8a","repo":"kubernetes/kops","slug":"error-draining-node-v","errorCode":null,"errorMessage":"error draining node: %v","messagePattern":"error draining node: (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"pkg/instancegroups/instancegroups.go","lineNumber":738,"sourceCode":"\t\t\t\tklog.Infof(\"skipping deregistration of instance %q, as part of instancegroup with role %q\", u.ID, role)\n\t\t\t\tshouldDeregister = false\n\t\t\t}\n\t\t}\n\t}\n\n\tif shouldDeregister {\n\t\tif err := c.Cloud.DeregisterInstance(u); err != nil {\n\t\t\treturn &DeregisterError{\n\t\t\t\terr: fmt.Errorf(\"error deregistering instance %q, node %q: %w\", u.ID, u.Node.Name, err),\n\t\t\t}\n\t\t}\n\t}\n\n\tif err := drain.RunNodeDrain(helper, u.Node.Name); err != nil {\n\t\tif apierrors.IsNotFound(err) {\n\t\t\treturn nil\n\t\t}\n\t\treturn fmt.Errorf(\"error draining node: %v\", err)\n\t}\n\n\tif c.PostDrainDelay > 0 {\n\t\tklog.Infof(\"Waiting for %s for pods to stabilize after draining.\", c.PostDrainDelay)\n\t\ttime.Sleep(c.PostDrainDelay)\n\t}\n\n\treturn nil\n}\n\n// deleteNode deletes a node from the k8s API.  It does not delete the underlying instance.\nfunc (c *RollingUpdateCluster) deleteNode(ctx context.Context, node *corev1.Node) error {\n\tvar options metav1.DeleteOptions\n\terr := c.K8sClient.CoreV1().Nodes().Delete(ctx, node.Name, options)\n\tif err != nil {\n\t\tif apierrors.IsNotFound(err) {\n\t\t\treturn nil\n\t\t}","sourceCodeStart":720,"sourceCodeEnd":756,"githubUrl":"https://github.com/kubernetes/kops/blob/4c8573c808a73d578c5eadc86d410646ea0b0d73/pkg/instancegroups/instancegroups.go#L720-L756","documentation":"drainNode invokes k8s.io/kubectl drain.RunNodeDrain to evict pods; any non-NotFound error is wrapped as 'error draining node'. This typically means pod eviction requests failed — e.g. pods without a controller, unmanaged DaemonSet pods, or PodDisruptionBudgets preventing eviction within the timeout.","triggerScenarios":"RunNodeDrain fails because eviction API calls time out due to PDBs (allowed disruptions=0), pods with emptyDir local storage cannot be evicted, or pods are in CrashLoopBackOff and never terminate within drain timeout.","commonSituations":"Rolling update hangs/fails on clusters with strict PodDisruptionBudgets on system components (e.g. kube-dns, metrics-server); workloads with `podAntiAffinity` making rescheduling impossible; single-replica statefulsets.","solutions":["Inspect PDBs: `kubectl get pdb -A` and temporarily raise maxUnavailable or delete the blocking PDB","Increase drain timeouts or add `--post-drain-delay` and retry the rolling update","Force with `kops rolling-update cluster --cloudonly` if eviction is not required and instances can be terminated directly","Fix the workload: ensure replicas > 1 and controllers own the pods so they reschedule on other nodes"],"exampleFix":"// before\nmaxUnavailable: 0\n// after\nmaxUnavailable: 1","handlingStrategy":"try-catch","validationCode":"pdbs, _ := client.PolicyV1().PodDisruptionBudgets(\"\").List(ctx, metav1.ListOptions{})\nfor _, p := range pdbs.Items {\n    if p.Status.DisruptionsAllowed < 1 {\n        klog.Warningf(\"PDB %s/%s blocks drain\", p.Namespace, p.Name)\n    }\n}","typeGuard":null,"tryCatchPattern":"if err := c.drainNode(ctx, u); err != nil {\n    var ev *evictionError\n    if errors.As(err, &ev) {\n        klog.Warningf(\"drain blocked by PDB: %v\", ev)\n        // relax PDB or use --cloudonly\n    }\n    return err\n}","preventionTips":["Audit PDBs (kubectl get pdb -A) before rolling updates","Ensure all pods are owned by controllers with replicas >= 2","Add --post-drain-delay to give pods time to stabilize"],"tags":["kubernetes","drain","eviction","pod-disruption-budget"],"backgroundTag":"pod-eviction-blocked","analyzedSha":"4c8573c808a73d578c5eadc86d410646ea0b0d73","analyzedAt":"2026-09-05T04:13:19.212Z","contentChangedAt":"2026-09-05T04:13:19.212Z","schemaVersion":2},"datasetVersion":"2026-09-12T07:17:12.445Z"}