{"record":{"id":"851b635ec820a729","repo":"kubernetes/kops","slug":"failed-to-drain-node-q-w","errorCode":null,"errorMessage":"failed to drain node %q: %w","messagePattern":"failed to drain node %q: %w","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"pkg/instancegroups/instancegroups.go","lineNumber":457,"sourceCode":"\n\tnodeName := \"\"\n\tif u.Node != nil {\n\t\tnodeName = u.Node.Name\n\t}\n\n\tisBastion := u.CloudInstanceGroup.InstanceGroup.IsBastion()\n\n\tif isBastion {\n\t\t// We don't want to validate for bastions - they aren't part of the cluster\n\t} else if c.CloudOnly {\n\t\tklog.Warning(\"Not draining cluster nodes as 'cloudonly' flag is set.\")\n\t} else {\n\t\tif u.Node != nil {\n\t\t\tklog.Infof(\"Draining the node: %q.\", nodeName)\n\n\t\t\tif err := c.drainNode(ctx, u); err != nil {\n\t\t\t\tif c.FailOnDrainError {\n\t\t\t\t\treturn fmt.Errorf(\"failed to drain node %q: %w\", nodeName, err)\n\t\t\t\t}\n\t\t\t\tklog.Infof(\"Ignoring error draining node %q: %v\", nodeName, err)\n\t\t\t}\n\t\t} else {\n\t\t\tklog.Warningf(\"Skipping drain of instance %q, because it is not registered in kubernetes\", instanceID)\n\t\t}\n\t}\n\n\t// GCE often re-uses names, so we delete the node object to prevent the new instance from using the cordoned Node object\n\t// Scaleway has the same behavior\n\tif (c.Cluster.GetCloudProvider() == api.CloudProviderGCE || c.Cluster.GetCloudProvider() == api.CloudProviderScaleway) &&\n\t\t!isBastion && !c.CloudOnly {\n\t\tif u.Node == nil {\n\t\t\tklog.Warningf(\"no kubernetes Node associated with %s, skipping node deletion\", instanceID)\n\t\t} else {\n\t\t\tklog.Infof(\"deleting node %q from kubernetes\", nodeName)\n\t\t\tif err := c.deleteNode(ctx, u.Node); err != nil {\n\t\t\t\treturn fmt.Errorf(\"error deleting node %q: %v\", nodeName, err)","sourceCodeStart":439,"sourceCodeEnd":475,"githubUrl":"https://github.com/kubernetes/kops/blob/4c8573c808a73d578c5eadc86d410646ea0b0d73/pkg/instancegroups/instancegroups.go#L439-L475","documentation":"Before terminating an instance during a rolling update, kOps drains its Kubernetes node (evicting pods). This error wraps drainNode failure and is returned only when FailOnDrainError is true; otherwise it is logged and the roll proceeds to delete the instance anyway.","triggerScenarios":"c.drainNode(ctx, u) fails while FailOnDrainError is set: pod eviction blocked by PodDisruptionBudgets, node NotReady, API server errors, or K8sClient not configured (surfaces 'K8sClient not set').","commonSituations":"PDBs preventing eviction of critical pods; node already NotReady/deleted; long-running jobs without proper eviction grace; cluster unreachable from the kops host.","solutions":["Fix blocking PodDisruptionBudgets or temporarily allow evictions","Check node health (kubectl describe node) and evict stuck pods manually","Retry the rolling update after the API server is reachable","Only enable FailOnDrainError once drain reliability is confirmed; otherwise the error is logged and roll continues"],"exampleFix":"// before\nkubectl create pdb --min-available=100% ...   # blocks all evictions\n// after\nkubectl patch pdb my-pdb -p '{\"spec\":{\"minAvailable\":1}}'  # allow one eviction during roll","handlingStrategy":"retry","validationCode":"// pre-flight: ensure evictable workloads\npdbs, _ := clientset.PolicyV1().PodDisruptionBudgets(ns).List(ctx, metav1.ListOptions{})\n// verify disruptionsAllowed > 0 for PDBs targeting pods on the node","typeGuard":"func canDrain(u *cloudinstances.CloudInstance) bool { return u != nil && u.Node != nil }","tryCatchPattern":"if err := c.drainNode(ctx, u); err != nil {\n    if c.FailOnDrainError {\n        return fmt.Errorf(\"failed to drain node %q: %w\", nodeName, err)\n    }\n    klog.Infof(\"Ignoring error draining node %q: %v\", nodeName, err)\n}","preventionTips":["Audit PodDisruptionBudgets so eviction is always allowed for at least one replica","Set --drain-timeout appropriately and pre-check node health","Use cordon+taint dry-runs before production rolls","Enable FailOnDrainError only once drains are reliable"],"tags":["kubernetes","drain","rolling-update","pdb"],"backgroundTag":"node-drain-failed","analyzedSha":"4c8573c808a73d578c5eadc86d410646ea0b0d73","analyzedAt":"2026-09-05T04:13:19.212Z","contentChangedAt":"2026-09-05T04:13:19.212Z","schemaVersion":2},"datasetVersion":"2026-09-12T07:17:12.445Z"}