{"record":{"id":"3d97e1f0cfe772e0","repo":"kubernetes/kops","slug":"error-deleting-node-v","errorCode":null,"errorMessage":"error deleting node: %v","messagePattern":"error deleting node: (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"pkg/instancegroups/instancegroups.go","lineNumber":758,"sourceCode":"\n\tif c.PostDrainDelay > 0 {\n\t\tklog.Infof(\"Waiting for %s for pods to stabilize after draining.\", c.PostDrainDelay)\n\t\ttime.Sleep(c.PostDrainDelay)\n\t}\n\n\treturn nil\n}\n\n// deleteNode deletes a node from the k8s API.  It does not delete the underlying instance.\nfunc (c *RollingUpdateCluster) deleteNode(ctx context.Context, node *corev1.Node) error {\n\tvar options metav1.DeleteOptions\n\terr := c.K8sClient.CoreV1().Nodes().Delete(ctx, node.Name, options)\n\tif err != nil {\n\t\tif apierrors.IsNotFound(err) {\n\t\t\treturn nil\n\t\t}\n\n\t\treturn fmt.Errorf(\"error deleting node: %v\", err)\n\t}\n\n\treturn nil\n}\n\n// UpdateSingleInstance performs a rolling update on a single instance\nfunc (c *RollingUpdateCluster) UpdateSingleInstance(ctx context.Context, cloudMember *cloudinstances.CloudInstance, detach bool) error {\n\tif detach {\n\t\tif cloudMember.CloudInstanceGroup.InstanceGroup.IsControlPlane() {\n\t\t\tklog.Warning(\"cannot detach control-plane instances. Assuming --surge=false\")\n\t\t} else if cloudMember.CloudInstanceGroup.InstanceGroup.Spec.Manager != api.InstanceManagerKarpenter {\n\t\t\terr := c.detachInstance(cloudMember)\n\t\t\tif err != nil {\n\t\t\t\treturn fmt.Errorf(\"failed to detach instance: %v\", err)\n\t\t\t}\n\t\t\tif err := c.maybeValidate(\" after detaching instance\", c.ValidateCount, cloudMember.CloudInstanceGroup); err != nil {\n\t\t\t\treturn err\n\t\t\t}","sourceCodeStart":740,"sourceCodeEnd":776,"githubUrl":"https://github.com/kubernetes/kops/blob/4c8573c808a73d578c5eadc86d410646ea0b0d73/pkg/instancegroups/instancegroups.go#L740-L776","documentation":"deleteNode issues a CoreV1 Nodes().Delete against the Kubernetes API after the instance is terminated; any non-NotFound failure is wrapped as 'error deleting node'. This error means the Node object could not be removed from cluster state even though the underlying instance is (or will be) gone.","triggerScenarios":"The Nodes().Delete call fails with non-NotFound errors: RBAC denial on nodes delete, API server unreachable, or a conflict from finalizers/other controllers mutating the node object concurrently.","commonSituations":"Cloud-controller-manager or node-lifecycle-controller recreating/locking node objects mid-delete; kOps user lacking delete permission on nodes; transient API server outage during a large rolling update.","solutions":["Manually remove the stale object: `kubectl delete node <name> --force` (or retry the rolling update, which re-derives the node list)","Verify RBAC allows deleting nodes for the kOps credentials","Check API server health (`kubectl get --raw /readyz`) and retry once connectivity is restored","Upgrade kOps if finalizer-related conflicts persist on your Kubernetes version"],"exampleFix":"// before\nNAME                     STATUS\nip-10-0-1-23             NotReady,SchedulingDisabled\n// after (manual cleanup)\nkubectl delete node ip-10-0-1-23","handlingStrategy":"try-catch","validationCode":"if _, err := client.CoreV1().Nodes().Get(ctx, nodeName, metav1.GetOptions{}); apierrors.IsNotFound(err) {\n    return nil // already gone, delete will be a no-op\n}","typeGuard":null,"tryCatchPattern":"if err := c.deleteNode(ctx, node); err != nil {\n    if apierrors.IsNotFound(err) {\n        return nil\n    }\n    klog.Warningf(\"node delete failed: %v; cleaning up manually\", err)\n    _ = client.CoreV1().Nodes().Delete(ctx, node.Name, metav1.DeleteOptions{})\n}","preventionTips":["Grant node delete RBAC to kOps credentials","Verify API server health before/while updating","Delete stale Node objects after out-of-band instance termination"],"tags":["kubernetes","node-delete","api","rolling-update"],"backgroundTag":"kubernetes-api-request-failed","analyzedSha":"4c8573c808a73d578c5eadc86d410646ea0b0d73","analyzedAt":"2026-09-05T04:13:19.212Z","contentChangedAt":"2026-09-05T04:13:19.212Z","schemaVersion":2},"datasetVersion":"2026-09-12T07:17:12.445Z"}