kubernetes/kops · error

error deleting node: %v

Error message

error deleting node: %v

What it means

deleteNode issues a CoreV1 Nodes().Delete against the Kubernetes API after the instance is terminated; any non-NotFound failure is wrapped as 'error deleting node'. This error means the Node object could not be removed from cluster state even though the underlying instance is (or will be) gone.

Source

Thrown at pkg/instancegroups/instancegroups.go:758

	if c.PostDrainDelay > 0 {
		klog.Infof("Waiting for %s for pods to stabilize after draining.", c.PostDrainDelay)
		time.Sleep(c.PostDrainDelay)
	}

	return nil
}

// deleteNode deletes a node from the k8s API.  It does not delete the underlying instance.
func (c *RollingUpdateCluster) deleteNode(ctx context.Context, node *corev1.Node) error {
	var options metav1.DeleteOptions
	err := c.K8sClient.CoreV1().Nodes().Delete(ctx, node.Name, options)
	if err != nil {
		if apierrors.IsNotFound(err) {
			return nil
		}

		return fmt.Errorf("error deleting node: %v", err)
	}

	return nil
}

// UpdateSingleInstance performs a rolling update on a single instance
func (c *RollingUpdateCluster) UpdateSingleInstance(ctx context.Context, cloudMember *cloudinstances.CloudInstance, detach bool) error {
	if detach {
		if cloudMember.CloudInstanceGroup.InstanceGroup.IsControlPlane() {
			klog.Warning("cannot detach control-plane instances. Assuming --surge=false")
		} else if cloudMember.CloudInstanceGroup.InstanceGroup.Spec.Manager != api.InstanceManagerKarpenter {
			err := c.detachInstance(cloudMember)
			if err != nil {
				return fmt.Errorf("failed to detach instance: %v", err)
			}
			if err := c.maybeValidate(" after detaching instance", c.ValidateCount, cloudMember.CloudInstanceGroup); err != nil {
				return err
			}

View on GitHub (pinned to 4c8573c808)

Solutions

1. Manually remove the stale object: `kubectl delete node <name> --force` (or retry the rolling update, which re-derives the node list)
2. Verify RBAC allows deleting nodes for the kOps credentials
3. Check API server health (`kubectl get --raw /readyz`) and retry once connectivity is restored
4. Upgrade kOps if finalizer-related conflicts persist on your Kubernetes version

Example fix

// before
NAME                     STATUS
ip-10-0-1-23             NotReady,SchedulingDisabled
// after (manual cleanup)
kubectl delete node ip-10-0-1-23

if _, err := client.CoreV1().Nodes().Get(ctx, nodeName, metav1.GetOptions{}); apierrors.IsNotFound(err) {
    return nil // already gone, delete will be a no-op
}

if err := c.deleteNode(ctx, node); err != nil {
    if apierrors.IsNotFound(err) {
        return nil
    }
    klog.Warningf("node delete failed: %v; cleaning up manually", err)
    _ = client.CoreV1().Nodes().Delete(ctx, node.Name, metav1.DeleteOptions{})
}

When it happens

Trigger: The Nodes().Delete call fails with non-NotFound errors: RBAC denial on nodes delete, API server unreachable, or a conflict from finalizers/other controllers mutating the node object concurrently.

Common situations: Cloud-controller-manager or node-lifecycle-controller recreating/locking node objects mid-delete; kOps user lacking delete permission on nodes; transient API server outage during a large rolling update.

Related errors

• error adding needs-update label: %v
• error patching needs-update label: %v
• error listing nodes in cluster: %v
• reading instance groups: %w
• failed to taint node %q: %v

AI-assisted analysis of kubernetes/kops@4c8573c808 (2026-09-05). Data as JSON: /api/errors/3d97e1f0cfe772e0. Report an issue: GitHub.