kubernetes/kops · error

error deleting node %q: %v

Error message

error deleting node %q: %v

What it means

After draining and terminating an instance, kOps deletes the corresponding Node object from the Kubernetes API. This error wraps a deleteNode failure and is always fatal to the rolling update of that instance, unlike drain/taint errors which can be ignored.

Source

Thrown at pkg/instancegroups/instancegroups.go:475

					return fmt.Errorf("failed to drain node %q: %w", nodeName, err)
				}
				klog.Infof("Ignoring error draining node %q: %v", nodeName, err)
			}
		} else {
			klog.Warningf("Skipping drain of instance %q, because it is not registered in kubernetes", instanceID)
		}
	}

	// GCE often re-uses names, so we delete the node object to prevent the new instance from using the cordoned Node object
	// Scaleway has the same behavior
	if (c.Cluster.GetCloudProvider() == api.CloudProviderGCE || c.Cluster.GetCloudProvider() == api.CloudProviderScaleway) &&
		!isBastion && !c.CloudOnly {
		if u.Node == nil {
			klog.Warningf("no kubernetes Node associated with %s, skipping node deletion", instanceID)
		} else {
			klog.Infof("deleting node %q from kubernetes", nodeName)
			if err := c.deleteNode(ctx, u.Node); err != nil {
				return fmt.Errorf("error deleting node %q: %v", nodeName, err)
			}
		}
	}

	if err := c.deleteInstance(u); err != nil {
		klog.Errorf("error deleting instance %q, node %q: %v", instanceID, nodeName, err)
		return err
	}

	if err := c.reconcileInstanceGroup(ctx); err != nil {
		klog.Errorf("error reconciling instance group %q: %v", u.CloudInstanceGroup.HumanName, err)
		return err
	}

	// Wait for the minimum interval
	klog.Infof("waiting for %v after terminating instance", sleepAfterTerminate)
	time.Sleep(sleepAfterTerminate)

View on GitHub (pinned to 4c8573c808)

Solutions

1. Retry the rolling update; the node object may already be gone
2. Check RBAC delete permission on nodes
3. Verify API server connectivity
4. Manually delete the stale node object (kubectl delete node <name>) then re-run

Example fix

// before
// stale node blocks the roll
// after
kubectl delete node <node-name> && kops rolling-update cluster --yes

// pre-check the node object exists and is deletable
_, err := clientset.CoreV1().Nodes().Get(ctx, nodeName, metav1.GetOptions{})
// err == nil and no deletingFinalizers => safe to attempt delete

func nodeExists(n *corev1.Node) bool { return n != nil && n.UID != "" }

if err := c.deleteNode(ctx, u.Node); err != nil {
    if k8serrors.IsNotFound(err) { return nil } // already deleted; treat as success
    return fmt.Errorf("error deleting node %q: %v", nodeName, err)
}

When it happens

Trigger: c.deleteNode(ctx, u.Node) returns an error: the Node object no longer exists (404 handled?), RBAC denies node deletion, or the API server call fails. u.Node is non-nil (registered node) so the deletion path is taken.

Common situations: Node already garbage-collected by the node controller causing a conflict; API server outage mid-roll; service account lacking delete permission on nodes.

Related errors

• error adding needs-update label: %v
• error patching needs-update label: %v
• error listing nodes in cluster: %v
• failed to taint node %q: %v
• failed to drain node %q: %w

AI-assisted analysis of kubernetes/kops@4c8573c808 (2026-09-05). Data as JSON: /api/errors/9d976d5ccb47cb1e. Report an issue: GitHub.