kubernetes/kops · error

error cordoning node: %v

Error message

error cordoning node: %v

What it means

After marking the node with a cordon (drain.RunCordonOrUncordon with wantUncordon=false), drainNode wraps any non-NotFound error as 'error cordoning node'. This means the k8s API rejected or failed the PATCH that sets spec.unschedulable=true on the Node.

Source

Thrown at pkg/instancegroups/instancegroups.go:704

		Force:               true,
		GracePeriodSeconds:  -1,
		IgnoreAllDaemonSets: true,
		Out:                 os.Stdout,
		ErrOut:              os.Stderr,
		Timeout:             c.DrainTimeout,

		// The zero value would retry evictions without any delay
		EvictErrorRetryDelay: 5 * time.Second,

		// We want to proceed even when pods are using emptyDir volumes
		DeleteEmptyDirData: true,
	}

	if err := drain.RunCordonOrUncordon(helper, u.Node, true); err != nil {
		if apierrors.IsNotFound(err) {
			return nil
		}
		return fmt.Errorf("error cordoning node: %v", err)
	}

	if err := c.patchExcludeFromLB(ctx, u.Node); err != nil {
		if apierrors.IsNotFound(err) {
			return nil
		}
		return fmt.Errorf("error excluding node from load balancer: %v", err)
	}

	shouldDeregister := true
	if !c.Options.DeregisterControlPlaneNodes {
		if u.CloudInstanceGroup != nil && u.CloudInstanceGroup.InstanceGroup != nil {
			role := u.CloudInstanceGroup.InstanceGroup.Spec.Role
			switch role {
			case api.InstanceGroupRoleAPIServer, api.InstanceGroupRoleControlPlane:
				klog.Infof("skipping deregistration of instance %q, as part of instancegroup with role %q", u.ID, role)
				shouldDeregister = false
			}

View on GitHub (pinned to 4c8573c808)

Solutions

1. Confirm the credentials used by kOps have RBAC rights to patch nodes (system:node / cluster-admin style binding)
2. Re-run the rolling update — transient API errors are retriable and NotFound is treated as success
3. Check `kubectl get node <name> -o yaml` to confirm the node exists and inspect its conditions
4. Verify connectivity to the cluster API endpoint from the machine running kOps

Example fix

// before
kubectl auth can-i patch nodes --as=system:serviceaccount:kube-system:kops
// after (grant RBAC)
kubectl create clusterrolebinding kops-rolling --clusterrole=cluster-admin --user=<kops-user>

if !rbac.CanIPatch("nodes") {
    return errors.New("credentials cannot patch nodes; cordon will fail")
}

err := c.drainNode(ctx, u)
var notFound bool
if apierrors.IsNotFound(errors.Unwrap(err)) {
    notFound = true
}
if err != nil && !notFound {
    klog.Warningf("cordon failed (%v); retrying once", err)
    err = c.drainNode(ctx, u)
}

When it happens

Trigger: The cordon PATCH fails with any error other than NotFound: RBAC denial on nodes update, API server unreachable, conflict/timeout on the node object, or a stale Node object that was deleted mid-request.

Common situations: kOps service account lacking clusterrole permissions for nodes during `kops rolling-update cluster`; API server briefly unavailable while an LB is being switched; concurrent controllers fighting over node spec.

Related errors

• error adding needs-update label: %v
• error applying annotation to record addon installation: %v
• error patching needs-update label: %v
• error querying namespace %q: %v
• error applying annotation to namespace: %v

AI-assisted analysis of kubernetes/kops@4c8573c808 (2026-09-05). Data as JSON: /api/errors/d5f17fcf576bea29. Report an issue: GitHub.