kubernetes/kops · error

failed to taint node %q: %v

Error message

failed to taint node %q: %v

What it means

During a rolling update, kOps taints Kubernetes nodes that need updating so no new workloads are scheduled there. This error wraps a failure from patching the node's taint via the Kubernetes API when FailOnDrainError is enabled; otherwise the error is only logged and rolling update continues.

Source

Thrown at pkg/instancegroups/instancegroups.go:368

				if taint.Key == rollingUpdateTaintKey {
					foundTaint = true
				}
			}
			if !foundTaint {
				toTaint = append(toTaint, u.Node)
			}
		}
	}
	if len(toTaint) > 0 {
		noun := "nodes"
		if len(toTaint) == 1 {
			noun = "node"
		}
		klog.Infof("Tainting %d %s in %q instancegroup.", len(toTaint), noun, group.InstanceGroup.Name)
		for _, n := range toTaint {
			if err := c.patchTaint(ctx, n); err != nil {
				if c.FailOnDrainError {
					return fmt.Errorf("failed to taint node %q: %v", n, err)
				}
				klog.Infof("Ignoring error tainting node %q: %v", n, err)
			}
		}
	}
	return nil
}

func (c *RollingUpdateCluster) patchTaint(ctx context.Context, node *corev1.Node) error {
	oldData, err := json.Marshal(node)
	if err != nil {
		return err
	}

	node.Spec.Taints = append(node.Spec.Taints, corev1.Taint{
		Key:    rollingUpdateTaintKey,
		Effect: corev1.TaintEffectPreferNoSchedule,
	})

View on GitHub (pinned to 4c8573c808)

Solutions

1. Re-run the rolling update; tainting is transient and often succeeds on retry
2. Verify the node exists and is Ready via kubectl get nodes
3. Check RBAC permissions to patch nodes for the kops controller identity
4. Inspect connectivity to the API server (kubeconfig, VPN, security groups)

Example fix

// before
klog.Infof("Ignoring error tainting node %q: %v", n, err) // silently continuing
// after
// run with --fail-on-drain-error (FailOnDrainError=true) so taint failures abort the roll and can be retried cleanly

// before rolling update
nodes, _ := clientset.CoreV1().Nodes().List(ctx, metav1.ListOptions{})
for _, n := range nodes.Items {
    if n.DeletionTimestamp != nil || !isNodeReady(n) { /* exclude from taint or wait */ }
}
// also verify RBAC: auth can-i patch nodes

func isTaintable(n *corev1.Node) bool {
    return n != nil && n.DeletionTimestamp == nil && hasReadyConditionTrue(n)
}

err := c.patchTaint(ctx, n)
if err != nil {
    if k8serrors.IsNotFound(err) || k8serrors.IsConflict(err) { continue } // node gone; safe to proceed
    return fmt.Errorf("failed to taint node %q: %v", n, err)
}

When it happens

Trigger: c.patchTaint(ctx, n) returns an error for a node in the toTaint list and c.FailOnDrainError is true. Typical causes: node is already deleted/NotReady, RBAC forbids patching nodes, or the API server is unreachable.

Common situations: API server briefly unavailable mid-roll; node object stale after cloud instance was replaced out-of-band; kOps service account lacking nodes patch permission; K8s version skew causing patch failures.

Related errors

• error listing nodes in cluster: %v
• error adding needs-update label: %v
• error patching needs-update label: %v
• error querying namespace %q: %v
• listing nodes in cluster: %v

AI-assisted analysis of kubernetes/kops@4c8573c808 (2026-09-05). Data as JSON: /api/errors/c9ee65cc3a2b1de4. Report an issue: GitHub.