kubernetes/kops · error

applying patch to node: %w

Error message

applying patch to node: %w

What it means

patchNodePodCIDRs issues a strategic-merge PATCH against the Node resource (client.Nodes().Patch) to set spec.podCIDR/podCIDRs. This error wraps any failure returned by the Kubernetes API server for that patch request, such as RBAC denial, conflicts, validation rejection, or connectivity problems. Reconcile requeues and retries with backoff when this occurs.

Source

Thrown at cmd/kops-controller/controllers/awsipam.go:187

	nodePatchSpec := &nodePatchSpec{
		PodCIDRs: podCIDRs,
	}
	if len(podCIDRs) > 0 {
		nodePatchSpec.PodCIDR = podCIDRs[0]
	}
	nodePatch := &nodePatch{
		Spec: nodePatchSpec,
	}
	nodePatchJson, err := json.Marshal(nodePatch)
	if err != nil {
		return fmt.Errorf("building node patch: %w", err)
	}

	klog.V(2).Infof("sending patch for node %q: %q", node.Name, string(nodePatchJson))

	_, err = client.Nodes().Patch(ctx, node.Name, types.StrategicMergePatchType, nodePatchJson, metav1.PatchOptions{})
	if err != nil {
		return fmt.Errorf("applying patch to node: %w", err)
	}

	return nil
}

View on GitHub (pinned to 4c8573c808)

Solutions

1. Check the wrapped error and controller logs for the HTTP status: 403 means grant the service account patch/get/list/watch on nodes (the +kubebuilder:rbac marker at awsipam.go:92 must be rendered into the RBAC manifests).
2. For conflicts/validation errors on podCIDR, confirm the node's podCIDRs are actually empty (the controller only patches empty ones) and that kube-controller-manager's node IPAM controller is not assigning them concurrently.
3. Retry — controller-runtime requeues automatically; transient API server issues (timeouts, 5xx, token refresh) usually resolve on the next attempt.
4. Verify API server connectivity from the controller pod (dns, kubeconfig, CA/token mounts) if errors are connection-related.

Example fix

// before
_, err = client.Nodes().Patch(ctx, node.Name, types.StrategicMergePatchType, nodePatchJson, metav1.PatchOptions{})
if err != nil {
	return fmt.Errorf("applying patch to node: %w", err)
}
// after
_, err = client.Nodes().Patch(ctx, node.Name, types.StrategicMergePatchType, nodePatchJson, metav1.PatchOptions{})
if err != nil {
	if apierrors.IsConflict(err) || apierrors.IsNotFound(err) {
		klog.V(2).Infof("skipping podCIDR patch for node %q: %v", node.Name, err)
		return nil
	}
	return fmt.Errorf("applying patch to node: %w", err)
}

// pre-flight: verify RBAC allows patching nodes
authClient := authorizationv1.NewForConfigOrDie(restConfig)
ok, err := authClient.SelfSubjectAccessReviews().Create(ctx, &authorizationv1.SelfSubjectAccessReview{
	Spec: authorizationv1.SelfSubjectAccessReviewSpec{
		ResourceAttributes: &authorizationv1.ResourceAttributes{Verb: "patch", Resource: "nodes"},
	},
})
if err != nil || !ok.Status.Allowed {
	return fmt.Errorf("service account cannot patch nodes")
}

func isRetryablePatchError(err error) bool {
	return apierrors.IsConflict(err) || apierrors.IsServerTimeout(err) || apierrors.IsTooManyRequests(err)
}

_, err = client.Nodes().Patch(ctx, node.Name, types.StrategicMergePatchType, nodePatchJson, metav1.PatchOptions{})
if err != nil {
	switch {
	case apierrors.IsNotFound(err):
		return nil // node deleted; nothing to patch
	case apierrors.IsConflict(err) || apierrors.IsServerTimeout(err):
		return fmt.Errorf("applying patch to node: %w", err) // requeue/retry
	default:
		return fmt.Errorf("applying patch to node: %w", err)
	}
}

When it happens

Trigger: client.Nodes().Patch(ctx, node.Name, types.StrategicMergePatchType, nodePatchJson, metav1.PatchOptions{}) fails: the kops-controller service account lacks patch permission on nodes (RBAC), the API server rejects modifying an immutable/defaulted field (e.g. podCIDR already set or kube-controller-manager ownership conflict), a 409 conflict from a simultaneous update, or network/TLS errors reaching the API server.

Common situations: Cluster upgrades or kubebuilder RBAC markers (awsipam.go:92) not reflected in the deployed ClusterRole; kube-controller-manager node IPAM controller fighting over podCIDR assignment; the node object changed/removed concurrently; controller cannot reach the API server due to networking or an expired service-account token.

Related errors

• error applying annotation to namespace: %v
• error applying patch to node: %v
• error adding needs-update label: %v
• error applying annotation to record addon installation: %v
• error patching needs-update label: %v

AI-assisted analysis of kubernetes/kops@4c8573c808 (2026-09-05). Data as JSON: /api/errors/862e94b47d3f5bbb. Report an issue: GitHub.