What causes "error excluding node from load balancer: %v" in kubernetes/kops?

Check webhook/policy controllers that might reject node label PATCHes and add an exemption for kops.k8s.io/* labels

error excluding node from load balancer: %v

Error message

error excluding node from load balancer: %v

What it means

drainNode calls patchExcludeFromLB to add the kops.k8s.io/exclude-from-external-load-balancers label so the node is taken out of load-balancer pools before draining. A non-NotFound failure is wrapped as 'error excluding node from load balancer'.

Source

Thrown at pkg/instancegroups/instancegroups.go:711

		// The zero value would retry evictions without any delay
		EvictErrorRetryDelay: 5 * time.Second,

		// We want to proceed even when pods are using emptyDir volumes
		DeleteEmptyDirData: true,
	}

	if err := drain.RunCordonOrUncordon(helper, u.Node, true); err != nil {
		if apierrors.IsNotFound(err) {
			return nil
		}
		return fmt.Errorf("error cordoning node: %v", err)
	}

	if err := c.patchExcludeFromLB(ctx, u.Node); err != nil {
		if apierrors.IsNotFound(err) {
			return nil
		}
		return fmt.Errorf("error excluding node from load balancer: %v", err)
	}

	shouldDeregister := true
	if !c.Options.DeregisterControlPlaneNodes {
		if u.CloudInstanceGroup != nil && u.CloudInstanceGroup.InstanceGroup != nil {
			role := u.CloudInstanceGroup.InstanceGroup.Spec.Role
			switch role {
			case api.InstanceGroupRoleAPIServer, api.InstanceGroupRoleControlPlane:
				klog.Infof("skipping deregistration of instance %q, as part of instancegroup with role %q", u.ID, role)
				shouldDeregister = false
			}
		}
	}

	if shouldDeregister {
		if err := c.Cloud.DeregisterInstance(u); err != nil {
			return &DeregisterError{
				err: fmt.Errorf("error deregistering instance %q, node %q: %w", u.ID, u.Node.Name, err),

View on GitHub (pinned to 4c8573c808)

Solutions

Check webhook/policy controllers that might reject node label PATCHes and add an exemption for kops.k8s.io/* labels
Ensure RBAC allows patching nodes for the user kOps authenticates as
Retry the rolling update after confirming API server health via `kubectl get --raw /readyz`
If the API-server LB still routes to the node, update the target group/backend registration manually then continue

Defensive patterns

Strategy: try-catch

Validate before calling

_, err := client.CoreV1().Nodes().Patch(ctx, nodeName, types.StrategicMergePatchType, []byte(`{"metadata":{"labels":{"kops.k8s.io/exclude-from-external-load-balancers":"true"}}}`), metav1.PatchOptions{})
if err != nil && !apierrors.IsNotFound(err) {
    return fmt.Errorf("pre-check LB-exclude patch failed: %w", err)
}

Try / catch

if err := c.drainNode(ctx, u); err != nil {
    if !apierrors.IsNotFound(errors.Unwrap(err)) {
        klog.Errorf("exclude-from-LB failed: %v; check webhooks/RBAC", err)
        return err
    }
}

Prevention

Exempt kops.k8s.io/* node labels from validating webhooks
Grant node patch RBAC to the kOps identity
Check API server health before rolling updates

When it happens

Trigger: The label PATCH on the Node fails with a non-NotFound error: RBAC restrictions, API server timeout, or a modified webhook validating node labels on control-plane/APIServer nodes.

Common situations: Admission webhooks or policies (OPA/Gatekeeper) blocking label changes on nodes; API server load balancer itself pointing at the node being patched (control-plane self-hosted LB); permission errors for the acting user.

Related errors

AI-assisted analysis of kubernetes/kops@4c8573c808 (2026-09-05). Data as JSON: /api/errors/2b3ecff2701813dd. Report an issue: GitHub.