kubernetes/kops · error · DeregisterError

error deregistering instance %q, node %q: %w

Error message

error deregistering instance %q, node %q: %w

What it means

Before draining pods, drainNode may deregister the instance from cloud load balancers via c.Cloud.DeregisterInstance(u). Any failure is wrapped in a DeregisterError carrying 'error deregistering instance %q, node %q: %w'. This prevents the node from being safely removed from backend target pools before it disappears.

Source

Thrown at pkg/instancegroups/instancegroups.go:729

		return fmt.Errorf("error excluding node from load balancer: %v", err)
	}

	shouldDeregister := true
	if !c.Options.DeregisterControlPlaneNodes {
		if u.CloudInstanceGroup != nil && u.CloudInstanceGroup.InstanceGroup != nil {
			role := u.CloudInstanceGroup.InstanceGroup.Spec.Role
			switch role {
			case api.InstanceGroupRoleAPIServer, api.InstanceGroupRoleControlPlane:
				klog.Infof("skipping deregistration of instance %q, as part of instancegroup with role %q", u.ID, role)
				shouldDeregister = false
			}
		}
	}

	if shouldDeregister {
		if err := c.Cloud.DeregisterInstance(u); err != nil {
			return &DeregisterError{
				err: fmt.Errorf("error deregistering instance %q, node %q: %w", u.ID, u.Node.Name, err),
			}
		}
	}

	if err := drain.RunNodeDrain(helper, u.Node.Name); err != nil {
		if apierrors.IsNotFound(err) {
			return nil
		}
		return fmt.Errorf("error draining node: %v", err)
	}

	if c.PostDrainDelay > 0 {
		klog.Infof("Waiting for %s for pods to stabilize after draining.", c.PostDrainDelay)
		time.Sleep(c.PostDrainDelay)
	}

	return nil
}

View on GitHub (pinned to 4c8573c808)

Solutions

1. Verify IAM permissions for deregistering instances from load balancers (e.g. elasticloadbalancing:* on the target groups)
2. Confirm the instance still exists: `aws ec2 describe-instances --instance-ids <id>`; if already gone, retry the rolling update (NotFound-like cases often proceed)
3. Check cloud API rate limits/errors in kOps logs and reduce MaxSurge/parallelism or increase intervals
4. Refresh credentials/config (`kops export kubecfg`) if auth errors appear

Example fix

// before: policy without ELB permissions
{"Effect":"Deny","Action":"elasticloadbalancing:DeregisterTargets","Resource":"*"}
// after: allow deregistration
{"Effect":"Allow","Action":"elasticloadbalancing:DeregisterTargets","Resource":"*"}

if _, err := elbSvc.DescribeTargetHealth(tgArn); err != nil {
    return fmt.Errorf("cannot reach load balancer API: %w", err)
}

var de *DeregisterError
if err := c.drainNode(ctx, u); err != nil {
    if errors.As(err, &de) {
        time.Sleep(backoff)
        err = c.drainNode(ctx, u) // cloud API errors are often transient
    }
    return err
}

When it happens

Trigger: DeregisterInstance fails on the cloud provider side: instance already terminated (AWS InvalidInstanceID), target group / backend service not found, cloud API throttling or credential issues.

Common situations: Stale kOps state where the instance was deleted externally; IAM policy missing elbv2/autoscaling deregister permissions; API rate limits during large rolling updates; provider SDK errors after region/credential changes.

Related errors

• failed to deregister instance from loadBalancer before termi
• errShutdown
• timed out waiting for volume to detach
• error excluding node from load balancer: %v
• failed to detach instance: %v

AI-assisted analysis of kubernetes/kops@4c8573c808 (2026-09-05). Data as JSON: /api/errors/a1e089d5c1dd154c. Report an issue: GitHub.