kubernetes/kops · error

failed to deregister instance from load balancers

Error message

failed to deregister instance from load balancers: %v

What it means

After describing the ASG, deregisterInstance drains the instance concurrently from all Classic ELBs and all target groups using an errgroup; if ANY of those goroutines fails, the aggregated error is wrapped with this message. The instance is not considered safe to terminate. Note the wording can be misleading — target-group failures also surface here (nested under the 'failed to register instance from targetGroups' error).

Solutions

1. Read the nested wrapped error to see whether the Classic ELB or target group path failed and which LB/TG
2. Grant elb:DescribeInstanceHealth, elb:DeregisterInstancesFromLoadBalancer, elasticloadbalancing:DescribeTargetHealth, elasticloadbalancing:DeregisterTargets in IAM
3. Verify all load balancers/target groups on the ASG exist; detach deleted ones from the ASG
4. Retry after transient API errors; consider draining manually via AWS console before deleting the instance

Example fix

// before: nested error shows cause
// failed to deregister instance from load balancers: failed to register instance from targetGroups: error describing target health: AccessDenied
// after: fix IAM
{"Effect":"Allow","Action":["elasticloadbalancing:DescribeTargetHealth","elasticloadbalancing:DeregisterTargets","elasticloadbalancing:DescribeInstanceHealth","elasticloadbalancing:DeregisterInstancesFromLoadBalancer"],"Resource":"*"}

// verify all LBs/TGs on the ASG still exist before draining
for _, tgArn := range asg.TargetGroupARNs {
	if _, err := elbv2Svc.DescribeTargetGroups(&elbv2.DescribeTargetGroupsInput{TargetGroupArns: []string{tgArn}}); err != nil {
		return fmt.Errorf("stale target group %s on ASG: %w", tgArn, err)
	}
}

func isDeregistrationErr(err error) bool {
	return err != nil && strings.Contains(err.Error(), "failed to deregister instance from load balancers")
}

if err := cloud.DeregisterInstance(inst); err != nil {
	klog.Errorf("LB drain failed for %s: %v", inst.ID, err)
	// inspect nested cause: classic ELB vs target group failure, then fix IAM or stale LB refs
	return err
}

When it happens

Trigger: eg.Wait() returns non-nil because deregisterInstanceFromClassicLoadBalancer hit a DescribeInstanceHealth error, or deregisterInstanceFromTargetGroups hit a DescribeTargetHealth/DeregisterTargets error, for any of the ASG's load balancers.

Common situations: Missing ELB/ELBv2 IAM permissions; a load balancer referenced by the ASG was deleted out-of-band; deregistration draining loop interrupted by API errors; throttling with many LBs/target groups attached.

Related errors

• failed to deregister instance from loadBalancer before…
• error attaching load balancer to new subnets
• error attaching LoadBalancers
• error building load balancers
• error deleting ELB LoadBalancer

AI-assisted analysis of kubernetes/kops@4c8573c808 (2026-09-05). Data as JSON: /api/errors/2c7da5d5b5b3889e. Report an issue: GitHub.

Appendix: source

Thrown at upup/pkg/fi/cloudup/awsup/aws_cloud.go:548

	loadBalancerNames := asgDetails.AutoScalingGroups[0].LoadBalancerNames
	targetGroupArns := asgDetails.AutoScalingGroups[0].TargetGroupARNs

	eg, _ := errgroup.WithContext(context.Background())

	if len(loadBalancerNames) != 0 {
		eg.Go(func() error {
			return deregisterInstanceFromClassicLoadBalancer(ctx, c, loadBalancerNames, i.ID)
		})
	}

	if len(targetGroupArns) != 0 {
		eg.Go(func() error {
			return deregisterInstanceFromTargetGroups(ctx, c, targetGroupArns, i.ID)
		})
	}

	if err := eg.Wait(); err != nil {
		return fmt.Errorf("failed to deregister instance from load balancers: %v", err)
	}

	return nil
}

// deregisterInstanceFromClassicLoadBalancer ensures that connectionDraining completes for the associated classic loadBalancer to ensure no dropped connections.
func deregisterInstanceFromClassicLoadBalancer(ctx context.Context, c AWSCloud, loadBalancerNames []string, instanceId string) error {
	klog.Infof("Deregistering instance from classic loadBalancers: %v", loadBalancerNames)

	for {
		instanceDraining := false
		for _, loadBalancerName := range loadBalancerNames {
			response, err := c.ELB().DescribeInstanceHealth(ctx, &elb.DescribeInstanceHealthInput{
				LoadBalancerName: aws.String(loadBalancerName),
				Instances: []elbtypes.Instance{{
					InstanceId: aws.String(instanceId),
				}},
			})

View on GitHub (pinned to 4c8573c808)