kubernetes/kops · error

failed to deregister instance from load balancers: %v

Error message

failed to deregister instance from load balancers: %v

What it means

After describing the ASG, deregisterInstance drains the instance concurrently from all Classic ELBs and all target groups using an errgroup; if ANY of those goroutines fails, the aggregated error is wrapped with this message. The instance is not considered safe to terminate. Note the wording can be misleading — target-group failures also surface here (nested under the 'failed to register instance from targetGroups' error).

Source

Thrown at upup/pkg/fi/cloudup/awsup/aws_cloud.go:548

	loadBalancerNames := asgDetails.AutoScalingGroups[0].LoadBalancerNames
	targetGroupArns := asgDetails.AutoScalingGroups[0].TargetGroupARNs

	eg, _ := errgroup.WithContext(context.Background())

	if len(loadBalancerNames) != 0 {
		eg.Go(func() error {
			return deregisterInstanceFromClassicLoadBalancer(ctx, c, loadBalancerNames, i.ID)
		})
	}

	if len(targetGroupArns) != 0 {
		eg.Go(func() error {
			return deregisterInstanceFromTargetGroups(ctx, c, targetGroupArns, i.ID)
		})
	}

	if err := eg.Wait(); err != nil {
		return fmt.Errorf("failed to deregister instance from load balancers: %v", err)
	}

	return nil
}

// deregisterInstanceFromClassicLoadBalancer ensures that connectionDraining completes for the associated classic loadBalancer to ensure no dropped connections.
func deregisterInstanceFromClassicLoadBalancer(ctx context.Context, c AWSCloud, loadBalancerNames []string, instanceId string) error {
	klog.Infof("Deregistering instance from classic loadBalancers: %v", loadBalancerNames)

	for {
		instanceDraining := false
		for _, loadBalancerName := range loadBalancerNames {
			response, err := c.ELB().DescribeInstanceHealth(ctx, &elb.DescribeInstanceHealthInput{
				LoadBalancerName: aws.String(loadBalancerName),
				Instances: []elbtypes.Instance{{
					InstanceId: aws.String(instanceId),
				}},
			})

View on GitHub (pinned to 4c8573c808)

Solutions

1. Read the nested wrapped error to see whether the Classic ELB or target group path failed and which LB/TG
2. Grant elb:DescribeInstanceHealth, elb:DeregisterInstancesFromLoadBalancer, elasticloadbalancing:DescribeTargetHealth, elasticloadbalancing:DeregisterTargets in IAM
3. Verify all load balancers/target groups on the ASG exist; detach deleted ones from the ASG
4. Retry after transient API errors; consider draining manually via AWS console before deleting the instance

Example fix

// before: nested error shows cause
// failed to deregister instance from load balancers: failed to register instance from targetGroups: error describing target health: AccessDenied
// after: fix IAM
{"Effect":"Allow","Action":["elasticloadbalancing:DescribeTargetHealth","elasticloadbalancing:DeregisterTargets","elasticloadbalancing:DescribeInstanceHealth","elasticloadbalancing:DeregisterInstancesFromLoadBalancer"],"Resource":"*"}

// verify all LBs/TGs on the ASG still exist before draining
for _, tgArn := range asg.TargetGroupARNs {
	if _, err := elbv2Svc.DescribeTargetGroups(&elbv2.DescribeTargetGroupsInput{TargetGroupArns: []string{tgArn}}); err != nil {
		return fmt.Errorf("stale target group %s on ASG: %w", tgArn, err)
	}
}

func isDeregistrationErr(err error) bool {
	return err != nil && strings.Contains(err.Error(), "failed to deregister instance from load balancers")
}

if err := cloud.DeregisterInstance(inst); err != nil {
	klog.Errorf("LB drain failed for %s: %v", inst.ID, err)
	// inspect nested cause: classic ELB vs target group failure, then fix IAM or stale LB refs
	return err
}

When it happens

Trigger: eg.Wait() returns non-nil because deregisterInstanceFromClassicLoadBalancer hit a DescribeInstanceHealth error, or deregisterInstanceFromTargetGroups hit a DescribeTargetHealth/DeregisterTargets error, for any of the ASG's load balancers.

Common situations: Missing ELB/ELBv2 IAM permissions; a load balancer referenced by the ASG was deleted out-of-band; deregistration draining loop interrupted by API errors; throttling with many LBs/target groups attached.

Related errors

• failed to deregister instance from loadBalancer before termi
• error deregistering instance %q, node %q: %w
• unknown load balancer Type: %q
• unhandled bastion LoadBalancer type %q
• error building load balancers: %v

AI-assisted analysis of kubernetes/kops@4c8573c808 (2026-09-05). Data as JSON: /api/errors/2c7da5d5b5b3889e. Report an issue: GitHub.