kubernetes/kops · error

failed to deregister instance from loadBalancer before termi

Error message

failed to deregister instance from loadBalancer before terminating: %v

What it means

kOps wraps any failure from deregisterInstance() — the drain of an EC2 instance from all Classic ELBs and target groups — with this message before the instance is terminated during rolling updates or instance deletion. It means the instance could not be proven fully drained from its load balancers, so kOps refuses to continue terminating it (to avoid dropping live connections). The wrapped inner error carries the actual cause (ASG describe failure, ELB health-check failure, or target-group deregistration failure).

Source

Thrown at upup/pkg/fi/cloudup/awsup/aws_cloud.go:485

		}

		return spotinst.DeleteInstance(c.spotinst, i)
	}

	return deleteInstance(ctx, c, i)
}

// DeregisterInstance drains a cloud instance and load balancers.
func (c *awsCloudImplementation) DeregisterInstance(i *cloudinstances.CloudInstance) error {
	ctx := context.TODO()

	if c.spotinst != nil || i.CloudInstanceGroup.InstanceGroup.Spec.Manager == kops.InstanceManagerKarpenter {
		return nil
	}

	err := deregisterInstance(ctx, c, i)
	if err != nil {
		return fmt.Errorf("failed to deregister instance from loadBalancer before terminating: %v", err)
	}

	return nil
}

func deleteInstance(ctx context.Context, c AWSCloud, i *cloudinstances.CloudInstance) error {
	id := i.ID
	if id == "" {
		return fmt.Errorf("id was not set on CloudInstance: %v", i)
	}

	request := &ec2.TerminateInstancesInput{
		InstanceIds: []string{id},
	}

	if _, err := c.EC2().TerminateInstances(ctx, request); err != nil {
		if AWSErrorCode(err) == "InvalidInstanceID.NotFound" {
			klog.V(2).Infof("Got InvalidInstanceID.NotFound error deleting instance %q; will treat as already-deleted", id)

View on GitHub (pinned to 4c8573c808)

Solutions

1. Inspect the wrapped inner error (%v) to identify whether the ASG describe, ELB health check, or target group deregistration failed
2. Verify IAM policy grants elb:DescribeInstanceHealth, elb:DeregisterInstancesFromLoadBalancer, elasticloadbalancing:DescribeTargetHealth and elasticloadbalancing:DeregisterTargets
3. Check that the load balancers/target groups attached to the ASG still exist; remove stale LB references from the ASG if they were deleted
4. Retry the rolling update — transient throttling or network errors often resolve on retry

Example fix

// before: guessing at cause from the wrapper
// failed to deregister instance from loadBalancer before terminating: ...
// after: log the full wrapped chain and check the inner AWS error code
if err := cloud.DeregisterInstance(i); err != nil {
	klog.Errorf("deregister failed: %v", err) // inspect inner cause, e.g. AccessDenied
	// fix IAM: attach elasticloadbalancing:DescribeTargetHealth etc.
}

if i.CloudInstanceGroup.InstanceGroup.Spec.Manager == kops.InstanceManagerKarpenter {
	return nil // deregistration is skipped for Karpenter-managed instances
}
if i.ID == "" {
	return fmt.Errorf("instance has no ID; cannot drain from LBs")
}

func canDeregister(i *cloudinstances.CloudInstance) bool {
	return i != nil && i.CloudInstanceGroup != nil && i.ID != ""
}

if err := cloud.DeregisterInstance(inst); err != nil {
	klog.Errorf("drain failed for %s: %v; NOT terminating instance to avoid dropped connections", inst.ID, err)
	return err // propagate so the rolling update marks the instance unsafe to replace
}

When it happens

Trigger: CloudImplementation.DeregisterInstance is called during `kops rolling-update cluster` / instance deletion; deregisterInstance fails because DescribeAutoScalingGroups errors, or the errgroup returns an error from deregisterInstanceFromClassicLoadBalancer (DescribeInstanceHealth API error) or deregisterInstanceFromTargetGroups (DescribeTargetHealth or DeregisterTargets API error).

Common situations: Stale or insufficient IAM permissions on elb:DescribeInstanceHealth / elasticloadbalancing:DescribeTargetHealth / DeregisterTargets; ELB or target group already deleted while ASG metadata still references it; throttling during large rolling updates; network/region misconfiguration causing AWS API failures.

Related errors

• error deregistering instance %q, node %q: %w
• failed to deregister instance from load balancers: %v
• errShutdown
• timed out waiting for volume to detach
• failed to drain node %q: %w

AI-assisted analysis of kubernetes/kops@4c8573c808 (2026-09-05). Data as JSON: /api/errors/1fdc37584e4ac74d. Report an issue: GitHub.