kubernetes/kops · error

error describing instance health: %v

Error message

error describing instance health: %v

What it means

deregisterInstanceFromClassicLoadBalancer polls DescribeInstanceHealth in a loop to check the instance's state on each Classic ELB; an AWS SDK error from that call aborts the drain with this wrapped error. kOps cannot confirm whether the instance is InService, so it will not deregister/terminate and risk dropping connections.

Source

Thrown at upup/pkg/fi/cloudup/awsup/aws_cloud.go:568

	return nil
}

// deregisterInstanceFromClassicLoadBalancer ensures that connectionDraining completes for the associated classic loadBalancer to ensure no dropped connections.
func deregisterInstanceFromClassicLoadBalancer(ctx context.Context, c AWSCloud, loadBalancerNames []string, instanceId string) error {
	klog.Infof("Deregistering instance from classic loadBalancers: %v", loadBalancerNames)

	for {
		instanceDraining := false
		for _, loadBalancerName := range loadBalancerNames {
			response, err := c.ELB().DescribeInstanceHealth(ctx, &elb.DescribeInstanceHealthInput{
				LoadBalancerName: aws.String(loadBalancerName),
				Instances: []elbtypes.Instance{{
					InstanceId: aws.String(instanceId),
				}},
			})
			if err != nil {
				return fmt.Errorf("error describing instance health: %v", err)
			}

			// describeInstanceHealth can return an empty list if the instance was already terminated.
			if len(response.InstanceStates) == 0 {
				continue
			}

			// there will be only one instance in the DescribeInstanceHealth response.
			if aws.ToString(response.InstanceStates[0].State) == instanceInServiceState {
				c.ELB().DeregisterInstancesFromLoadBalancer(ctx, &elb.DeregisterInstancesFromLoadBalancerInput{
					LoadBalancerName: aws.String(loadBalancerName),
					Instances: []elbtypes.Instance{{
						InstanceId: aws.String(instanceId),
					}},
				})
				instanceDraining = true
			}
		}

View on GitHub (pinned to 4c8573c808)

Solutions

1. Check whether the named Classic ELB still exists; if deleted, remove it from the ASG's LoadBalancerNames (or update the cluster spec) and retry
2. Ensure IAM grants elb:DescribeInstanceHealth
3. Retry on transient throttling/network errors
4. Migrate off Classic ELBs to NLB/ALB target groups if the legacy ELB stack keeps causing drift

Example fix

// before
// error describing instance health: AccessDenied: not authorized to perform: elasticloadbalancing:DescribeInstanceHealth
// after: IAM statement
{"Effect":"Allow","Action":["elasticloadbalancing:DescribeInstanceHealth"],"Resource":"*"}

_, err := elbSvc.DescribeLoadBalancers(&elb.DescribeLoadBalancersInput{LoadBalancerNames: lbNames})
if err != nil {
	return fmt.Errorf("referenced classic ELB missing/unreachable: %w", err)
}

func isELBNotFound(err error) bool {
	var ae smithy.APIError
	return errors.As(err, &ae) && ae.ErrorCode() == "LoadBalancerNotFound"
}

if err := cloud.DeregisterInstance(inst); err != nil {
	if strings.Contains(err.Error(), "error describing instance health") {
		var ae smithy.APIError
		if errors.As(err, &ae) && ae.ErrorCode() == "LoadBalancerNotFound" {
			return detachStaleELBFromASG(asgName) // remove stale ref, then retry
		}
		return retryWithBackoff(3, 5*time.Second, func() error { return cloud.DeregisterInstance(inst) })
	}
	return err
}

When it happens

Trigger: c.ELB().DescribeInstanceHealth returns an error — typically AccessDenied (missing elb:DescribeInstanceHealth), LoadBalancerNotFound (ELB deleted while ASG still references it), throttling, or a network failure during the polling loop.

Common situations: Classic ELB deleted out-of-band but still attached to the ASG's LoadBalancerNames; IAM policy gaps; regional misconfiguration; ELB API throttling during big rolling updates.

Related errors

• error listing ELBs: %v
• Found multiple ELBs with name %q
• error deleting LoadBalancer %q: %v
• error listing elbs: %v
• error listing elb Tags: %v

AI-assisted analysis of kubernetes/kops@4c8573c808 (2026-09-05). Data as JSON: /api/errors/ac9932e13df61e6b. Report an issue: GitHub.