kubernetes/kops · error

error describing target health

Error message

error describing target health: %w

What it means

deregisterInstanceFromTargetGroup polls DescribeTargetHealth to determine whether the instance target is still serving traffic; an AWS SDK error from that call aborts with this wrapped error. Since the drain loop cannot know the target's state, it stops and the instance is not terminated.

Solutions

1. Check the wrapped AWS error: AccessDenied → fix IAM; TargetGroupNotFound → remove the stale TG from the ASG
2. Ensure IAM policy includes elasticloadbalancing:DescribeTargetHealth
3. Verify kOps client region matches the target group's region
4. Retry on throttling/transient errors

Example fix

// before
// error describing target health: ValidationError: Target group 'arn:aws:elasticloadbalancing:...' not found
// after: detach stale TG from ASG then retry
aws autoscaling detach-load-balancer-target-groups --auto-scaling-group-name nodes --target-group-arns arn:aws:elasticloadbalancing:...:stale-tg

out, err := elbv2Svc.DescribeTargetGroups(&elbv2.DescribeTargetGroupsInput{TargetGroupArns: []string{tgArn}})
if err != nil || len(out.TargetGroups) == 0 {
	return fmt.Errorf("target group %s not found in client region before drain", tgArn)
}

func isTargetGroupNotFound(err error) bool {
	var ae smithy.APIError
	return errors.As(err, &ae) && ae.ErrorCode() == "TargetGroupNotFound"
}

if err := cloud.DeregisterInstance(inst); err != nil {
	var ae smithy.APIError
	if errors.As(err, &ae) && ae.ErrorCode() == "TargetGroupNotFound" {
		return detachStaleTargetGroup(asgName, tgArn) // then retry drain
	}
	return err
}

When it happens

Trigger: c.ELBV2().DescribeTargetHealth returns an error during the drain polling loop: AccessDenied (missing elasticloadbalancing:DescribeTargetHealth), TargetGroupNotFound (TG deleted while still on the ASG), throttling, or network failure.

Common situations: Target group removed out-of-band but still listed in the ASG's TargetGroupARNs; IAM gaps; API throttling; cross-region ARN mismatch where the TG lives in a different region than the client.

Related errors

• modifying NLB target group health check
• creating NLB target group
• error deleting ELB TargetGroup
• error deleting TargetGroup
• error deregistering target

AI-assisted analysis of kubernetes/kops@4c8573c808 (2026-09-05). Data as JSON: /api/errors/383538ab162fbe7a. Report an issue: GitHub.

Appendix: source

Thrown at upup/pkg/fi/cloudup/awsup/aws_cloud.go:629

	}

	return nil
}

func deregisterInstanceFromTargetGroup(ctx context.Context, c AWSCloud, targetGroupArn string, instanceId string) error {
	klog.Infof("Deregistering instance from targetGroup: %s", targetGroupArn)

	for {
		instanceDraining := false

		response, err := c.ELBV2().DescribeTargetHealth(ctx, &elbv2.DescribeTargetHealthInput{
			TargetGroupArn: aws.String(targetGroupArn),
			Targets: []elbv2types.TargetDescription{{
				Id: aws.String(instanceId),
			}},
		})
		if err != nil {
			return fmt.Errorf("error describing target health: %w", err)
		}

		// there will be only one target in the DescribeTargetHealth response.
		// DescribeTargetHealth response will contain a target even if the targetId doesn't exist.
		// all other states besides TargetHealthStateUnused means that the instance may still be serving traffic.
		if response.TargetHealthDescriptions[0].TargetHealth.State != elbv2types.TargetHealthStateEnumUnused {
			_, err = c.ELBV2().DeregisterTargets(ctx, &elbv2.DeregisterTargetsInput{
				TargetGroupArn: aws.String(targetGroupArn),
				Targets: []elbv2types.TargetDescription{{
					Id: aws.String(instanceId),
				}},
			})

			if err != nil {
				return fmt.Errorf("error deregistering target: %w", err)
			}

			instanceDraining = true

View on GitHub (pinned to 4c8573c808)