kubernetes/kops · error

error deregistering target: %w

Error message

error deregistering target: %w

What it means

When DescribeTargetHealth shows the target not in the 'unused' state, kOps calls DeregisterTargets to start draining; if that call fails the error is wrapped with this message. The deregistration never took effect, so the drain loop aborts and the instance is kept alive rather than terminated with live traffic.

Source

Thrown at upup/pkg/fi/cloudup/awsup/aws_cloud.go:644

			}},
		})
		if err != nil {
			return fmt.Errorf("error describing target health: %w", err)
		}

		// there will be only one target in the DescribeTargetHealth response.
		// DescribeTargetHealth response will contain a target even if the targetId doesn't exist.
		// all other states besides TargetHealthStateUnused means that the instance may still be serving traffic.
		if response.TargetHealthDescriptions[0].TargetHealth.State != elbv2types.TargetHealthStateEnumUnused {
			_, err = c.ELBV2().DeregisterTargets(ctx, &elbv2.DeregisterTargetsInput{
				TargetGroupArn: aws.String(targetGroupArn),
				Targets: []elbv2types.TargetDescription{{
					Id: aws.String(instanceId),
				}},
			})

			if err != nil {
				return fmt.Errorf("error deregistering target: %w", err)
			}

			instanceDraining = true
		}

		if !instanceDraining {
			break
		}

		time.Sleep(5 * time.Second)
	}

	klog.Infof("Successfully drained instance from targetGroup: %s", targetGroupArn)

	return nil
}

// DetachInstance causes an aws instance to no longer be counted against the ASG's size limits.

View on GitHub (pinned to 4c8573c808)

Solutions

1. Grant elasticloadbalancing:DeregisterTargets in the IAM policy used by kOps
2. Check the wrapped error code: TargetGroupNotFound → detach the stale TG from the ASG
3. Retry the rolling update; throttling errors are transient — reduce drain concurrency
4. As a fallback, deregister the target manually in the AWS console, wait for drain, then re-run kOps

Example fix

// before
// error deregistering target: AccessDenied: not authorized to perform: elasticloadbalancing:DeregisterTargets
// after: IAM statement
{"Effect":"Allow","Action":["elasticloadbalancing:DeregisterTargets"],"Resource":"*"}

// confirm permissions up front
_, err := elbv2Svc.DeregisterTargets(&elbv2.DeregisterTargetsInput{
	TargetGroupArn: aws.String(tgArn), Targets: []elbv2types.TargetDescription{{Id: aws.String(inst.ID)}}})
// ignore/dry-check result; a preemptive AccessDenied here catches IAM gaps before kOps runs

func isAccessDenied(err error) bool {
	var ae smithy.APIError
	return errors.As(err, &ae) && (ae.ErrorCode() == "AccessDenied" || ae.ErrorCode() == "UnauthorizedOperation")
}

if err := cloud.DeregisterInstance(inst); err != nil {
	if strings.Contains(err.Error(), "error deregistering target") {
		var ae smithy.APIError
		if errors.As(err, &ae) && isAccessDenied(fmt.Errorf("%s", ae.ErrorCode())) {
			return fixIAMAndRetry() // add elasticloadbalancing:DeregisterTargets
		}
		return retryWithBackoff(3, 5*time.Second, func() error { return cloud.DeregisterInstance(inst) })
	}
	return err
}

When it happens

Trigger: c.ELBV2().DeregisterTargets returns an error for the instance in a target group: AccessDenied (missing elasticloadbalancing:DeregisterTargets), TargetGroupNotFound, throttling, or network failure during rolling update drain.

Common situations: IAM policy missing DeregisterTargets; target group deleted out-of-band; throttling when draining many instances simultaneously; transient AWS API errors.

Related errors

• error deleting TargetGroup %q: %v
• target group not yet created (arn not set)
• creating NLB target group: %w
• modifying NLB target group health check: %w
• error deleting ELB TargetGroup %q: %w

AI-assisted analysis of kubernetes/kops@4c8573c808 (2026-09-05). Data as JSON: /api/errors/67d82e6b6420571d. Report an issue: GitHub.