{"record":{"id":"67d82e6b6420571d","repo":"kubernetes/kops","slug":"error-deregistering-target-w","errorCode":null,"errorMessage":"error deregistering target: %w","messagePattern":"error deregistering target: %w","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"upup/pkg/fi/cloudup/awsup/aws_cloud.go","lineNumber":644,"sourceCode":"\t\t\t}},\n\t\t})\n\t\tif err != nil {\n\t\t\treturn fmt.Errorf(\"error describing target health: %w\", err)\n\t\t}\n\n\t\t// there will be only one target in the DescribeTargetHealth response.\n\t\t// DescribeTargetHealth response will contain a target even if the targetId doesn't exist.\n\t\t// all other states besides TargetHealthStateUnused means that the instance may still be serving traffic.\n\t\tif response.TargetHealthDescriptions[0].TargetHealth.State != elbv2types.TargetHealthStateEnumUnused {\n\t\t\t_, err = c.ELBV2().DeregisterTargets(ctx, &elbv2.DeregisterTargetsInput{\n\t\t\t\tTargetGroupArn: aws.String(targetGroupArn),\n\t\t\t\tTargets: []elbv2types.TargetDescription{{\n\t\t\t\t\tId: aws.String(instanceId),\n\t\t\t\t}},\n\t\t\t})\n\n\t\t\tif err != nil {\n\t\t\t\treturn fmt.Errorf(\"error deregistering target: %w\", err)\n\t\t\t}\n\n\t\t\tinstanceDraining = true\n\t\t}\n\n\t\tif !instanceDraining {\n\t\t\tbreak\n\t\t}\n\n\t\ttime.Sleep(5 * time.Second)\n\t}\n\n\tklog.Infof(\"Successfully drained instance from targetGroup: %s\", targetGroupArn)\n\n\treturn nil\n}\n\n// DetachInstance causes an aws instance to no longer be counted against the ASG's size limits.","sourceCodeStart":626,"sourceCodeEnd":662,"githubUrl":"https://github.com/kubernetes/kops/blob/4c8573c808a73d578c5eadc86d410646ea0b0d73/upup/pkg/fi/cloudup/awsup/aws_cloud.go#L626-L662","documentation":"When DescribeTargetHealth shows the target not in the 'unused' state, kOps calls DeregisterTargets to start draining; if that call fails the error is wrapped with this message. The deregistration never took effect, so the drain loop aborts and the instance is kept alive rather than terminated with live traffic.","triggerScenarios":"c.ELBV2().DeregisterTargets returns an error for the instance in a target group: AccessDenied (missing elasticloadbalancing:DeregisterTargets), TargetGroupNotFound, throttling, or network failure during rolling update drain.","commonSituations":"IAM policy missing DeregisterTargets; target group deleted out-of-band; throttling when draining many instances simultaneously; transient AWS API errors.","solutions":["Grant elasticloadbalancing:DeregisterTargets in the IAM policy used by kOps","Check the wrapped error code: TargetGroupNotFound → detach the stale TG from the ASG","Retry the rolling update; throttling errors are transient — reduce drain concurrency","As a fallback, deregister the target manually in the AWS console, wait for drain, then re-run kOps"],"exampleFix":"// before\n// error deregistering target: AccessDenied: not authorized to perform: elasticloadbalancing:DeregisterTargets\n// after: IAM statement\n{\"Effect\":\"Allow\",\"Action\":[\"elasticloadbalancing:DeregisterTargets\"],\"Resource\":\"*\"}","handlingStrategy":"retry","validationCode":"// confirm permissions up front\n_, err := elbv2Svc.DeregisterTargets(&elbv2.DeregisterTargetsInput{\n\tTargetGroupArn: aws.String(tgArn), Targets: []elbv2types.TargetDescription{{Id: aws.String(inst.ID)}}})\n// ignore/dry-check result; a preemptive AccessDenied here catches IAM gaps before kOps runs","typeGuard":"func isAccessDenied(err error) bool {\n\tvar ae smithy.APIError\n\treturn errors.As(err, &ae) && (ae.ErrorCode() == \"AccessDenied\" || ae.ErrorCode() == \"UnauthorizedOperation\")\n}","tryCatchPattern":"if err := cloud.DeregisterInstance(inst); err != nil {\n\tif strings.Contains(err.Error(), \"error deregistering target\") {\n\t\tvar ae smithy.APIError\n\t\tif errors.As(err, &ae) && isAccessDenied(fmt.Errorf(\"%s\", ae.ErrorCode())) {\n\t\t\treturn fixIAMAndRetry() // add elasticloadbalancing:DeregisterTargets\n\t\t}\n\t\treturn retryWithBackoff(3, 5*time.Second, func() error { return cloud.DeregisterInstance(inst) })\n\t}\n\treturn err\n}","preventionTips":["Include elasticloadbalancing:DeregisterTargets in the kOps IAM policy","Don't delete target groups mid-rolling-update","Rate-limit concurrent instance drains to avoid throttling","If stuck, manually deregister the target and re-run kOps"],"tags":["aws","elbv2","target-group","deregister"],"backgroundTag":"target-group-deregistration-failed","analyzedSha":"4c8573c808a73d578c5eadc86d410646ea0b0d73","analyzedAt":"2026-09-05T04:13:19.212Z","contentChangedAt":"2026-09-05T04:13:19.212Z","schemaVersion":2},"datasetVersion":"2026-09-12T12:17:11.808Z"}