{"record":{"id":"1fdc37584e4ac74d","repo":"kubernetes/kops","slug":"failed-to-deregister-instance-from-loadbalancer-be","errorCode":null,"errorMessage":"failed to deregister instance from loadBalancer before terminating: %v","messagePattern":"failed to deregister instance from loadBalancer before terminating: (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"upup/pkg/fi/cloudup/awsup/aws_cloud.go","lineNumber":485,"sourceCode":"\t\t}\n\n\t\treturn spotinst.DeleteInstance(c.spotinst, i)\n\t}\n\n\treturn deleteInstance(ctx, c, i)\n}\n\n// DeregisterInstance drains a cloud instance and load balancers.\nfunc (c *awsCloudImplementation) DeregisterInstance(i *cloudinstances.CloudInstance) error {\n\tctx := context.TODO()\n\n\tif c.spotinst != nil || i.CloudInstanceGroup.InstanceGroup.Spec.Manager == kops.InstanceManagerKarpenter {\n\t\treturn nil\n\t}\n\n\terr := deregisterInstance(ctx, c, i)\n\tif err != nil {\n\t\treturn fmt.Errorf(\"failed to deregister instance from loadBalancer before terminating: %v\", err)\n\t}\n\n\treturn nil\n}\n\nfunc deleteInstance(ctx context.Context, c AWSCloud, i *cloudinstances.CloudInstance) error {\n\tid := i.ID\n\tif id == \"\" {\n\t\treturn fmt.Errorf(\"id was not set on CloudInstance: %v\", i)\n\t}\n\n\trequest := &ec2.TerminateInstancesInput{\n\t\tInstanceIds: []string{id},\n\t}\n\n\tif _, err := c.EC2().TerminateInstances(ctx, request); err != nil {\n\t\tif AWSErrorCode(err) == \"InvalidInstanceID.NotFound\" {\n\t\t\tklog.V(2).Infof(\"Got InvalidInstanceID.NotFound error deleting instance %q; will treat as already-deleted\", id)","sourceCodeStart":467,"sourceCodeEnd":503,"githubUrl":"https://github.com/kubernetes/kops/blob/4c8573c808a73d578c5eadc86d410646ea0b0d73/upup/pkg/fi/cloudup/awsup/aws_cloud.go#L467-L503","documentation":"kOps wraps any failure from deregisterInstance() — the drain of an EC2 instance from all Classic ELBs and target groups — with this message before the instance is terminated during rolling updates or instance deletion. It means the instance could not be proven fully drained from its load balancers, so kOps refuses to continue terminating it (to avoid dropping live connections). The wrapped inner error carries the actual cause (ASG describe failure, ELB health-check failure, or target-group deregistration failure).","triggerScenarios":"CloudImplementation.DeregisterInstance is called during `kops rolling-update cluster` / instance deletion; deregisterInstance fails because DescribeAutoScalingGroups errors, or the errgroup returns an error from deregisterInstanceFromClassicLoadBalancer (DescribeInstanceHealth API error) or deregisterInstanceFromTargetGroups (DescribeTargetHealth or DeregisterTargets API error).","commonSituations":"Stale or insufficient IAM permissions on elb:DescribeInstanceHealth / elasticloadbalancing:DescribeTargetHealth / DeregisterTargets; ELB or target group already deleted while ASG metadata still references it; throttling during large rolling updates; network/region misconfiguration causing AWS API failures.","solutions":["Inspect the wrapped inner error (%v) to identify whether the ASG describe, ELB health check, or target group deregistration failed","Verify IAM policy grants elb:DescribeInstanceHealth, elb:DeregisterInstancesFromLoadBalancer, elasticloadbalancing:DescribeTargetHealth and elasticloadbalancing:DeregisterTargets","Check that the load balancers/target groups attached to the ASG still exist; remove stale LB references from the ASG if they were deleted","Retry the rolling update — transient throttling or network errors often resolve on retry"],"exampleFix":"// before: guessing at cause from the wrapper\n// failed to deregister instance from loadBalancer before terminating: ...\n// after: log the full wrapped chain and check the inner AWS error code\nif err := cloud.DeregisterInstance(i); err != nil {\n\tklog.Errorf(\"deregister failed: %v\", err) // inspect inner cause, e.g. AccessDenied\n\t// fix IAM: attach elasticloadbalancing:DescribeTargetHealth etc.\n}","handlingStrategy":"try-catch","validationCode":"if i.CloudInstanceGroup.InstanceGroup.Spec.Manager == kops.InstanceManagerKarpenter {\n\treturn nil // deregistration is skipped for Karpenter-managed instances\n}\nif i.ID == \"\" {\n\treturn fmt.Errorf(\"instance has no ID; cannot drain from LBs\")\n}","typeGuard":"func canDeregister(i *cloudinstances.CloudInstance) bool {\n\treturn i != nil && i.CloudInstanceGroup != nil && i.ID != \"\"\n}","tryCatchPattern":"if err := cloud.DeregisterInstance(inst); err != nil {\n\tklog.Errorf(\"drain failed for %s: %v; NOT terminating instance to avoid dropped connections\", inst.ID, err)\n\treturn err // propagate so the rolling update marks the instance unsafe to replace\n}","preventionTips":["Keep kOps IAM policy current with ELB/ELBv2 describe+deregister permissions","Avoid deleting load balancers/target groups out-of-band while a rolling update runs","Watch for throttling on large clusters; run rolling updates with lower concurrency","Verify ASG LB/TG attachments match the kops cluster spec before updating"],"tags":["aws","load-balancer","rolling-update","drain"],"backgroundTag":"load-balancer-deregistration-failed","analyzedSha":"4c8573c808a73d578c5eadc86d410646ea0b0d73","analyzedAt":"2026-09-05T04:13:19.212Z","contentChangedAt":"2026-09-05T04:13:19.212Z","schemaVersion":2},"datasetVersion":"2026-09-12T07:17:12.445Z"}