{"record":{"id":"9085a4bb5cc0b8a0","repo":"kubernetes/kops","slug":"failed-to-complete-lifecycle-hook-q-for-q-v","errorCode":null,"errorMessage":"failed to complete lifecycle hook %q for %q: %v","messagePattern":"failed to complete lifecycle hook %q for %q: (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"upup/pkg/fi/nodeup/command.go","lineNumber":452,"sourceCode":"\thookName := \"kops-warmpool\"\n\thooks, err := cloud.DescribeLifecycleHooks(ctx, &autoscaling.DescribeLifecycleHooksInput{\n\t\tAutoScalingGroupName: &asgName,\n\t\tLifecycleHookNames:   []string{hookName},\n\t})\n\tif err != nil {\n\t\treturn fmt.Errorf(\"failed to find lifecycle hook %q: %w\", hookName, err)\n\t}\n\n\tif len(hooks.LifecycleHooks) > 0 {\n\t\tklog.Info(\"Found ASG lifecycle hook\")\n\t\t_, err := cloud.CompleteLifecycleAction(ctx, &autoscaling.CompleteLifecycleActionInput{\n\t\t\tAutoScalingGroupName:  &asgName,\n\t\t\tInstanceId:            &modelContext.InstanceID,\n\t\t\tLifecycleHookName:     &hookName,\n\t\t\tLifecycleActionResult: new(\"CONTINUE\"),\n\t\t})\n\t\tif err != nil {\n\t\t\treturn fmt.Errorf(\"failed to complete lifecycle hook %q for %q: %v\", hookName, modelContext.InstanceID, err)\n\t\t}\n\t\tklog.Info(\"Lifecycle action completed\")\n\t} else {\n\t\tklog.Info(\"No ASG lifecycle hook found\")\n\t}\n\treturn nil\n}\n\nfunc evaluateSpec(nodeupConfig *nodeup.Config, cloudProvider api.CloudProviderID, region string) error {\n\thostnameOverride, err := evaluateHostnameOverride(cloudProvider, nodeupConfig.UseIPBasedNodeNames, region)\n\tif err != nil {\n\t\treturn err\n\t}\n\n\tnodeupConfig.KubeletConfig.HostnameOverride = hostnameOverride\n\n\tif nodeupConfig.KubeProxy != nil {\n\t\tnodeupConfig.KubeProxy.HostnameOverride = hostnameOverride","sourceCodeStart":434,"sourceCodeEnd":470,"githubUrl":"https://github.com/kubernetes/kops/blob/4c8573c808a73d578c5eadc86d410646ea0b0d73/upup/pkg/fi/nodeup/command.go#L434-L470","documentation":"The 'kops-warmpool' hook exists, and nodeup tried to complete it with result CONTINUE for the instance, but CompleteLifecycleAction returned an error. The instance stays in the startup wait state and the ASG will abandon/terminate it when the hook timeout expires. Note this message uses %v, so the cause is not error-wrapped.","triggerScenarios":"cloud.CompleteLifecycleAction fails: IAM denied autoscaling:CompleteLifecycleAction, the instance is not actually associated with a waiting lifecycle action (e.g. hook already completed/timed out, instance not part of the ASG), or the ASG name/instance ID don't match AWS state.","commonSituations":"Hook heartbeat timeout elapsed before nodeup ran, so AWS already ABANDONED the action; instance ID missing from modelContext (empty InstanceID) making the call invalid; node role missing CompleteLifecycleAction; nodeup run outside ASG context (e.g. manually on a plain EC2).","solutions":["Check AWS events / ASG activity: if the hook already timed out and fired ABANDON, increase the hook's HeartbeatTimeout in the cluster spec.","Grant the instance role autoscaling:CompleteLifecycleAction.","Verify modelContext.InstanceID is populated (IMDS instance-id readable) so the API call identifies the right instance.","Inspect the %v-attached AWS error code: ValidationError usually means no pending lifecycle action for that instance."],"exampleFix":"// before: hook heartbeat too short, action already abandoned\n// after: extend timeout on the lifecycle hook\naws autoscaling put-lifecycle-hook --lifecycle-hook-name kops-warmpool \\\n  --auto-scaling-group-name nodes.cluster --heartbeat-timeout 900 \\\n  --lifecycle-transition autoscaling:EC2_INSTANCE_LAUNCHING","handlingStrategy":"try-catch","validationCode":"aws autoscaling describe-auto-scaling-instances --instance-ids i-xxx --query 'AutoScalingInstances[].LifecycleState'\naws iam simulate-principal-policy --policy-source-arn <nodeRoleArn> --action-names autoscaling:CompleteLifecycleAction","typeGuard":null,"tryCatchPattern":"_, err := cloud.CompleteLifecycleAction(ctx, input)\nif err != nil {\n    var ae smithy.APIError\n    if errors.As(err, &ae) && ae.ErrorCode() == \"ValidationError\" {\n        // no pending lifecycle action for this instance: likely already\n        // completed or abandoned after heartbeat timeout; log and continue\n    }\n    return fmt.Errorf(\"failed to complete lifecycle hook: %w\", err)\n}","preventionTips":["Set HeartbeatTimeout comfortably above nodeup's worst-case runtime.","Grant CompleteLifecycleAction to the node instance profile.","Ensure InstanceID is populated from IMDS before signaling.","Only signal when the ASG lifecycle state is 'Pending:Wait'."],"tags":["aws","autoscaling","lifecycle-hook","iam"],"backgroundTag":"lifecycle-hook-complete-failed","analyzedSha":"4c8573c808a73d578c5eadc86d410646ea0b0d73","analyzedAt":"2026-09-05T04:13:19.212Z","contentChangedAt":"2026-09-05T04:13:19.212Z","schemaVersion":2},"datasetVersion":"2026-09-12T12:17:11.808Z"}