kubernetes/kops · error

failed to complete lifecycle hook %q for %q: %v

Error message

failed to complete lifecycle hook %q for %q: %v

What it means

The 'kops-warmpool' hook exists, and nodeup tried to complete it with result CONTINUE for the instance, but CompleteLifecycleAction returned an error. The instance stays in the startup wait state and the ASG will abandon/terminate it when the hook timeout expires. Note this message uses %v, so the cause is not error-wrapped.

Source

Thrown at upup/pkg/fi/nodeup/command.go:452

	hookName := "kops-warmpool"
	hooks, err := cloud.DescribeLifecycleHooks(ctx, &autoscaling.DescribeLifecycleHooksInput{
		AutoScalingGroupName: &asgName,
		LifecycleHookNames:   []string{hookName},
	})
	if err != nil {
		return fmt.Errorf("failed to find lifecycle hook %q: %w", hookName, err)
	}

	if len(hooks.LifecycleHooks) > 0 {
		klog.Info("Found ASG lifecycle hook")
		_, err := cloud.CompleteLifecycleAction(ctx, &autoscaling.CompleteLifecycleActionInput{
			AutoScalingGroupName:  &asgName,
			InstanceId:            &modelContext.InstanceID,
			LifecycleHookName:     &hookName,
			LifecycleActionResult: new("CONTINUE"),
		})
		if err != nil {
			return fmt.Errorf("failed to complete lifecycle hook %q for %q: %v", hookName, modelContext.InstanceID, err)
		}
		klog.Info("Lifecycle action completed")
	} else {
		klog.Info("No ASG lifecycle hook found")
	}
	return nil
}

func evaluateSpec(nodeupConfig *nodeup.Config, cloudProvider api.CloudProviderID, region string) error {
	hostnameOverride, err := evaluateHostnameOverride(cloudProvider, nodeupConfig.UseIPBasedNodeNames, region)
	if err != nil {
		return err
	}

	nodeupConfig.KubeletConfig.HostnameOverride = hostnameOverride

	if nodeupConfig.KubeProxy != nil {
		nodeupConfig.KubeProxy.HostnameOverride = hostnameOverride

View on GitHub (pinned to 4c8573c808)

Solutions

1. Check AWS events / ASG activity: if the hook already timed out and fired ABANDON, increase the hook's HeartbeatTimeout in the cluster spec.
2. Grant the instance role autoscaling:CompleteLifecycleAction.
3. Verify modelContext.InstanceID is populated (IMDS instance-id readable) so the API call identifies the right instance.
4. Inspect the %v-attached AWS error code: ValidationError usually means no pending lifecycle action for that instance.

Example fix

// before: hook heartbeat too short, action already abandoned
// after: extend timeout on the lifecycle hook
aws autoscaling put-lifecycle-hook --lifecycle-hook-name kops-warmpool \
  --auto-scaling-group-name nodes.cluster --heartbeat-timeout 900 \
  --lifecycle-transition autoscaling:EC2_INSTANCE_LAUNCHING

aws autoscaling describe-auto-scaling-instances --instance-ids i-xxx --query 'AutoScalingInstances[].LifecycleState'
aws iam simulate-principal-policy --policy-source-arn <nodeRoleArn> --action-names autoscaling:CompleteLifecycleAction

_, err := cloud.CompleteLifecycleAction(ctx, input)
if err != nil {
    var ae smithy.APIError
    if errors.As(err, &ae) && ae.ErrorCode() == "ValidationError" {
        // no pending lifecycle action for this instance: likely already
        // completed or abandoned after heartbeat timeout; log and continue
    }
    return fmt.Errorf("failed to complete lifecycle hook: %w", err)
}

When it happens

Trigger: cloud.CompleteLifecycleAction fails: IAM denied autoscaling:CompleteLifecycleAction, the instance is not actually associated with a waiting lifecycle action (e.g. hook already completed/timed out, instance not part of the ASG), or the ASG name/instance ID don't match AWS state.

Common situations: Hook heartbeat timeout elapsed before nodeup ran, so AWS already ABANDONED the action; instance ID missing from modelContext (empty InstanceID) making the call invalid; node role missing CompleteLifecycleAction; nodeup run outside ASG context (e.g. manually on a plain EC2).

Related errors

• failed to find lifecycle hook %q: %w
• IP version is incorrect
• provider ID cannot be empty
• provider ID number cannot be empty
• error creating AutoScalingGroup: %s

AI-assisted analysis of kubernetes/kops@4c8573c808 (2026-09-05). Data as JSON: /api/errors/9085a4bb5cc0b8a0. Report an issue: GitHub.