kubernetes/kops · error

failed to complete lifecylce action: %w

Error message

failed to complete lifecylce action: %w

What it means

nodeup's Run() wraps any failure from completeWarmingLifecycleAction with this message when EnableLifecycleHook is set on AWS. It means the node finished bootstrapping but could not signal the ASG 'CONTINUE' for the kops-warmpool lifecycle hook, so the instance remains in aws:autoscaling:startup wait state and will eventually time out and be terminated/replaced by the ASG.

Source

Thrown at upup/pkg/fi/nodeup/command.go:402

	// Return rather than exit, so that the retry loop in cmd/nodeup gets to run:
	// kops-configuration.service is Type=oneshot, so a bootstrap that exits here is never
	// retried and the node never joins the cluster.
	err = context.RunTasks(options)
	if err != nil {
		return fmt.Errorf("error running tasks: %w", err)
	}

	err = target.Finish(taskMap)
	if err != nil {
		return fmt.Errorf("error closing target: %w", err)
	}

	if nodeupConfig.EnableLifecycleHook {
		if bootConfig.CloudProvider == api.CloudProviderAWS {
			err := completeWarmingLifecycleAction(ctx, cloud, modelContext)
			if err != nil {
				return fmt.Errorf("failed to complete lifecylce action: %w", err)
			}
		}
	}
	return nil
}

func getMachineType(ctx context.Context) (string, error) {
	config, err := awsconfig.LoadDefaultConfig(ctx)
	if err != nil {
		return "", fmt.Errorf("failed to load AWS config: %w", err)
	}

	metadata := imds.NewFromConfig(config)

	// Get the actual instance type by querying the EC2 instance metadata service.
	result, err := metadata.GetMetadata(ctx, &imds.GetMetadataInput{
		Path: "instance-type",
	})

View on GitHub (pinned to 4c8573c808)

Solutions

1. Verify the instance's IAM role grants autoscaling:DescribeLifecycleHooks and autoscaling:CompleteLifecycleAction on the cluster's ASGs.
2. Confirm the ASG lifecycle hook named 'kops-warmpool' exists on the ASG (aws autoscaling describe-lifecycle-hooks --auto-scaling-group-name <ig>.<cluster>).
3. Check cluster spec enables warm-pool lifecycle hooks so kOps creates the hook that nodeup expects.
4. Inspect the wrapped cause (%w) in the nodeup log to distinguish IAM/denied errors from not-found errors, then fix accordingly.

Example fix

// before: no autoscaling permissions on node role
// after: attach policy to instance profile
{
  "Effect": "Allow",
  "Action": ["autoscaling:DescribeLifecycleHooks", "autoscaling:CompleteLifecycleAction"],
  "Resource": "*"
}

aws autoscaling describe-lifecycle-hooks --auto-scaling-group-name <ig>.<cluster> --lifecycle-hook-names kops-warmpool
aws iam simulate-principal-policy --policy-source-arn <nodeRoleArn> --action-names autoscaling:CompleteLifecycleAction autoscaling:DescribeLifecycleHooks

err := runNodeup(ctx)
if err != nil && strings.Contains(err.Error(), "failed to complete lifecycle") {
    // instance will sit in startup-wait; retry nodeup or complete the action manually:
    // aws autoscaling complete-lifecycle-action --lifecycle-action-result CONTINUE ...
    return fmt.Errorf("lifecycle signal failed, instance will time out in ASG wait state: %w", err)
}

When it happens

Trigger: nodeupConfig.EnableLifecycleHook==true AND CloudProvider==AWS AND completeWarmingLifecycleAction returns an error (DescribeLifecycleHooks or CompleteLifecycleAction API failure, ASG name mismatch, or missing IAM permission autoscaling:DescribeLifecycleHooks/CompleteLifecycleAction).

Common situations: Instance profile lacks autoscaling:CompleteLifecycleAction permission; the ASG name built as <instancegroup>.<cluster> does not match the real ASG; the hook 'kops-warmpool' was not created because the cluster spec lacks the warm pool lifecycle configuration; IMDS/network outage breaking the AWS SDK call.

Related errors

• error listing ASG Lifecycle Hooks: %v
• found multiple ASG Lifecycle Hooks with the same name
• failed to find lifecycle hook %q: %w
• failed to complete lifecycle hook %q for %q: %v
• DIGITALOCEAN_ACCESS_TOKEN is required

AI-assisted analysis of kubernetes/kops@4c8573c808 (2026-09-05). Data as JSON: /api/errors/655e58b800634943. Report an issue: GitHub.