kubernetes/kops · error

error querying EC2 for user metadata for instance %q: %v

Error message

error querying EC2 for user metadata for instance %q: %v

What it means

After locating the instance, Find() fetches its UserData via DescribeInstanceAttribute (Attribute=userData). Any SDK error from that call is wrapped with the instance ID for context. Without UserData, kOps cannot compare the desired node config to the actual instance, so reconciliation stops for this task.

Source

Thrown at upup/pkg/fi/cloudup/awstasks/instance.go:128

		return nil, fmt.Errorf("found instance, but InstanceId was nil")
	}

	actual := &Instance{
		ID:               i.InstanceId,
		PrivateIPAddress: i.PrivateIpAddress,
		InstanceType:     i.InstanceType,
		ImageID:          i.ImageId,
		Name:             findNameTag(i.Tags),
	}

	// Fetch instance UserData
	{
		request := &ec2.DescribeInstanceAttributeInput{}
		request.InstanceId = i.InstanceId
		request.Attribute = ec2types.InstanceAttributeNameUserData
		response, err := cloud.EC2().DescribeInstanceAttribute(ctx, request)
		if err != nil {
			return nil, fmt.Errorf("error querying EC2 for user metadata for instance %q: %v", *i.InstanceId, err)
		}
		if response.UserData != nil {
			b, err := base64.StdEncoding.DecodeString(aws.ToString(response.UserData.Value))
			if err != nil {
				return nil, fmt.Errorf("error decoding EC2 UserData: %v", err)
			}
			actual.UserData = fi.NewBytesResource(b)
		}
	}

	if i.SubnetId != nil {
		actual.Subnet = &Subnet{ID: i.SubnetId}
	}
	if i.KeyName != nil {
		actual.SSHKey = &SSHKey{Name: i.KeyName}
	}

	for _, sg := range i.SecurityGroups {

View on GitHub (pinned to 4c8573c808)

Solutions

1. Add ec2:DescribeInstanceAttribute to the kops IAM policy (the stock kops policy already includes it).
2. Check the wrapped AWS error; if the instance no longer exists, re-run apply so kops recreates it.
3. Retry after backoff if throttled.
4. Verify credentials/region if the error is auth-related.

Example fix

// before: IAM policy without DescribeInstanceAttribute
{"Action":["ec2:DescribeInstances"], ...}
// after
{"Action":["ec2:DescribeInstances","ec2:DescribeInstanceAttribute"], ...}

// verify the operator role can read instance attributes before apply
sim, err := iam.SimulatePrincipalPolicy(&iam.SimulatePrincipalPolicyInput{
  PolicySourceArn: aws.String(callerArn), ActionNames: []string{"ec2:DescribeInstanceAttribute"}})
// check sim.EvaluationResults[0].EvaluateDecision == "allowed"

var ae smithy.APIError
if errors.As(err, &ae) {
  switch ae.ErrorCode() {
  case "UnauthorizedOperation": // IAM missing permission
  case "Throttling": // retry
  case "InvalidInstanceID.NotFound": // instance gone; recreate
  }
}

var ae smithy.APIError
if errors.As(err, &ae) {
  if ae.ErrorCode() == "Throttling" {
    return retryWithBackoff(op, 5, time.Second)
  }
  if ae.ErrorCode() == "InvalidInstanceID.NotFound" {
    return recreateInstance() // instance terminated mid-reconcile
  }
}
return err

When it happens

Trigger: cloud.EC2().DescribeInstanceAttribute(ctx, request) fails for the discovered instance: UnauthorizedOperation (missing ec2:DescribeInstanceAttribute), throttling, invalid instance ID (deleted concurrently), or connectivity/auth issues.

Common situations: IAM policy for kops missing ec2:DescribeInstanceAttribute (common with hand-rolled least-privilege policies); the instance was terminated between describe and attribute query; API throttling on big fleets.

Related errors

• describing instance for arn %q
• error terminating instances: %v
• error describing instances: %v
• error describing addresses: %v
• error describing volumes: %v

AI-assisted analysis of kubernetes/kops@4c8573c808 (2026-09-05). Data as JSON: /api/errors/d095e0c810b9fa1c. Report an issue: GitHub.