{"record":{"id":"d095e0c810b9fa1c","repo":"kubernetes/kops","slug":"error-querying-ec2-for-user-metadata-for-instance","errorCode":null,"errorMessage":"error querying EC2 for user metadata for instance %q: %v","messagePattern":"error querying EC2 for user metadata for instance %q: (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"upup/pkg/fi/cloudup/awstasks/instance.go","lineNumber":128,"sourceCode":"\t\treturn nil, fmt.Errorf(\"found instance, but InstanceId was nil\")\n\t}\n\n\tactual := &Instance{\n\t\tID:               i.InstanceId,\n\t\tPrivateIPAddress: i.PrivateIpAddress,\n\t\tInstanceType:     i.InstanceType,\n\t\tImageID:          i.ImageId,\n\t\tName:             findNameTag(i.Tags),\n\t}\n\n\t// Fetch instance UserData\n\t{\n\t\trequest := &ec2.DescribeInstanceAttributeInput{}\n\t\trequest.InstanceId = i.InstanceId\n\t\trequest.Attribute = ec2types.InstanceAttributeNameUserData\n\t\tresponse, err := cloud.EC2().DescribeInstanceAttribute(ctx, request)\n\t\tif err != nil {\n\t\t\treturn nil, fmt.Errorf(\"error querying EC2 for user metadata for instance %q: %v\", *i.InstanceId, err)\n\t\t}\n\t\tif response.UserData != nil {\n\t\t\tb, err := base64.StdEncoding.DecodeString(aws.ToString(response.UserData.Value))\n\t\t\tif err != nil {\n\t\t\t\treturn nil, fmt.Errorf(\"error decoding EC2 UserData: %v\", err)\n\t\t\t}\n\t\t\tactual.UserData = fi.NewBytesResource(b)\n\t\t}\n\t}\n\n\tif i.SubnetId != nil {\n\t\tactual.Subnet = &Subnet{ID: i.SubnetId}\n\t}\n\tif i.KeyName != nil {\n\t\tactual.SSHKey = &SSHKey{Name: i.KeyName}\n\t}\n\n\tfor _, sg := range i.SecurityGroups {","sourceCodeStart":110,"sourceCodeEnd":146,"githubUrl":"https://github.com/kubernetes/kops/blob/4c8573c808a73d578c5eadc86d410646ea0b0d73/upup/pkg/fi/cloudup/awstasks/instance.go#L110-L146","documentation":"After locating the instance, Find() fetches its UserData via DescribeInstanceAttribute (Attribute=userData). Any SDK error from that call is wrapped with the instance ID for context. Without UserData, kOps cannot compare the desired node config to the actual instance, so reconciliation stops for this task.","triggerScenarios":"cloud.EC2().DescribeInstanceAttribute(ctx, request) fails for the discovered instance: UnauthorizedOperation (missing ec2:DescribeInstanceAttribute), throttling, invalid instance ID (deleted concurrently), or connectivity/auth issues.","commonSituations":"IAM policy for kops missing ec2:DescribeInstanceAttribute (common with hand-rolled least-privilege policies); the instance was terminated between describe and attribute query; API throttling on big fleets.","solutions":["Add ec2:DescribeInstanceAttribute to the kops IAM policy (the stock kops policy already includes it).","Check the wrapped AWS error; if the instance no longer exists, re-run apply so kops recreates it.","Retry after backoff if throttled.","Verify credentials/region if the error is auth-related."],"exampleFix":"// before: IAM policy without DescribeInstanceAttribute\n{\"Action\":[\"ec2:DescribeInstances\"], ...}\n// after\n{\"Action\":[\"ec2:DescribeInstances\",\"ec2:DescribeInstanceAttribute\"], ...}","handlingStrategy":"retry","validationCode":"// verify the operator role can read instance attributes before apply\nsim, err := iam.SimulatePrincipalPolicy(&iam.SimulatePrincipalPolicyInput{\n  PolicySourceArn: aws.String(callerArn), ActionNames: []string{\"ec2:DescribeInstanceAttribute\"}})\n// check sim.EvaluationResults[0].EvaluateDecision == \"allowed\"","typeGuard":"var ae smithy.APIError\nif errors.As(err, &ae) {\n  switch ae.ErrorCode() {\n  case \"UnauthorizedOperation\": // IAM missing permission\n  case \"Throttling\": // retry\n  case \"InvalidInstanceID.NotFound\": // instance gone; recreate\n  }\n}","tryCatchPattern":"var ae smithy.APIError\nif errors.As(err, &ae) {\n  if ae.ErrorCode() == \"Throttling\" {\n    return retryWithBackoff(op, 5, time.Second)\n  }\n  if ae.ErrorCode() == \"InvalidInstanceID.NotFound\" {\n    return recreateInstance() // instance terminated mid-reconcile\n  }\n}\nreturn err","preventionTips":["Use the default kops IAM policy, which includes ec2:DescribeInstanceAttribute.","Never hand-trim ec2:Describe* actions from the kops role.","Handle instance churn: expect instances to disappear during rolling updates and re-apply."],"tags":["aws","ec2","iam-permissions","api-error"],"backgroundTag":"aws-api-call-failed","analyzedSha":"4c8573c808a73d578c5eadc86d410646ea0b0d73","analyzedAt":"2026-09-05T04:13:19.212Z","contentChangedAt":"2026-09-05T04:13:19.212Z","schemaVersion":2},"datasetVersion":"2026-09-12T07:17:12.445Z"}