{"record":{"id":"cfb0420ddedb474c","repo":"kubernetes/kops","slug":"error-querying-ec2-metadata-service-for-region","errorCode":null,"errorMessage":"error querying ec2 metadata service (for region): %v","messagePattern":"error querying ec2 metadata service \\(for region\\): (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"critical","filePath":"cmd/kops-controller/controllers/awsipam.go","lineNumber":67,"sourceCode":"\t\tlog:    ctrl.Log.WithName(\"controllers\").WithName(\"IPAM\"),\n\t}\n\n\tcoreClient, err := corev1client.NewForConfig(mgr.GetConfig())\n\tif err != nil {\n\t\treturn nil, fmt.Errorf(\"error building corev1 client: %v\", err)\n\t}\n\tr.coreV1Client = coreClient\n\n\tconfig, err := awsconfig.LoadDefaultConfig(ctx, awslog.WithAWSLogger())\n\tif err != nil {\n\t\treturn nil, fmt.Errorf(\"error loading default AWS config: %v\", err)\n\t}\n\n\tmetadata := imds.NewFromConfig(config)\n\n\tresp, err := metadata.GetRegion(ctx, &imds.GetRegionInput{})\n\tif err != nil {\n\t\treturn nil, fmt.Errorf(\"error querying ec2 metadata service (for region): %v\", err)\n\t}\n\n\tec2Config := config.Copy()\n\tec2Config.Region = resp.Region\n\tr.ec2Client = ec2.NewFromConfig(ec2Config)\n\n\treturn r, nil\n}\n\n// AWSIPAMReconciler observes Node objects, and labels them with the correct labels for the instancegroup\n// This used to be done by the kubelet, but is moving to a central controller for greater security in 1.16\ntype AWSIPAMReconciler struct {\n\t// client is the controller-runtime client\n\tclient client.Client\n\n\t// log is a logr\n\tlog logr.Logger\n","sourceCodeStart":49,"sourceCodeEnd":85,"githubUrl":"https://github.com/kubernetes/kops/blob/4c8573c808a73d578c5eadc86d410646ea0b0d73/cmd/kops-controller/controllers/awsipam.go#L49-L85","documentation":"NewAWSIPAMReconciler calls the EC2 Instance Metadata Service (IMDS) GetRegion API to discover which AWS region the controller pod is running in, so it can configure the EC2 client. This error is returned when the IMDS HTTP query fails at startup (constructor returns nil reconciler and the controller never starts). It wraps the underlying aws-sdk-go-v2 imds error, which may be a network timeout, a 404/connection failure, or an empty region.","triggerScenarios":"Calling metadata.GetRegion(ctx, &imds.GetRegionInput{}) fails: the pod has no route to 169.254.169.254, the node is not an EC2 instance (bare metal, non-AWS test environment), IMDSv2 hop limit is too low for containers (default 1 blocks pod traffic through a bridge/overlay), or the IMDS endpoint returns no region (e.g. placeholder metadata on non-AWS environments).","commonSituations":"Running kops-controller locally or in CI where there is no EC2 metadata endpoint; pods on EKS with IMDSv2 enabled and hop-limit 1 (container sees no response due to NAT hop count); security group / iptables rules blocking 169.254.169.254; running the controller on non-EC2 nodes such as on-premises or other clouds with a non-AWS placeholder metadata service (e.g. 169.254.169.254 serving GCE/Azure metadata).","solutions":["Ensure the controller runs on an EC2 instance with a working IMDS endpoint; curl -s http://169.254.169.254/latest/meta-data/placement/region from the pod/node to verify.","If using IMDSv2, raise the hop limit to 2 (aws ec2 modify-instance-metadata-options --http-put-response-hop-limit 2) so containerized pods can reach IMDS.","Check network configuration: no iptables/network policy blocking 169.254.169.254 from the pod, and the pod is not on a host without EC2 metadata (use static region config instead if not on AWS).","If the metadata service is not available, hardcode/set the AWS_REGION environment variable and refactor the reconciler to skip IMDS discovery."],"exampleFix":"// before\nresp, err := metadata.GetRegion(ctx, &imds.GetRegionInput{})\nif err != nil {\n\treturn nil, fmt.Errorf(\"error querying ec2 metadata service (for region): %v\", err)\n}\n// after\nregion := os.Getenv(\"AWS_REGION\")\nif region == \"\" {\n\tresp, err := metadata.GetRegion(ctx, &imds.GetRegionInput{})\n\tif err != nil {\n\t\treturn nil, fmt.Errorf(\"error querying ec2 metadata service (for region): %v\", err)\n\t}\n\tregion = resp.Region\n}","handlingStrategy":"fallback","validationCode":"if !isEC2Environment() { /* skip IMDS, use AWS_REGION */ }\nfunc isEC2Environment() bool {\n\t_, err := http.Get(\"http://169.254.169.254/latest/api/token\")\n\treturn err == nil\n}","typeGuard":"func imdsRegionValid(resp *imds.GetRegionOutput) bool {\n\treturn resp != nil && resp.Region != \"\"\n}","tryCatchPattern":"r, err := NewAWSIPAMReconciler(ctx, mgr)\nif err != nil {\n\tvar terr *retry.Error // or inspect for timeout/connection\n\tif errors.As(err, &terr) || strings.Contains(err.Error(), \"timeout\") {\n\t\tklog.Errorf(\"IMDS unreachable (metadata service?): %v\", err)\n\t}\n\treturn err\n}","preventionTips":["Always run the controller on EC2 instances with reachable IMDS; verify with curl to 169.254.169.254 before deploying.","Set IMDSv2 hop limit to 2 when the controller runs in a container (modify-instance-metadata-options).","Provide AWS_REGION as an env fallback so IMDS is not a hard startup dependency.","Check that security groups / network policies never block link-local 169.254.169.254 from the controller pods."],"tags":["aws","imds","network","controller-startup"],"backgroundTag":"ec2-metadata-service-unreachable","analyzedSha":"4c8573c808a73d578c5eadc86d410646ea0b0d73","analyzedAt":"2026-09-05T04:13:19.212Z","contentChangedAt":"2026-09-05T04:13:19.212Z","schemaVersion":2},"datasetVersion":"2026-09-12T07:17:12.445Z"}