kubernetes/kops · critical

error querying ec2 metadata service (for region): %v

Error message

error querying ec2 metadata service (for region): %v

What it means

NewAWSIPAMReconciler calls the EC2 Instance Metadata Service (IMDS) GetRegion API to discover which AWS region the controller pod is running in, so it can configure the EC2 client. This error is returned when the IMDS HTTP query fails at startup (constructor returns nil reconciler and the controller never starts). It wraps the underlying aws-sdk-go-v2 imds error, which may be a network timeout, a 404/connection failure, or an empty region.

Source

Thrown at cmd/kops-controller/controllers/awsipam.go:67

		log:    ctrl.Log.WithName("controllers").WithName("IPAM"),
	}

	coreClient, err := corev1client.NewForConfig(mgr.GetConfig())
	if err != nil {
		return nil, fmt.Errorf("error building corev1 client: %v", err)
	}
	r.coreV1Client = coreClient

	config, err := awsconfig.LoadDefaultConfig(ctx, awslog.WithAWSLogger())
	if err != nil {
		return nil, fmt.Errorf("error loading default AWS config: %v", err)
	}

	metadata := imds.NewFromConfig(config)

	resp, err := metadata.GetRegion(ctx, &imds.GetRegionInput{})
	if err != nil {
		return nil, fmt.Errorf("error querying ec2 metadata service (for region): %v", err)
	}

	ec2Config := config.Copy()
	ec2Config.Region = resp.Region
	r.ec2Client = ec2.NewFromConfig(ec2Config)

	return r, nil
}

// AWSIPAMReconciler observes Node objects, and labels them with the correct labels for the instancegroup
// This used to be done by the kubelet, but is moving to a central controller for greater security in 1.16
type AWSIPAMReconciler struct {
	// client is the controller-runtime client
	client client.Client

	// log is a logr
	log logr.Logger

View on GitHub (pinned to 4c8573c808)

Solutions

1. Ensure the controller runs on an EC2 instance with a working IMDS endpoint; curl -s http://169.254.169.254/latest/meta-data/placement/region from the pod/node to verify.
2. If using IMDSv2, raise the hop limit to 2 (aws ec2 modify-instance-metadata-options --http-put-response-hop-limit 2) so containerized pods can reach IMDS.
3. Check network configuration: no iptables/network policy blocking 169.254.169.254 from the pod, and the pod is not on a host without EC2 metadata (use static region config instead if not on AWS).
4. If the metadata service is not available, hardcode/set the AWS_REGION environment variable and refactor the reconciler to skip IMDS discovery.

Example fix

// before
resp, err := metadata.GetRegion(ctx, &imds.GetRegionInput{})
if err != nil {
	return nil, fmt.Errorf("error querying ec2 metadata service (for region): %v", err)
}
// after
region := os.Getenv("AWS_REGION")
if region == "" {
	resp, err := metadata.GetRegion(ctx, &imds.GetRegionInput{})
	if err != nil {
		return nil, fmt.Errorf("error querying ec2 metadata service (for region): %v", err)
	}
	region = resp.Region
}

if !isEC2Environment() { /* skip IMDS, use AWS_REGION */ }
func isEC2Environment() bool {
	_, err := http.Get("http://169.254.169.254/latest/api/token")
	return err == nil
}

func imdsRegionValid(resp *imds.GetRegionOutput) bool {
	return resp != nil && resp.Region != ""
}

r, err := NewAWSIPAMReconciler(ctx, mgr)
if err != nil {
	var terr *retry.Error // or inspect for timeout/connection
	if errors.As(err, &terr) || strings.Contains(err.Error(), "timeout") {
		klog.Errorf("IMDS unreachable (metadata service?): %v", err)
	}
	return err
}

When it happens

Trigger: Calling metadata.GetRegion(ctx, &imds.GetRegionInput{}) fails: the pod has no route to 169.254.169.254, the node is not an EC2 instance (bare metal, non-AWS test environment), IMDSv2 hop limit is too low for containers (default 1 blocks pod traffic through a bridge/overlay), or the IMDS endpoint returns no region (e.g. placeholder metadata on non-AWS environments).

Common situations: Running kops-controller locally or in CI where there is no EC2 metadata endpoint; pods on EKS with IMDSv2 enabled and hop-limit 1 (container sees no response due to NAT hop count); security group / iptables rules blocking 169.254.169.254; running the controller on non-EC2 nodes such as on-premises or other clouds with a non-AWS placeholder metadata service (e.g. 169.254.169.254 serving GCE/Azure metadata).

Related errors

• failed to load AWS config: %w
• failed to get local-ipv4 address from ec2 metadata: %w
• getting primary MAC address from ec2 metadata: %w
• reading primary MAC address from ec2 metadata: %w
• failed to get region from ec2 metadata: %w

AI-assisted analysis of kubernetes/kops@4c8573c808 (2026-09-05). Data as JSON: /api/errors/cfb0420ddedb474c. Report an issue: GitHub.