kubernetes/kops · error

error reading from AWS metadata service: %v

Error message

error reading from AWS metadata service: %v

What it means

After loading config, readAWSMetadata calls the EC2 IMDS client GetMetadata for /meta-data/ paths. Any error from the instance metadata service (unreachable endpoint, timeout, throttling, token failure) is wrapped as 'error reading from AWS metadata service'.

Source

Thrown at util/pkg/vfs/context.go:239

	return nil, fmt.Errorf("unknown / unhandled path type: %q", p)
}

// readAWSMetadata reads the specified path from the AWS EC2 metadata service
func (c *VFSContext) readAWSMetadata(ctx context.Context, path string) ([]byte, error) {
	config, err := awsconfig.LoadDefaultConfig(ctx)
	if err != nil {
		return nil, fmt.Errorf("failed to load AWS config: %w", err)
	}

	client := imds.NewFromConfig(config)

	if strings.HasPrefix(path, "/meta-data/") {
		s, err := client.GetMetadata(ctx, &imds.GetMetadataInput{
			Path: strings.TrimPrefix(path, "/meta-data/"),
		})
		if err != nil {
			return nil, fmt.Errorf("error reading from AWS metadata service: %v", err)
		}
		defer s.Content.Close()
		return io.ReadAll(s.Content)
	}
	// There are others (e.g. user-data), but as we don't use them yet let's not expose them
	return nil, fmt.Errorf("unhandled aws metadata path %q", path)
}

// readHTTPLocation reads an http (or https) url.
// It returns the contents, or an error on any non-200 response.  On a 404, it will return os.ErrNotExist
// It will retry a few times on a 500 class error
func (c *VFSContext) readHTTPLocation(httpURL string, httpHeaders map[string]string, opts vfsOptions) ([]byte, error) {
	var body []byte

	done, err := RetryWithBackoff(opts.backoff, func() (bool, error) {
		klog.V(4).Infof("Performing HTTP request: GET %s", httpURL)
		req, err := http.NewRequest("GET", httpURL, nil)
		if err != nil {

View on GitHub (pinned to 4c8573c808)

Solutions

1. Verify IMDS is reachable: `curl -s http://169.254.169.254/latest/meta-data/` from the instance
2. Check the instance metadata options (aws ec2 describe-instances / modify-instance-metadata-options) and re-enable metadata access if disabled
3. Confirm you are actually running on an EC2 instance - this scheme only works on AWS
4. Retry on 5xx/throttling; the http path already retries, but IMDS errors here are returned immediately

Example fix

// before
// running kops on a laptop
kops.ReadFile("metadata://aws/meta-data/instance-id")
// after
if onEC2, _ := isEC2(); onEC2 {
	kops.ReadFile("metadata://aws/meta-data/instance-id")
} else {
	kops.ReadFile("file:///etc/instance-id")
}

func imdsReachable() bool {
	client := &http.Client{Timeout: 2 * time.Second}
	resp, err := client.Get("http://169.254.169.254/latest/meta-data/")
	return err == nil && resp.StatusCode == 200
}

func runningOnEC2() bool { return imdsReachable() }

var imdsBackoff = wait.Backoff{Duration: 500 * time.Millisecond, Factor: 2, Steps: 6}
data, err := vfs.RetryWithBackoff(imdsBackoff, func() (bool, error) {
	b, err := vfs.Context.ReadFile("metadata://aws/meta-data/instance-id")
	if err != nil {
		if strings.Contains(err.Error(), "error reading from AWS metadata service") {
			return false, err // retry transient IMDS failures
		}
		return true, err
	}
	return true, nil
})
_ = data

When it happens

Trigger: ReadFile('metadata://aws/meta-data/<key>') when the IMDS endpoint at 169.254.169.254 is unreachable, IMDSv2 token requests fail, the instance is not on AWS, or the metadata option (e.g. instance-id) is disabled via IMDS settings.

Common situations: Running the code locally/off-EC2 where 169.254.169.254 doesn't route; IMDS hop limit or firewall rules blocking link-local traffic; instance metadata access set to 'disabled' on the instance; transient IMDS throttling under heavy polling.

Related errors

• failed to load AWS config: %w
• failed to get local-ipv4 address from ec2 metadata: %w
• error reading local-ipv4 from AWS metadata: %v
• error querying ec2 metadata service (for region): %v
• getting primary MAC address from ec2 metadata: %w

AI-assisted analysis of kubernetes/kops@4c8573c808 (2026-09-05). Data as JSON: /api/errors/0c5974c3e269b432. Report an issue: GitHub.