hashicorp/nomad · warning

failed to query AWS metadata: %w

Error message

failed to query AWS metadata: %w

What it means

awsProbe queries IMDS for the ami-id path to confirm the node runs on EC2. If client.GetMetadata returns any error (transport failure, 404/403, IMDSv2 token denial, timeout), the error is wrapped with this message and the AWS fingerprint treats the host as not-AWS or unreachable.

Source

Thrown at client/fingerprint/env_aws.go:284

		config.WithRetryMaxAttempts(0),
	)
	if err != nil {
		return nil, err
	}

	imdsClient := imds.NewFromConfig(cfg, func(o *imds.Options) {
		// endpoint should only be overridden for testing
		if f.endpoint != "" {
			o.Endpoint = f.endpoint
		}
	})
	return imdsClient, nil
}

func awsProbe(ctx context.Context, client *imds.Client) error {
	resp, err := client.GetMetadata(ctx, &imds.GetMetadataInput{Path: "ami-id"})
	if err != nil {
		return fmt.Errorf("failed to query AWS metadata: %w", err)
	}

	s, err := readMetadataResponse(resp)
	if err != nil {
		return fmt.Errorf("failed to read respose: %w", err)
	}

	if s == "" {
		return errors.New("empty response from AWS metadata")
	}

	return nil
}

// readImdsResponse reads and formats the IMDS response
// and most importantly, closes the io.ReadCloser
func readMetadataResponse(resp *imds.GetMetadataOutput) (string, error) {
	defer resp.Content.Close()

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Test curl http://169.254.169.254/latest/meta-data/ami-id from the host
2. Raise IMDSv2 hop limit (aws ec2 modify-instance-metadata-options --http-put-response-hop-limit 2)
3. Remove/disable the aws fingerprint in client options if not on EC2
4. Check firewall/iptables rules for link-local traffic
5. Retry — transient metadata service outages resolve on agent restart

Example fix

# before: fingerprint fails, node not detected as AWS
# after: allow IMDSv2 through container bridge
aws ec2 modify-instance-metadata-options --instance-id i-123 --http-put-response-hop-limit 2 --http-tokens required

ctx, cancel := context.WithTimeout(context.Background(), 2*time.Second)
req, _ := http.NewRequestWithContext(ctx, "GET", "http://169.254.169.254/latest/meta-data/ami-id", nil)
req.Header.Set("X-aws-ec2-metadata-token-ttl-seconds", "21600")
if _, err := http.DefaultClient.Do(req); err != nil { /* skip aws fingerprint */ }

err := awsProbe(ctx, imdsClient)
if err != nil {
    var rerr *retry.Error
    if errors.As(err, &rerr) && rerr.HTTPStatusCode() == 404 { /* not EC2 — disable fingerprint */ }
    return wrapProbeError(err)
}

When it happens

Trigger: IMDS unreachable (not on EC2, network firewall blocking link-local, iptables rules), IMDSv2 hop-limit too low, request timeout, or metadata service disabled on the instance.

Common situations: Running Nomad in a non-AWS environment where the fingerprinter still probes; security-hardened images disabling IMDS; container networking dropping 169.254.169.254; transient network failures during startup.

Related errors

• empty response from AWS metadata
• failed to setup IMDS client: %v
• failed to read respose: %w
• network namespace already exists but was misconfigured
• network already configured but not found in state

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/0948ce1bcd99de84. Report an issue: GitHub.