hashicorp/nomad · warning

empty response from AWS metadata

Error message

empty response from AWS metadata

What it means

The AWS environment fingerprint probes the EC2 instance metadata service (IMDS) to detect attributes like the placement availability zone. awsProbe reads a metadata endpoint via readMetadataResponse and rejects an HTTP 200 with an empty body, because a successful but empty response indicates the endpoint exists yet returned no usable data.

Source

Thrown at client/fingerprint/env_aws.go:293

			o.Endpoint = f.endpoint
		}
	})
	return imdsClient, nil
}

func awsProbe(ctx context.Context, client *imds.Client) error {
	resp, err := client.GetMetadata(ctx, &imds.GetMetadataInput{Path: "ami-id"})
	if err != nil {
		return fmt.Errorf("failed to query AWS metadata: %w", err)
	}

	s, err := readMetadataResponse(resp)
	if err != nil {
		return fmt.Errorf("failed to read respose: %w", err)
	}

	if s == "" {
		return errors.New("empty response from AWS metadata")
	}

	return nil
}

// readImdsResponse reads and formats the IMDS response
// and most importantly, closes the io.ReadCloser
func readMetadataResponse(resp *imds.GetMetadataOutput) (string, error) {
	defer resp.Content.Close()

	b, err := io.ReadAll(resp.Content)
	if err != nil {
		return "", err
	}
	return strings.TrimSpace(string(b)), nil
}

// Reload is a no-op but implements ReloadableFingerprint

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Verify IMDS works on the host: curl the specific metadata URL the probe uses (e.g. .../meta-data/placement/availability-zone) and check for a non-empty body
2. Check for proxies, firewalls, or security software intercepting 169.254.169.254 and returning empty 200 responses
3. If the node is not truly AWS (or IMDS is intentionally blocked), disable AWS fingerprinting via client config so the empty probe does not matter
4. Ensure IMDSv2 token handling is functioning if the environment requires token-based requests

Example fix

// before (client config leaves AWS fingerprint enabled on a host with blocked/filtered IMDS)
client { }
// after
client {
  fingerprint {
    "env_aws" { disabled = true }
  }
}

// Pre-check before relying on AWS attributes
resp, err := http.Get("http://169.254.169.254/latest/meta-data/placement/availability-zone")
if err != nil || resp.StatusCode != 200 {
    // treat as non-AWS or IMDS blocked
}
body, _ := io.ReadAll(resp.Body)
if len(strings.TrimSpace(string(body))) == 0 {
    // empty metadata: disable env_aws fingerprint or investigate proxy
}

func hasNonEmptyMetadata(body []byte) bool {
    return len(strings.TrimSpace(string(body))) > 0
}

When it happens

Trigger: Fingerprinting a Nomad client on an AWS EC2 node where the IMDS endpoint (169.254.169.254) returned an empty body — e.g. an empty IMDS document, an empty AZ/attribute response, or a proxy/hypervisor that answers with 200 and no content.

Common situations: Hardened EC2 environments with IMDS filtering, metadata proxies returning empty payloads, or network middleboxes intercepting link-local metadata traffic.

Related errors

• empty AZ Environment value
• empty region value
• failed to query AWS metadata: %w
• network namespace already exists but was misconfigured
• network already configured but not found in state

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/f3dded8aa946f761. Report an issue: GitHub.