kubernetes/kops · critical

querying instance metadata: %w

Error message

querying instance metadata: %w

What it means

CreateToken wraps any failure from QueryComputeInstanceMetadata with the "querying instance metadata" prefix. It means the node could not fetch its compute metadata from the Azure IMDS endpoint (http://169.254.169.254/metadata/instance/compute), so a bootstrap token backed by the VM identity cannot be minted. The library throws this because bootstrap authentication requires the VM's Azure resource ID as the node identity.

Source

Thrown at upup/pkg/fi/cloudup/azure/azuremetadata/authenticator.go:53

// NewAzureAuthenticator returns an authenticator that mints Azure bootstrap tokens backed by IMDS
// metadata and an attested document signature.
func NewAzureAuthenticator() (bootstrap.Authenticator, error) {
	return &azureAuthenticator{}, nil
}

// CreateToken fetches the local VM identity from IMDS and returns a bootstrap token containing the
// resource ID and signed attested document.
func (h *azureAuthenticator) CreateToken(body []byte) (string, error) {
	klog.V(4).Infof("Azure authenticator creating bootstrap token")

	// bootstrap.Authenticator.CreateToken carries no context; the IMDS HTTP client's own timeout
	// bounds these calls.
	ctx := context.TODO()

	// Query IMDS for the VM's resource ID.
	metadata, err := QueryComputeInstanceMetadata(ctx)
	if err != nil {
		return "", fmt.Errorf("querying instance metadata: %w", err)
	}
	if metadata.ResourceID == "" {
		return "", fmt.Errorf("missing resource ID")
	}
	klog.V(4).Infof("Azure authenticator obtained resource ID %q", metadata.ResourceID)

	// Query IMDS for a PKCS7-signed attested document containing the nonce.
	nonce := NonceForBody(body)
	doc, err := queryIMDSAttestedDocument(ctx, nonce)
	if err != nil {
		return "", fmt.Errorf("querying attested document: %w", err)
	}
	if doc.Signature == "" {
		return "", fmt.Errorf("empty attested document signature")
	}
	klog.V(2).Infof("Azure authenticator obtained attested document for %q", metadata.ResourceID)

	// Token format: "x-azure-id <resourceID> <base64-pkcs7-signature>"

View on GitHub (pinned to 4c8573c808)

Solutions

1. Verify the node is an Azure VM and IMDS is reachable: curl -H Metadata:true 'http://169.254.169.254/metadata/instance/compute?api-version=2025-04-07&format=json'
2. Check the underlying wrapped error in the message (%w chain) to distinguish network failure vs bad status and fix accordingly
3. If it occurs at boot, add retry/backoff — IMDS can be briefly unavailable after VM start
4. Ensure no NSG/firewall rules block 169.254.169.254 and no broken proxy is configured for the node process
5. Confirm the IMDS api-version used by this kOps build is still supported by Azure

// Check IMDS reachability before invoking bootstrap
func imdsReachable() error {
    c := &http.Client{Timeout: 5 * time.Second, Transport: &http.Transport{Proxy: nil}}
    req, _ := http.NewRequest("GET", "http://169.254.169.254/metadata/instance/compute?api-version=2025-04-07&format=json", nil)
    req.Header.Set("Metadata", "true")
    resp, err := c.Do(req)
    if err != nil { return err }
    defer resp.Body.Close()
    if resp.StatusCode != 200 { return fmt.Errorf("imds status %d", resp.StatusCode) }
    return nil
}

// Retry transient IMDS failures with backoff
var token string
err := wait.PollImmediate(2*time.Second, 2*time.Minute, func() (bool, error) {
    t, err := authenticator.CreateToken(body)
    if err != nil {
        klog.V(2).Infof("retrying CreateToken: %v", err)
        return false, nil
    }
    token = t
    return true, nil
})

When it happens

Trigger: azureAuthenticator.CreateToken is invoked during node bootstrap (nodeup) and the inner queryIMDS call to /metadata/instance/compute fails: request creation error, HTTP client error (timeout, connection refused), non-200 status, body read error, or JSON unmarshal error.

Common situations: Running on a non-Azure VM or a local/dev environment where 169.254.169.254 is unroutable; NSG or firewall rules blocking the link-local IMDS address; IMDS throttling (429); a proxy env var interfering (though the client bypasses proxies); IMDS API version retired; transient IMDS unavailability right after VM boot.

Related errors

• failed to get region from ec2 metadata: %w
• missing resource ID
• querying attested document: %w
• querying IMDS %s: %w
• reading IMDS response: %w

AI-assisted analysis of kubernetes/kops@4c8573c808 (2026-09-05). Data as JSON: /api/errors/6f5ead8cf219c9de. Report an issue: GitHub.