JuliusBrussee/caveman · error

kms: %s returned HTTP %d

Error message

kms: %s returned HTTP %d

What it means

The KMS HTTP call completed but returned a status other than 200; the body (up to 32 KiB) is discarded and the status is reported. Because responses are strict, even 2xx variants other than 200 fail. The status code is the diagnostic: 401/403 credential or permission, 404 wrong region or key ID, 429 rate limiting, 5xx provider-side incident.

Source

Thrown at shared/platform/kms/kms.go:367

		return fmt.Errorf("kms: encode %s request: %w", operation, err)
	}
	endpoint := c.apiBaseURL + "/key-manager/v1alpha1/regions/" + url.PathEscape(region) +
		"/keys/" + url.PathEscape(keyID) + "/" + operation
	req, err := http.NewRequestWithContext(ctx, http.MethodPost, endpoint, bytes.NewReader(body))
	if err != nil {
		return fmt.Errorf("kms: create %s request: %w", operation, err)
	}
	req.Header.Set("content-type", "application/json")
	req.Header.Set("accept", "application/json")
	req.Header.Set("x-auth-token", c.token)
	resp, err := c.httpClient.Do(req)
	if err != nil {
		return fmt.Errorf("kms: %s request failed: %w", operation, err)
	}
	defer resp.Body.Close()
	if resp.StatusCode != http.StatusOK {
		_, _ = io.Copy(io.Discard, io.LimitReader(resp.Body, 32<<10))
		return fmt.Errorf("kms: %s returned HTTP %d", operation, resp.StatusCode)
	}
	data, err := io.ReadAll(io.LimitReader(resp.Body, maxResponseBytes+1))
	if err != nil {
		return fmt.Errorf("kms: read %s response: %w", operation, err)
	}
	if len(data) > maxResponseBytes {
		return fmt.Errorf("kms: %s response exceeds limit", operation)
	}
	if err := json.Unmarshal(data, output); err != nil {
		return fmt.Errorf("kms: decode %s response: %w", operation, err)
	}
	return nil
}

View on GitHub (pinned to 27d5a3981a)

Solutions

1. Map the status: 401/403 -> fix token/permissions, 404 -> verify region+keyID, 429 -> add backoff and reduce call rate, 5xx -> check provider status page
2. After rotating credentials, redeploy so all instances use the new token
3. Confirm the key ID belongs to the same project the token can access
4. For 429, wrap calls with exponential backoff and respect Retry-After

Example fix

// before
// no retry: transient 429/503 fails the operation immediately
pt, err := client.Encrypt(ctx, plaintext)

// after
var pt []byte
err := retry.Do(func() error {
    var e error
    pt, e = client.Encrypt(ctx, plaintext)
    return e
}, retry.OnHTTP(429, 500, 502, 503), retry.Backoff(100*time.Millisecond))

null

pt, err := client.Encrypt(ctx, plaintext)
for attempt := 0; isRetryableKMSStatus(err, 429, 500, 502, 503) && attempt < 3; attempt++ {
	time.Sleep(time.Duration(1<<attempt) * 250 * time.Millisecond)
	pt, err = client.Encrypt(ctx, plaintext)
}
func isRetryableKMSStatus(err error, codes ...int) bool {
	for _, c := range codes {
		if strings.Contains(err.Error(), fmt.Sprintf("HTTP %d", c)) { return true }
	}
	return false
}

When it happens

Trigger: Expired or revoked auth token (401); token lacking permission on this key (403); typo in region or key ID hitting a nonexistent resource (404); burst of encrypt/decrypt calls tripping rate limits (429); Scaleway incident returning 502/503.

Common situations: Rotated tokens not updated in deployed secrets; least-privilege policy missing kms actions; dev config pointing at a key from another project; startup probes across many instances simultaneously hitting quotas.

Related errors

• kms: %s request failed: %w
• Claude usage request failed with HTTP %d
• production KMS configuration: %w
• kms: unsupported provider %q
• kms: plaintext exceeds %d bytes

AI-assisted analysis of JuliusBrussee/caveman@27d5a3981a (2026-08-15). Data as JSON: /api/errors/233a82240de821c3. Report an issue: GitHub.