JuliusBrussee/caveman · error

kms: returned HTTP

Error message

kms: %s returned HTTP %d

What it means

The KMS HTTP endpoint responded with a status code other than 200. The client discards up to 32KB of the response body and returns this error carrying the operation name and actual status code. It means the request reached the KMS server but the server rejected it or returned an unexpected status (auth failure, bad request, server error).

Solutions

1. Read the status code from the error string and check the KMS server logs for the corresponding request to learn the rejection reason.
2. Verify the x-auth-token is current and valid; rotate/refresh credentials if expired.
3. Confirm the request payload and endpoint path match the KMS API version deployed.
4. For 5xx responses, retry with backoff; the failure may be transient on the server side.
5. Check RBAC/key ACLs on the KMS side if the status is 403.

Example fix

// before
err: kms: Decrypt returned HTTP 403
// after
if strings.Contains(err.Error(), "returned HTTP 403") || strings.Contains(err.Error(), "returned HTTP 401") {
    if err := refreshKMSToken(ctx); err != nil { return err }
    out, err = kms.Decrypt(ctx, ct) // retry once with fresh token
}

// preflight: confirm credentials/endpoint accept a cheap request
resp, err := http.Get(cfg.KMSEndpoint + "/health")
if err == nil && (resp.StatusCode == 401 || resp.StatusCode == 403) {
    return errors.New("KMS auth rejected: refresh x-auth-token before calling Encrypt/Decrypt")
}

if err != nil {
    if strings.Contains(err.Error(), "returned HTTP 5") {
        // retry with backoff: transient server-side failure
    } else if strings.Contains(err.Error(), "returned HTTP 4") {
        // do not retry: fix token/path/payload first
    }
    return err
}

When it happens

Trigger: Encrypt or Decrypt calls where the KMS server returns 401/403 (invalid or expired x-auth-token), 404 (wrong path), 400 (malformed request payload), or 5xx (server-side failure).

Common situations: Expired or rotated KMS auth token not refreshed in config, wrong API path for the KMS deployment version, sending a request to a load balancer returning 502/503, or insufficient permissions for the key being used.

Related errors

• kms: create request
• kms: decode response
• kms: probe decrypt
• kms: probe encrypt
• kms: read response

AI-assisted analysis of JuliusBrussee/caveman@3ee70a1026 (2026-09-20). Data as JSON: /api/errors/233a82240de821c3. Report an issue: GitHub.

Appendix: source

Thrown at shared/platform/kms/kms.go:367

		return fmt.Errorf("kms: encode %s request: %w", operation, err)
	}
	endpoint := c.apiBaseURL + "/key-manager/v1alpha1/regions/" + url.PathEscape(region) +
		"/keys/" + url.PathEscape(keyID) + "/" + operation
	req, err := http.NewRequestWithContext(ctx, http.MethodPost, endpoint, bytes.NewReader(body))
	if err != nil {
		return fmt.Errorf("kms: create %s request: %w", operation, err)
	}
	req.Header.Set("content-type", "application/json")
	req.Header.Set("accept", "application/json")
	req.Header.Set("x-auth-token", c.token)
	resp, err := c.httpClient.Do(req)
	if err != nil {
		return fmt.Errorf("kms: %s request failed: %w", operation, err)
	}
	defer resp.Body.Close()
	if resp.StatusCode != http.StatusOK {
		_, _ = io.Copy(io.Discard, io.LimitReader(resp.Body, 32<<10))
		return fmt.Errorf("kms: %s returned HTTP %d", operation, resp.StatusCode)
	}
	data, err := io.ReadAll(io.LimitReader(resp.Body, maxResponseBytes+1))
	if err != nil {
		return fmt.Errorf("kms: read %s response: %w", operation, err)
	}
	if len(data) > maxResponseBytes {
		return fmt.Errorf("kms: %s response exceeds limit", operation)
	}
	if err := json.Unmarshal(data, output); err != nil {
		return fmt.Errorf("kms: decode %s response: %w", operation, err)
	}
	return nil
}

View on GitHub (pinned to 3ee70a1026)