JuliusBrussee/caveman · error

kms: request failed

Error message

kms: %s request failed: %w

What it means

This error wraps a transport-level failure from the KMS client's HTTP request in kms.go. The client builds an authenticated request (content-type, accept, x-auth-token headers) and performs it via c.httpClient.Do; if the request never completes (connection failure, DNS, timeout, TLS), it returns this wrapped error with the operation name (Encrypt or Decrypt). It indicates the KMS endpoint was not reached at all, not that the server rejected the request.

Solutions

1. Verify the KMS endpoint URL/host and port in configuration and that the KMS service is reachable (curl the health endpoint).
2. Check network connectivity: DNS resolution, firewall/security-group rules, and proxy environment variables.
3. Inspect the wrapped error with errors.Unwrap or %v to see the underlying cause (connection refused, timeout, x509 error).
4. If the cause is a timeout or stale connection, retry with backoff or tune the httpClient's Timeout/Transport settings.
5. If TLS-related, confirm the CA bundle/trusted roots available to the process.

Example fix

// before
out, err := kms.Encrypt(ctx, plaintext) // fails: kms: Encrypt request failed: dial tcp 10.0.0.5:8200: connect: connection refused
// after
if err != nil {
    var netErr net.Error
    if errors.As(err, &netErr) && netErr.Timeout() {
        // retry with backoff
    } else {
        log.Fatalf("KMS unreachable, check endpoint %q: %v", cfg.KMSEndpoint, err)
    }
}

// before calling Encrypt/Decrypt
u, err := url.Parse(cfg.KMSEndpoint)
if err != nil || u.Scheme == "" || u.Host == "" {
    return fmt.Errorf("invalid KMS endpoint %q", cfg.KMSEndpoint)
}
conn, err := net.DialTimeout("tcp", u.Host, 2*time.Second)
if err != nil {
    return fmt.Errorf("KMS unreachable at %s: %w", u.Host, err)
}
conn.Close()

if err != nil {
    var netErr net.Error
    if errors.As(err, &netErr) && netErr.Timeout() {
        // retry with backoff
    }
    return fmt.Errorf("kms transport: %w", err)
}

When it happens

Trigger: Calling Encrypt or Decrypt when the KMS server is unreachable: wrong host/port in the KMS endpoint config, server down, DNS failure, TLS handshake failure, network partition, or an http.Client timeout/expired connection being reused.

Common situations: Deploying with a stale or mistyped KMS URL, the KMS service container not running (connection refused), firewalls blocking egress, using https:// against a plaintext server or vice versa, or a proxy misconfiguration in the environment.

Understand the failure class

Background: 'Something went wrong' / 'Request failed (500)' / 'HTTP error! status: 404' — what failed HTTP requests actually mean and how to find the real cause — this error's family across 28 libraries.

Related errors

• kms: read response
• kms: create request
• kms: returned HTTP
• secretbox: KMS encrypt
• secretbox: payload KMS encrypt

AI-assisted analysis of JuliusBrussee/caveman@3ee70a1026 (2026-09-20). Data as JSON: /api/errors/124e113746692538. Report an issue: GitHub.

Appendix: source

Thrown at shared/platform/kms/kms.go:362

}

func (c *Client) call(ctx context.Context, region, keyID, operation string, input, output any) error {
	body, err := json.Marshal(input)
	if err != nil {
		return fmt.Errorf("kms: encode %s request: %w", operation, err)
	}
	endpoint := c.apiBaseURL + "/key-manager/v1alpha1/regions/" + url.PathEscape(region) +
		"/keys/" + url.PathEscape(keyID) + "/" + operation
	req, err := http.NewRequestWithContext(ctx, http.MethodPost, endpoint, bytes.NewReader(body))
	if err != nil {
		return fmt.Errorf("kms: create %s request: %w", operation, err)
	}
	req.Header.Set("content-type", "application/json")
	req.Header.Set("accept", "application/json")
	req.Header.Set("x-auth-token", c.token)
	resp, err := c.httpClient.Do(req)
	if err != nil {
		return fmt.Errorf("kms: %s request failed: %w", operation, err)
	}
	defer resp.Body.Close()
	if resp.StatusCode != http.StatusOK {
		_, _ = io.Copy(io.Discard, io.LimitReader(resp.Body, 32<<10))
		return fmt.Errorf("kms: %s returned HTTP %d", operation, resp.StatusCode)
	}
	data, err := io.ReadAll(io.LimitReader(resp.Body, maxResponseBytes+1))
	if err != nil {
		return fmt.Errorf("kms: read %s response: %w", operation, err)
	}
	if len(data) > maxResponseBytes {
		return fmt.Errorf("kms: %s response exceeds limit", operation)
	}
	if err := json.Unmarshal(data, output); err != nil {
		return fmt.Errorf("kms: decode %s response: %w", operation, err)
	}
	return nil
}

View on GitHub (pinned to 3ee70a1026)