JuliusBrussee/caveman · critical

secretbox: payload KMS encrypt

Error message

secretbox: payload KMS encrypt: %w

What it means

EncryptPayloadKey encrypts data-encryption keys. When useKMS() is true it delegates to kms.EncryptPayload with a 10s timeout and wraps any KMS failure as 'secretbox: payload KMS encrypt'. This is distinct from the master Encrypt path (kms.Encrypt) and means the payload-key envelope could not be produced. Note it wraps the error rather than falling back to local encryption, so a KMS outage fails closed.

Solutions

  1. Inspect the wrapped error (errors.Unwrap) to identify auth vs network vs timeout cause.
  2. Verify the payload-key-specific credentials and key ID/region env vars are current and match the Scaleway console.
  3. Check KMS service status and retry after transient outages; the errors are intentionally not masked by a local fallback.
  4. Confirm network egress from the workload to the Scaleway KMS endpoint; add NAT/proxy rules if missing.
  5. For requests that hit the 10s deadline, check KMS latency and consider the configured timeout budget.

Example fix

// before: rotated key not redeployed -> secretbox: payload KMS encrypt: 401 unauthorized
// process started before key rotation

// after: reload credentials on rotation and restart/re-auth
if err := kms.RefreshCredentials(ctx); err != nil { log.Fatal(err) }
wrapped, err := secretbox.EncryptPayloadKey(dek)
Defensive patterns

Strategy: try-catch

Validate before calling

if os.Getenv("CAVE_KMS_PROVIDER") == "scaleway" {
    ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
    defer cancel()
    if err := kms.Ping(ctx); err != nil { // or a tiny EncryptPayload round trip
        return fmt.Errorf("payload KMS not reachable at startup: %w", err)
    }
}

Try / catch

wrapped, err := secretbox.EncryptPayloadKey(dek)
if err != nil {
    if kmsErr := (*kms.Error)(nil); errors.As(err, &kmsErr) {
        // fail closed: never fall back to local key encryption for DEKs
        return fmt.Errorf("payload key wrapping failed, refusing local fallback: %w", err)
    }
    if errors.Is(err, context.DeadlineExceeded) {
        // retry with backoff for transient latency
        return retryWithBackoff(dek)
    }
    return err
}

Prevention

When it happens

Trigger: Calling EncryptPayloadKey while CAVE_KMS_PROVIDER is set and kms.EncryptPayload fails: bad or rotated Scaleway payload-key credentials, wrong key ID/region, KMS endpoint unreachable, or the 10s context deadline exceeded.

Common situations: KMS outage or rate limiting during a deploy stampede; payload-key rotated in the Scaleway console while running processes still hold old credentials; region mismatch between the configured key and the API endpoint; egress blocked in a private subnet without NAT to the KMS API.

Understand the failure class

Background: "API request failed": what wrapped HTTP errors from external APIs mean and how to find the real cause — this error's family across 29 libraries.

Related errors

AI-assisted analysis of JuliusBrussee/caveman@3ee70a1026 (2026-09-20). Data as JSON: /api/errors/0e79526eac2ef257. Report an issue: GitHub.

Appendix: source

Thrown at shared/platform/secretbox/secretbox.go:96

	nonce := make([]byte, gcm.NonceSize())
	if _, err := rand.Read(nonce); err != nil {
		return nil, fmt.Errorf("nonce entropy: %w", err)
	}
	// Seal appends the ciphertext+tag to nonce, so the returned slice is the
	// full nonce||ciphertext envelope.
	return gcm.Seal(nonce, nonce, plaintext, nil), nil
}

// EncryptPayloadKey wraps an artifact data-encryption key. Production uses the
// dedicated payload KEK; local development retains the same AES-GCM envelope as
// other local secrets.
func EncryptPayloadKey(plaintext []byte) ([]byte, error) {
	if useKMS() {
		ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
		defer cancel()
		wrapped, err := kms.EncryptPayload(ctx, plaintext)
		if err != nil {
			return nil, fmt.Errorf("secretbox: payload KMS encrypt: %w", err)
		}
		return wrapped, nil
	}
	return Encrypt(plaintext)
}

// Decrypt reverses Encrypt: it expects nonce(12) || ciphertext+tag.
func Decrypt(envelope []byte) ([]byte, error) {
	if kms.IsEnvelope(envelope) {
		ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
		defer cancel()
		plaintext, err := kms.Decrypt(ctx, envelope)
		if err != nil {
			return nil, fmt.Errorf("secretbox: KMS decrypt: %w", err)
		}
		return plaintext, nil
	}
	if runtimeenv.IsProduction() &&

View on GitHub (pinned to 3ee70a1026)