juspay/hyperswitch · critical · GcpKmsError

Failed to GCP KMS decrypt input data

Error message

Failed to GCP KMS decrypt input data

What it means

Thrown when the underlying google_cloud_kms client's decrypt RPC fails inside GcpKmsClient::decrypt. The request has already been base64 decoded successfully; this error means GCP Cloud KMS rejected or failed the DecryptRequest (network error, permission denied, key not found/enabled, or ciphertext was encrypted with a different key). The original gRPC error is logged (gcp_kms_error) and a failure metric incremented before this context is applied.

Source

Thrown at crates/external_services/src/gcp_kms/core.rs:156

        let output = consts::BASE64_ENGINE.encode(response.ciphertext);

        let time_taken = start.elapsed();
        metrics::GCP_KMS_ENCRYPT_TIME.record(time_taken.as_secs_f64(), &[]);

        Ok(output)
    }
}

/// Errors that could occur during GCP KMS operations.
#[derive(Debug, thiserror::Error)]
pub enum GcpKmsError {
    /// An error occurred when base64 decoding the input data.
    #[error("Failed to base64 decode input data")]
    Base64DecodingFailed,

    /// An error occurred when GCP KMS decrypting the input data.
    #[error("Failed to GCP KMS decrypt input data")]
    DecryptionFailed,

    /// An error occurred when GCP KMS encrypting the input data.
    #[error("Failed to GCP KMS encrypt input data")]
    EncryptionFailed,

    /// An error occurred UTF-8 decoding the GCP KMS decrypted output.
    #[error("Failed UTF-8 decode of GCP KMS decrypted output")]
    Utf8DecodingFailed,

    /// An error occurred when creating the GCP KMS client.
    #[error("Failed to create GCP KMS client")]
    ClientCreationFailed,
}

#[cfg(test)]
mod tests {
    use super::*;

View on GitHub (pinned to 806ec7dcc0)

Solutions

1. Check the router_env log line 'Failed to GCP KMS decrypt data' with gcp_kms_error=?error — the underlying gRPC status (PermissionDenied, NotFound, InvalidArgument, Unavailable) pinpoints the cause
2. Verify the GcpKmsConfig key path matches the real resource: gcloud kms keys list --location <loc> --keyring <ring>
3. Confirm the authenticated principal has roles/cloudkms.cryptoKeyDecrypter on the key: gcloud kms keys get-iam-policy <key> --location <loc> --keyring <ring>
4. Ensure the ciphertext was encrypted with the same cryptoKey (or re-encrypt data after key rotation)
5. If Unavailable/DeadlineExceeded, check network/egress to cloudkms.googleapis.com and consider retrying with backoff

Example fix

// before: opaque failure
let secret = client.decrypt(b64_ciphertext).await?;

// after: inspect and branch on the underlying error context via error_stack
match client.decrypt(b64_ciphertext).await {
    Ok(secret) => secret,
    Err(e) if e.current_context().to_string().contains("GCP KMS decrypt") => {
        // read the logged gcp_kms_error / report ops: likely IAM, key state, or wrong key
        return Err(e);
    }
    Err(e) => return Err(e),
}

// Validate config before constructing the client
if let Err(msg) = gcp_kms_config.validate() {
    return Err(msg);
}
// Ensure ciphertext matches what encrypt() emits (standard base64)
assert!(base64::engine::general_purpose::STANDARD
    .decode(ciphertext.trim())
    .is_ok());

use error_stack::ContextExt;

match client.decrypt(ct).await {
    Ok(p) => p,
    Err(e) if matches!(e.current_context(), GcpKmsError::DecryptionFailed) => {
        // transient (Unavailable/DeadlineExceeded) vs permanent (PermissionDenied/NotFound):
        // check the logged gcp_kms_error; retry only transient statuses with backoff
        return Err(e);
    }
    Err(e) => return Err(e),
}

When it happens

Trigger: Calling GcpKmsClient::decrypt and the inner_client.decrypt(request, None) RPC returning an error at crates/external_services/src/gcp_kms/core.rs:98-106. Causes include: wrong key name (project/location/keyRing/cryptoKey mismatch in GcpKmsConfig), IAM principal lacking roles/cloudkms.cryptoKeyDecrypter, key disabled or scheduled for destruction, ciphertext encrypted under a different key, malformed/truncated decoded ciphertext, or connectivity to cloudkms.googleapis.com failing.

Common situations: Config typo in project_id/location_id/key_ring_id/key_id; running locally or in CI without Application Default Credentials or with a service account missing KMS IAM roles; rotating or disabling a KMS key then decrypting old data; ciphertext copied from another environment/key; egress firewall blocking the KMS endpoint; gRPC channel issues in long-lived processes.

Related errors

• Failed to GCP KMS encrypt input data
• Failed to create GCP KMS client
• Failed to base64 decode input data
• Failed UTF-8 decode of GCP KMS decrypted output

AI-assisted analysis of juspay/hyperswitch@806ec7dcc0 (2026-08-28). Data as JSON: /api/errors/1bd65d5280213cb7. Report an issue: GitHub.