nautechsystems/nautilus_trader · error

Failed to decode RSA private key

Error message

Failed to decode RSA private key

What it means

The PEM parsed and passed the private-key tag check, but `KeyPair::from_pkcs8` could not decode the DER contents as a valid PKCS#8 RSA private key (ring rejects it). The underlying error is deliberately discarded, so the message is generic; it means the key material is structurally invalid, not RSA/PKCS#8, corrupted, or encrypted.

Source

Thrown at crates/cryptography/src/signing.rs:59

/// - `data` is empty.
/// - `private_key_pem` is not a valid PEM-encoded PKCS#8 RSA private key or cannot be parsed.
/// - Signature generation fails due to key or cryptographic errors.
pub fn rsa_signature(private_key_pem: &str, data: &str) -> anyhow::Result<String> {
    if data.is_empty() {
        anyhow::bail!("Query string cannot be empty");
    }

    // Remove PEM headings and decode to DER bytes using the `pem` crate
    let pem = pem::parse(private_key_pem.trim())
        .map_err(|e| anyhow::anyhow!("Failed to parse PEM: {e}"))?;

    // Ensure this is a private key
    if !pem.tag().ends_with("PRIVATE KEY") {
        anyhow::bail!("PEM does not contain a private key");
    }

    // Construct RSA key pair from PKCS#8 DER bytes
    let key_pair = KeyPair::from_pkcs8(pem.contents())
        .map_err(|_| anyhow::anyhow!("Failed to decode RSA private key"))?;

    // Prepare RNG and output buffer (signature length = modulus length)
    let rng = lc_rand::SystemRandom::new();
    let mut signature = vec![0u8; key_pair.public_modulus_len()];

    key_pair
        .sign(
            &lc_signature::RSA_PKCS1_SHA256,
            &rng,
            data.as_bytes(),
            &mut signature,
        )
        .map_err(|_| anyhow::anyhow!("Failed to generate RSA signature"))?;

    Ok(BASE64_STANDARD.encode(signature))
}

View on GitHub (pinned to 18893faf8b)

Solutions

1. Convert the key to unencrypted PKCS#8: openssl pkcs8 -topk8 -nocrypt -in key.pem -out key_pkcs8.pem
2. Decrypt if prompted for a passphrase and re-export without encryption before use
3. Verify the key type: openssl pkey -in key.pem -text -noout (must show an RSA private key)
4. Regenerate the keypair if the material is truncated or corrupted

Example fix

// before
let sig = rsa_signature(&encrypted_pem, query)?; // ENCRYPTED PRIVATE KEY
// after
// openssl pkcs8 -topk8 -nocrypt -in encrypted.pem -out decrypted.pem
let sig = rsa_signature(&std::fs::read_to_string("decrypted.pem")?, query)?;

// reject encrypted or non-RSA keys up front
if key_text.contains("ENCRYPTED PRIVATE KEY") {
    return Err(anyhow::anyhow!("decrypt the key to PKCS#8 first"));
}
// convert: openssl pkcs8 -topk8 -nocrypt -in key.pem -out key_pkcs8.pem

fn is_pkcs8_rsa_private_key(s: &str) -> bool {
    s.contains("-----BEGIN PRIVATE KEY-----") && !s.contains("ENCRYPTED")
}

match rsa_signature(&key, query) {
    Ok(sig) => use(sig),
    Err(e) if e.to_string().contains("Failed to decode RSA private key") => {
        tracing::error!("key is not unencrypted PKCS#8 RSA; re-export with openssl");
    }
    Err(e) => return Err(e),
}

When it happens

Trigger: Calling `rsa_signature` with PEM contents that are valid PEM with a private-key-looking tag but whose DER is not parseable PKCS#8 RSA — e.g. an 'ENCRYPTED PRIVATE KEY' block, an Ed25519/EC key, truncated base64 that still parses, or a PKCS#1 body mislabeled as 'PRIVATE KEY'.

Common situations: Using an encrypted (passphrase-protected) key without decrypting; keys generated in another format (PKCS#1 'BEGIN RSA PRIVATE KEY' bodies relabeled, or SEC1 EC keys); corrupted copy/paste; cloud secrets returning an unexpected key type.

Understand the failure class

Background: "Invalid ... format", "must be in format X", "does not look like a ..." — invalid argument format errors across CLI tools and libraries — this error's family across 17 libraries.

• Parsing and encoding errors: unexpected token, malformed input — why parsers reject input and how to find the real culprit.

Related errors

• Query string cannot be empty
• PEM does not contain a private key
• Failed to parse PEM: {e}
• Failed to generate RSA signature
• Invalid `key`, missing a '{REDIS_DELIMITER}' delimiter, was

AI-assisted analysis of nautechsystems/nautilus_trader@18893faf8b (2026-09-08). Data as JSON: /api/errors/2b292f21bace662a. Report an issue: GitHub.