basecamp/fizzy · error · ActionPack::WebAuthn::InvalidKeyError

Invalid RSA key: #{error.message}

Error message

Invalid RSA key: #{error.message}

What it means

ActionPack::WebAuthn::CoseKey wrapped an OpenSSL::PKey::PKeyError from OpenSSL::PKey::RSA.new: n and e (labels -1/-2) were present and passed the 2048-bit gate, but the values do not form a loadable RSA public key. OpenSSL::BN.new(bytes, 2) accepts almost any byte string, so malformed input usually survives until this final DER parse. The library re-raises as InvalidKeyError with the underlying OpenSSL message (for example an ASN.1/RSAError reason) appended.

Source

Thrown at lib/action_pack/web_authn/cose_key.rb:181

      n = OpenSSL::BN.new(n_bytes, 2)
      e = OpenSSL::BN.new(e_bytes, 2)

      asn1 = OpenSSL::ASN1::Sequence([
        OpenSSL::ASN1::Sequence([
          OpenSSL::ASN1::ObjectId("rsaEncryption"),
          OpenSSL::ASN1::Null.new(nil)
        ]),
        OpenSSL::ASN1::BitString(
          OpenSSL::ASN1::Sequence([
            OpenSSL::ASN1::Integer(n),
            OpenSSL::ASN1::Integer(e)
          ]).to_der
        )
      ])

      OpenSSL::PKey::RSA.new(asn1.to_der)
    rescue OpenSSL::PKey::PKeyError => error
      raise ActionPack::WebAuthn::InvalidKeyError, "Invalid RSA key: #{error.message}"
    end
end

View on GitHub (pinned to 7aabe74580)

Solutions

1. Verify -1/-2 are raw big-endian byte strings, not hex or base64 text: OpenSSL::BN.new(bytes, 2) accepts anything, so garbage only surfaces at OpenSSL::PKey::RSA.new
2. Re-derive both parameters from a known-good RSA public key (rsa.n.to_s(2), rsa.e.to_s(2)) and compare with what your code produced
3. Check for encoding drift upstream: JSON round-trips, base64url padding, or CBOR libraries that alter byte strings
4. Rescue ActionPack::WebAuthn::InvalidKeyError at registration and surface a malformed-credential error to the client, logging error.message which carries the OpenSSL reason

Example fix

# before: exponent passed as hex text; BN accepts it, RSA.new fails later
e_bytes = '010001'

# after: raw big-endian bytes
e_bytes = ['010001'].pack('H*') # or OpenSSL::BN.new(65537).to_s(2)

e = OpenSSL::BN.new(cose.parameters[-2], 2)
return unless [3, 5, 17, 257, 65537].include?(e.to_i) # conventional RSA public exponents
cose.to_openssl_key

begin
  openssl_key = cose_key.to_openssl_key
rescue ActionPack::WebAuthn::InvalidKeyError => e
  # e.message embeds the OpenSSL parse reason; keep it in the failure log
  render json: { error: e.message }, status: :unprocessable_entity
end

When it happens

Trigger: Calling to_openssl_key on an RS256 key where -1/-2 are wrong-typed or corrupted byte strings (hex text, sign-bit mistakes, swapped n and e, or garbage from misaligned slicing) so OpenSSL::BN accepts them but OpenSSL::PKey::RSA.new(asn1.to_der) raises during the final parse.

Common situations: Re-encoding keys through JSON or a database (bytes to hex to binary mismatches) before building the COSE map; test key factories with endian/sign errors; non-canonical CBOR decoders altering byte strings; credentials relayed through proxies that mangle binary payloads.

Related errors

• Invalid EC2 key: #{error.message}
• Invalid OKP key: #{error.message}
• Missing RSA key parameters
• RSA key must be at least #{MINIMUM_RSA_KEY_BITS} bits
• Unsupported COSE key type/algorithm: #{key_type}/#{algorithm

AI-assisted analysis of basecamp/fizzy@7aabe74580 (2026-08-21). Data as JSON: /api/errors/e3c4c1832d7c5039. Report an issue: GitHub.