ruby/ruby · error · OpenSSL::PKey::PKeyError

unsupported padding mode

Error message

unsupported padding mode

What it means

The private helper translate_padding_mode, shared by the four legacy RSA wrappers, maps the numeric padding constants (PKCS1_PADDING=1, SSLV23_PADDING=2, NO_PADDING=3, PKCS1_OAEP_PADDING=4) onto the string form the modern API accepts. Any other integer (or nil) falls through to else and raises PKeyError 'unsupported padding mode'. The error therefore always comes from the argument the caller passed.

Source

Thrown at ext/openssl/lib/openssl/pkey.rb:489

    end

    PKCS1_PADDING = 1
    SSLV23_PADDING = 2
    NO_PADDING = 3
    PKCS1_OAEP_PADDING = 4

    private def translate_padding_mode(num)
      case num
      when PKCS1_PADDING
        "pkcs1"
      when SSLV23_PADDING
        "sslv23"
      when NO_PADDING
        "none"
      when PKCS1_OAEP_PADDING
        "oaep"
      else
        raise PKeyError, "unsupported padding mode"
      end
    end
  end
end

View on GitHub (pinned to 0e5b888e1c)

Solutions

1. Always pass constants from OpenSSL::PKey::RSA: PKCS1_PADDING, SSLV23_PADDING, NO_PADDING, PKCS1_OAEP_PADDING.
2. Validate the padding argument against that set before calling the legacy methods.
3. Migrate to rsa.encrypt/decrypt/sign_raw/verify_recover with an options hash such as { 'rsa_padding_mode' => 'oaep' }.

Example fix

# before
rsa.public_encrypt(data, 9) # not a defined constant

# after
rsa.public_encrypt(data, OpenSSL::PKey::RSA::PKCS1_OAEP_PADDING)
# or the modern form:
rsa.encrypt(data, { 'rsa_padding_mode' => 'oaep' })

VALID_PADDING = [
  OpenSSL::PKey::RSA::PKCS1_PADDING,
  OpenSSL::PKey::RSA::SSLV23_PADDING,
  OpenSSL::PKey::RSA::NO_PADDING,
  OpenSSL::PKey::RSA::PKCS1_OAEP_PADDING
].freeze
raise ArgumentError, 'unsupported padding' unless VALID_PADDING.include?(padding)
rsa.public_encrypt(data, padding)

def rsa_padding?(num)
  [OpenSSL::PKey::RSA::PKCS1_PADDING,
   OpenSSL::PKey::RSA::SSLV23_PADDING,
   OpenSSL::PKey::RSA::NO_PADDING,
   OpenSSL::PKey::RSA::PKCS1_OAEP_PADDING].include?(num)
end

begin
  rsa.public_encrypt(data, padding)
rescue OpenSSL::PKey::PKeyError => e
  raise "bad padding #{padding.inspect}: #{e.message}"
end

When it happens

Trigger: Passing a padding value outside 1..4 to private_encrypt, public_decrypt, public_encrypt or private_decrypt, for example a constant borrowed from another library, a raw integer from config, or nil after a broken default.

Common situations: Copy-pasted padding constants from a different binding; config files storing padding as a string ('oaep') that gets passed through unconverted; constants removed during a refactor leaving nil or 0 in place.

Related errors

• incomplete RSA
• private key needed.
• unknown SSL method `%s'
• end of file reached
• OpenSSL::PKey::DH is immutable on OpenSSL 3.0; use OpenSSL::

AI-assisted analysis of ruby/ruby@0e5b888e1c (2026-08-21). Data as JSON: /api/errors/2215174b26e64820. Report an issue: GitHub.