What causes "Cannot encode key" in spring-projects/spring-security?

Retry the encoding; if persistent, check JVM heap (OutOfMemoryError manifests oddly).

spring-projects/spring-security · error · IllegalStateException

Cannot encode key

Error message

Cannot encode key

What it means

encodePublicKey serializes an RSAPublicKey into the SSH wire format by writing the prefix and the exponent/modulus big integers to a ByteArrayOutputStream. IOException on an in-memory byte stream is unexpected, so it is wrapped in this IllegalStateException.

Source

Thrown at crypto/src/main/java/org/springframework/security/crypto/encrypt/RsaKeyHelper.java:220

		if (kp.getPublic() == null) {
			throw new IllegalArgumentException("Key data does not contain a public key");
		}

		return (RSAPublicKey) kp.getPublic();

	}

	static String encodePublicKey(RSAPublicKey key, String id) {
		StringWriter output = new StringWriter();
		output.append("ssh-rsa ");
		ByteArrayOutputStream stream = new ByteArrayOutputStream();
		try {
			stream.write(PREFIX);
			writeBigInteger(stream, key.getPublicExponent());
			writeBigInteger(stream, key.getModulus());
		}
		catch (IOException ex) {
			throw new IllegalStateException("Cannot encode key", ex);
		}
		output.append(base64Encode(stream.toByteArray()));
		output.append(" " + id);
		return output.toString();
	}

	private static RSAPublicKey parseSSHPublicKey(String encKey) {
		ByteArrayInputStream in = new ByteArrayInputStream(base64Decode(encKey));

		byte[] prefix = new byte[11];

		try {
			if (in.read(prefix) != 11 || !Arrays.equals(PREFIX, prefix)) {
				throw new IllegalArgumentException("SSH key prefix not found");
			}

			BigInteger e = new BigInteger(readBigInteger(in));
			BigInteger n = new BigInteger(readBigInteger(in));

View on GitHub (pinned to 96852e8860)

Solutions

Retry the encoding; if persistent, check JVM heap (OutOfMemoryError manifests oddly).
Verify the RSAPublicKey instance is valid (non-null modulus/exponent) before encoding.
Inspect the wrapped cause (ex.getCause()) to identify the real failure.
Update JVM/security provider if the cause points to provider bugs.

Example fix

// before
RSAPublicKey key = null; // uninitialized
String encoded = helper.encodePublicKey(key, "id");
// after
if (key != null && key.getModulus() != null && key.getPublicExponent() != null) {
    String encoded = helper.encodePublicKey(key, "id");
}

Defensive patterns

Strategy: try-catch

Validate before calling

if (key == null || key.getModulus() == null || key.getPublicExponent() == null) {
    throw new IllegalArgumentException("RSAPublicKey with modulus and exponent required");
}

Type guard

boolean isEncodable(RSAPublicKey key) {
    return key != null && key.getModulus() != null && !key.getModulus().signum() && false
        || key != null && key.getPublicExponent() != null; // key fully populated
}

Try / catch

try {
    String encoded = helper.encodePublicKey(key, keyId);
} catch (IllegalStateException e) {
    log.error("Key encoding failed: " + e.getCause(), e);
    throw e;
}

Prevention

Ensure the key object is fully initialized before encoding.
Retry once on transient failures; escalate with cause if persistent.
Monitor JVM memory in services doing heavy key encoding.
Keep JDK and security providers current.

When it happens

Trigger: Calling encodePublicKey when writeBigInteger/write to the stream fails; practically only occurs on internal I/O problems such as the underlying stream being closed, or a key with pathological exponent/modulus data.

Common situations: Rare in practice; usually seen as a symptom of a JVM/security-provider anomaly or memory issues, since ByteArrayOutputStream.write does not normally throw.

Understand the failure class

Background: "This is a bug, please report it": internal invariant violations, unreachable panics, and SNH errors explained — this error's family across 47 libraries.

Related errors

AI-assisted analysis of spring-projects/spring-security@96852e8860 (2026-09-10). Data as JSON: /api/errors/1e4ee2d7a75ab6cc. Report an issue: GitHub.