shadowsocks/shadowsocks-windows · critical · System.Exception

openssl: fail to set key length

Error message

openssl: fail to set key length

What it means

Thrown from AEADOpenSSLEncryptor.InitCipher when EVP_CIPHER_CTX_set_key_length returns != 1. OpenSSL rejects a key length that does not match the cipher's defined key sizes; for fixed-key AEAD ciphers like AES-GCM the length must be exactly 16 or 32 bytes.

Source

Thrown at shadowsocks-csharp/Encryption/AEAD/AEADOpenSSLEncryptor.cs:71

            if (isEncrypt)
            {
                _encryptCtx = ctx;
            }
            else
            {
                _decryptCtx = ctx;
            }

            DeriveSessionKey(isEncrypt ? _encryptSalt : _decryptSalt, _Masterkey,
                isEncrypt ? _opensslEncSubkey : _opensslDecSubkey);

            var ret = OpenSSL.EVP_CipherInit_ex(ctx, _cipherInfoPtr, IntPtr.Zero, null, null,
                isEncrypt ? OpenSSL.OPENSSL_ENCRYPT : OpenSSL.OPENSSL_DECRYPT);
            if (ret != 1) throw new System.Exception("openssl: fail to init ctx");

            ret = OpenSSL.EVP_CIPHER_CTX_set_key_length(ctx, keyLen);
            if (ret != 1) throw new System.Exception("openssl: fail to set key length");

            ret = OpenSSL.EVP_CIPHER_CTX_ctrl(ctx, OpenSSL.EVP_CTRL_AEAD_SET_IVLEN,
                nonceLen, IntPtr.Zero);
            if (ret != 1) throw new System.Exception("openssl: fail to set AEAD nonce length");

            ret = OpenSSL.EVP_CipherInit_ex(ctx, IntPtr.Zero, IntPtr.Zero,
                isEncrypt ? _opensslEncSubkey : _opensslDecSubkey,
                null,
                isEncrypt ? OpenSSL.OPENSSL_ENCRYPT : OpenSSL.OPENSSL_DECRYPT);
            if (ret != 1) throw new System.Exception("openssl: cannot set key");
            OpenSSL.EVP_CIPHER_CTX_set_padding(ctx, 0);
        }

        public override void cipherEncrypt(byte[] plaintext, uint plen, byte[] ciphertext, ref uint clen)
        {
            OpenSSL.SetCtxNonce(_encryptCtx, _encNonce, true);
            // buf: all plaintext
            // outbuf: ciphertext + tag

View on GitHub (pinned to 891d971682)

Solutions

1. Set EncryptorInfo.KeySize to a valid size for the cipher (AES-128-GCM=16, AES-256-GCM=32, chacha20-poly1305=32).
2. Confirm the derived subkey buffer is exactly keyLen bytes.
3. Ensure the method string selects the cipher whose key size matches KeySize.

Example fix

// before
ret = OpenSSL.EVP_CIPHER_CTX_set_key_length(ctx, keyLen);
if (ret != 1) throw new System.Exception("openssl: fail to set key length");

// after
if (ret != 1)
    throw new System.Exception($"openssl: fail to set key length (keyLen={keyLen}); check EncryptorInfo.KeySize for {_method}");

// Validate key length against the cipher before set_key_length
int expected = _innerLibName.Contains("128") ? 16 : _innerLibName.Contains("256") ? 32 : -1;
if (keyLen != expected) throw new ArgumentException($"keyLen {keyLen} != expected {expected}");

bool KeyLenMatchesCipher(string innerName, int keyLen) =>
    (innerName.Contains("128") && keyLen == 16) ||
    (innerName.Contains("256") && keyLen == 32) ||
    (innerName.StartsWith("chacha20") && keyLen == 32);

try { OpenSSL.EVP_CIPHER_CTX_set_key_length(ctx, keyLen); }
catch (Exception ex) when (ex.Message.Contains("key length"))
{ /* fix EncryptorInfo.KeySize for this method */ }

When it happens

Trigger: keyLen (from EncryptorInfo.KeySize) does not equal the cipher's valid key size (e.g. 24 for AES-GCM); a buffer underlength so the derived key is short; a custom method with wrong KeySize metadata.

Common situations: EncryptorInfo.KeySize wrong for the method; master/subkey buffer shorter than declared; method string that maps to a cipher with a different key size than configured.

Related errors

• failed to set key
• openssl: fail to set AEAD nonce length
• openssl: cannot set key
• method not found
• openssl: cipher not found

AI-assisted analysis of shadowsocks/shadowsocks-windows@891d971682 (2026-08-13). Data as JSON: /api/errors/4fb53b8770a0d8bf. Report an issue: GitHub.