shadowsocks/shadowsocks-windows · critical · System.Exception

openssl: cannot set key

Error message

openssl: cannot set key

What it means

Thrown from AEADOpenSSLEncryptor.InitCipher on the second EVP_CipherInit_ex (the call that supplies the actual subkey) returning != 1. By this point the cipher, IV length, and key length are set; supplying the key finalises the context. Failure indicates the key itself was rejected (wrong length vs the set key length) or the context is inconsistent.

Source

Thrown at shadowsocks-csharp/Encryption/AEAD/AEADOpenSSLEncryptor.cs:81

            DeriveSessionKey(isEncrypt ? _encryptSalt : _decryptSalt, _Masterkey,
                isEncrypt ? _opensslEncSubkey : _opensslDecSubkey);

            var ret = OpenSSL.EVP_CipherInit_ex(ctx, _cipherInfoPtr, IntPtr.Zero, null, null,
                isEncrypt ? OpenSSL.OPENSSL_ENCRYPT : OpenSSL.OPENSSL_DECRYPT);
            if (ret != 1) throw new System.Exception("openssl: fail to init ctx");

            ret = OpenSSL.EVP_CIPHER_CTX_set_key_length(ctx, keyLen);
            if (ret != 1) throw new System.Exception("openssl: fail to set key length");

            ret = OpenSSL.EVP_CIPHER_CTX_ctrl(ctx, OpenSSL.EVP_CTRL_AEAD_SET_IVLEN,
                nonceLen, IntPtr.Zero);
            if (ret != 1) throw new System.Exception("openssl: fail to set AEAD nonce length");

            ret = OpenSSL.EVP_CipherInit_ex(ctx, IntPtr.Zero, IntPtr.Zero,
                isEncrypt ? _opensslEncSubkey : _opensslDecSubkey,
                null,
                isEncrypt ? OpenSSL.OPENSSL_ENCRYPT : OpenSSL.OPENSSL_DECRYPT);
            if (ret != 1) throw new System.Exception("openssl: cannot set key");
            OpenSSL.EVP_CIPHER_CTX_set_padding(ctx, 0);
        }

        public override void cipherEncrypt(byte[] plaintext, uint plen, byte[] ciphertext, ref uint clen)
        {
            OpenSSL.SetCtxNonce(_encryptCtx, _encNonce, true);
            // buf: all plaintext
            // outbuf: ciphertext + tag
            int ret;
            int tmpLen = 0;
            clen = 0;
            var tagBuf = new byte[tagLen];

            ret = OpenSSL.EVP_CipherUpdate(_encryptCtx, ciphertext, out tmpLen,
                plaintext, (int) plen);
            if (ret != 1) throw new CryptoErrorException("openssl: fail to encrypt AEAD");
            clen += (uint) tmpLen;
            // For AEAD cipher, it should not output anything

View on GitHub (pinned to 891d971682)

Solutions

1. Verify the subkey buffer passed is exactly keyLen bytes and was fully written by DeriveSessionKey.
2. Ensure no code path mutates the context between the two EVP_CipherInit_ex calls.
3. Log keyLen vs actual buffer length to catch derivation shortfalls.

Example fix

// before
ret = OpenSSL.EVP_CipherInit_ex(ctx, IntPtr.Zero, IntPtr.Zero, isEncrypt ? _opensslEncSubkey : _opensslDecSubkey, null, isEncrypt ? OpenSSL.OPENSSL_ENCRYPT : OpenSSL.OPENSSL_DECRYPT);
if (ret != 1) throw new System.Exception("openssl: cannot set key");

// after
var keyBuf = isEncrypt ? _opensslEncSubkey : _opensslDecSubkey;
if (ret != 1)
    throw new System.Exception($"openssl: cannot set key (ret={ret}, bufLen={keyBuf.Length}, expected={keyLen})");

// Verify the subkey buffer length matches keyLen before the second init
byte[] keyBuf = isEncrypt ? _opensslEncSubkey : _opensslDecSubkey;
if (keyBuf == null || keyBuf.Length != keyLen)
    throw new ArgumentException($"subkey length {keyBuf?.Length ?? -1} != keyLen {keyLen}");

bool SubKeyLengthOk(byte[] buf, int keyLen) =>
    buf != null && buf.Length == keyLen;

try { OpenSSL.EVP_CipherInit_ex(ctx, IntPtr.Zero, IntPtr.Zero, keyBuf, null, enc); }
catch (Exception ex) when (ex.Message.Contains("cannot set key"))
{ /* re-derive session key, recreate ctx, retry once */ }

When it happens

Trigger: The subkey buffer (_opensslEncSubkey/_opensslDecSubkey) length differs from the key length set in the prior step; DeriveSessionKey produced fewer bytes than keyLen; the context was mutated between the two init calls.

Common situations: A session-key derivation bug yielding a short key; reusing a context that was partially initialised; thread races writing the subkey buffer.

Related errors

• openssl: fail to init ctx
• openssl: fail to set key length
• failed to set key
• openssl: cipher not found
• openssl: fail to create ctx

AI-assisted analysis of shadowsocks/shadowsocks-windows@891d971682 (2026-08-13). Data as JSON: /api/errors/970f84219497e44a. Report an issue: GitHub.