shadowsocks/shadowsocks-windows · critical · System.Exception

failed to set key

Error message

failed to set key

What it means

Thrown from CipherSetKey when MbedTLS.cipher_setkey returns non-zero after the session subkey is derived. cipher_setkey configures the context for encrypt/decrypt with the given key and key length; failure indicates the key length or mode is invalid for the loaded cipher (e.g. key length not matching the cipher's expected size).

Source

Thrown at shadowsocks-csharp/Encryption/AEAD/AEADMbedTLSEncryptor.cs:66

            {
                _decryptCtx = ctx;
            }

            MbedTLS.cipher_init(ctx);
            if (MbedTLS.cipher_setup(ctx, MbedTLS.cipher_info_from_string(_innerLibName)) != 0)
                throw new System.Exception("Cannot initialize mbed TLS cipher context");

            DeriveSessionKey(isEncrypt ? _encryptSalt : _decryptSalt,
                _Masterkey, _sessionKey);
            CipherSetKey(isEncrypt, _sessionKey);
        }

        private void CipherSetKey(bool isEncrypt, byte[] key)
        {
            IntPtr ctx = isEncrypt ? _encryptCtx : _decryptCtx;
            int ret = MbedTLS.cipher_setkey(ctx, key, keyLen * 8,
                isEncrypt ? MbedTLS.MBEDTLS_ENCRYPT : MbedTLS.MBEDTLS_DECRYPT);
            if (ret != 0) throw new System.Exception("failed to set key");
            ret = MbedTLS.cipher_reset(ctx);
            if (ret != 0) throw new System.Exception("failed to finish preparation");
        }

        public override void cipherEncrypt(byte[] plaintext, uint plen, byte[] ciphertext, ref uint clen)
        {
            // buf: all plaintext
            // outbuf: ciphertext + tag
            int ret;
            byte[] tagbuf = new byte[tagLen];
            uint olen = 0;
            switch (_cipher)
            {
                case CIPHER_AES:
                    ret = MbedTLS.cipher_auth_encrypt(_encryptCtx,
                        /* nonce */
                        _encNonce, (uint) nonceLen,
                        /* AD */

View on GitHub (pinned to 891d971682)

Solutions

1. Confirm EncryptorInfo.KeySize matches the cipher spec (AES-128-GCM = 16, AES-256-GCM = 32, chacha20-poly1305 = 32).
2. Verify the master key buffer was fully derived to keyLen before SetKey.
3. Ensure the mbedTLS build supports the key length for that cipher.

Example fix

// before
int ret = MbedTLS.cipher_setkey(ctx, key, keyLen * 8, isEncrypt ? MbedTLS.MBEDTLS_ENCRYPT : MbedTLS.MBEDTLS_DECRYPT);
if (ret != 0) throw new System.Exception("failed to set key");

// after
if (ret != 0)
    throw new System.Exception($"failed to set key (ret={ret}, keyLen={keyLen}, bits={keyLen*8})");

// Confirm key length matches the cipher before setkey
if (key.Length != keyLen) throw new ArgumentException($"key length {key.Length} != {keyLen}");

bool KeyLengthMatchesCipher(string method, int keyLen) =>
    (method.Contains("-128-") && keyLen == 16) ||
    (method.Contains("-256-") && keyLen == 32) ||
    (method.StartsWith("chacha20") && keyLen == 32);

try { CipherSetKey(isEncrypt, key); }
catch (Exception ex) when (ex.Message == "failed to set key")
{ /* log keyLen vs cipher expectation, abort session */ }

When it happens

Trigger: keyLen does not match what the loaded cipher expects (e.g. 24-byte key passed to AES-128-GCM); the context was set up for a cipher that rejects the key size; keyLen computed incorrectly from EncryptorInfo.

Common situations: EncryptorInfo.KeySize wrong for the method; a salt/key buffer shorter than declared so derived key is wrong length; using a method whose key size the mbedTLS build does not support.

Related errors

• openssl: fail to set key length
• method not found
• failed to generate session key
• Cannot initialize mbed TLS cipher context
• failed to finish preparation

AI-assisted analysis of shadowsocks/shadowsocks-windows@891d971682 (2026-08-13). Data as JSON: /api/errors/53f45feacf553d5a. Report an issue: GitHub.