shadowsocks/shadowsocks-windows · critical · System.Exception

failed to set key

Error message

failed to set key

What it means

Thrown from CipherSetKey when MbedTLS.cipher_setkey returns non-zero after the session subkey is derived. cipher_setkey configures the context for encrypt/decrypt with the given key and key length; failure indicates the key length or mode is invalid for the loaded cipher (e.g. key length not matching the cipher's expected size).

Solutions

1. Confirm EncryptorInfo.KeySize matches the cipher spec (AES-128-GCM = 16, AES-256-GCM = 32, chacha20-poly1305 = 32).
2. Verify the master key buffer was fully derived to keyLen before SetKey.
3. Ensure the mbedTLS build supports the key length for that cipher.

Example fix

// before
int ret = MbedTLS.cipher_setkey(ctx, key, keyLen * 8, isEncrypt ? MbedTLS.MBEDTLS_ENCRYPT : MbedTLS.MBEDTLS_DECRYPT);
if (ret != 0) throw new System.Exception("failed to set key");

// after
if (ret != 0)
    throw new System.Exception($"failed to set key (ret={ret}, keyLen={keyLen}, bits={keyLen*8})");

// Confirm key length matches the cipher before setkey
if (key.Length != keyLen) throw new ArgumentException($"key length {key.Length} != {keyLen}");

bool KeyLengthMatchesCipher(string method, int keyLen) =>
    (method.Contains("-128-") && keyLen == 16) ||
    (method.Contains("-256-") && keyLen == 32) ||
    (method.StartsWith("chacha20") && keyLen == 32);

try { CipherSetKey(isEncrypt, key); }
catch (Exception ex) when (ex.Message == "failed to set key")
{ /* log keyLen vs cipher expectation, abort session */ }

When it happens

Trigger: keyLen does not match what the loaded cipher expects (e.g. 24-byte key passed to AES-128-GCM); the context was set up for a cipher that rejects the key size; keyLen computed incorrectly from EncryptorInfo.

Common situations: EncryptorInfo.KeySize wrong for the method; a salt/key buffer shorter than declared so derived key is wrong length; using a method whose key size the mbedTLS build does not support.

Related errors

• openssl: fail to set key length
• Cannot initialize mbed TLS cipher context
• failed to finish preparation
• failed to generate session key
• method not found

AI-assisted analysis of shadowsocks/shadowsocks-windows@891d971682 (2026-08-13). Data as JSON: /api/errors/53f45feacf553d5a. Report an issue: GitHub.

Appendix: source

Thrown at shadowsocks-csharp/Encryption/AEAD/AEADMbedTLSEncryptor.cs:66

            {
                _decryptCtx = ctx;
            }

            MbedTLS.cipher_init(ctx);
            if (MbedTLS.cipher_setup(ctx, MbedTLS.cipher_info_from_string(_innerLibName)) != 0)
                throw new System.Exception("Cannot initialize mbed TLS cipher context");

            DeriveSessionKey(isEncrypt ? _encryptSalt : _decryptSalt,
                _Masterkey, _sessionKey);
            CipherSetKey(isEncrypt, _sessionKey);
        }

        private void CipherSetKey(bool isEncrypt, byte[] key)
        {
            IntPtr ctx = isEncrypt ? _encryptCtx : _decryptCtx;
            int ret = MbedTLS.cipher_setkey(ctx, key, keyLen * 8,
                isEncrypt ? MbedTLS.MBEDTLS_ENCRYPT : MbedTLS.MBEDTLS_DECRYPT);
            if (ret != 0) throw new System.Exception("failed to set key");
            ret = MbedTLS.cipher_reset(ctx);
            if (ret != 0) throw new System.Exception("failed to finish preparation");
        }

        public override void cipherEncrypt(byte[] plaintext, uint plen, byte[] ciphertext, ref uint clen)
        {
            // buf: all plaintext
            // outbuf: ciphertext + tag
            int ret;
            byte[] tagbuf = new byte[tagLen];
            uint olen = 0;
            switch (_cipher)
            {
                case CIPHER_AES:
                    ret = MbedTLS.cipher_auth_encrypt(_encryptCtx,
                        /* nonce */
                        _encNonce, (uint) nonceLen,
                        /* AD */

View on GitHub (pinned to 891d971682)