shadowsocks/shadowsocks-windows · critical · System.Exception

failed to generate session key

Error message

failed to generate session key

What it means

Thrown from DeriveSessionKey when MbedTLS.hkdf returns a non-zero status, meaning HKDF (RFC 5869) key derivation failed inside the native mbedTLS library. HKDF is used to derive the per-session subkey from the salt and master key. A non-zero return almost always indicates an invalid parameter (bad lengths) rather than a random failure.

Source

Thrown at shadowsocks-csharp/Encryption/AEAD/AEADEncryptor.cs:124

                {
                    md5sum = MbedTLS.MD5(password);
                }
                else
                {
                    Array.Copy(md5sum, 0, result, 0, MD5_LEN);
                    Array.Copy(password, 0, result, MD5_LEN, password.Length);
                    md5sum = MbedTLS.MD5(result);
                }
                Array.Copy(md5sum, 0, key, i, Math.Min(MD5_LEN, keylen - i));
                i += MD5_LEN;
            }
        }

        public void DeriveSessionKey(byte[] salt, byte[] masterKey, byte[] sessionKey)
        {
            int ret = MbedTLS.hkdf(salt, saltLen, masterKey, keyLen, InfoBytes, InfoBytes.Length, sessionKey,
                keyLen);
            if (ret != 0) throw new System.Exception("failed to generate session key");
        }

        protected void IncrementNonce(bool isEncrypt)
        {
            lock (_nonceIncrementLock) {
                Sodium.sodium_increment(isEncrypt ? _encNonce : _decNonce, nonceLen);
            }
        }

        public virtual void InitCipher(byte[] salt, bool isEncrypt, bool isUdp)
        {
            if (isEncrypt) {
                _encryptSalt = new byte[saltLen];
                Array.Copy(salt, _encryptSalt, saltLen);
            } else {
                _decryptSalt = new byte[saltLen];
                Array.Copy(salt, _decryptSalt, saltLen);
            }

View on GitHub (pinned to 891d971682)

Solutions

1. Verify the EncryptorInfo for the selected method declares correct KeySize, SaltSize, and InfoBytes.
2. Check that the salt buffer length equals saltLen and masterKey length is at least keyLen before calling.
3. If you added a custom AEAD method, confirm its sizes match the spec (e.g. GCM salt = 16, key = 16/32).
4. Log the exact lengths and the mbedTLS return code to pinpoint the bad parameter.

Example fix

// before
int ret = MbedTLS.hkdf(salt, saltLen, masterKey, keyLen, InfoBytes, InfoBytes.Length, sessionKey, keyLen);
if (ret != 0) throw new System.Exception("failed to generate session key");

// after: include ret and lengths for diagnosis
if (ret != 0)
    throw new System.Exception($"failed to generate session key (ret={ret}, saltLen={saltLen}, keyLen={keyLen})");

// Validate buffer sizes before calling hkdf
if (salt == null || salt.Length < saltLen) throw new ArgumentException("bad salt");
if (masterKey == null || masterKey.Length < keyLen) throw new ArgumentException("bad masterKey");
if (sessionKey == null || sessionKey.Length < keyLen) throw new ArgumentException("bad sessionKey");

bool SizesConsistent(EncryptorInfo i) =>
    i.KeySize > 0 && i.SaltSize > 0 && i.NonceSize > 0 && i.TagSize > 0;

try { DeriveSessionKey(salt, masterKey, sessionKey); }
catch (Exception ex) when (ex.Message.Contains("session key"))
{ /* abort connection; log saltLen/keyLen/InfoBytes length */ }

When it happens

Trigger: saltLen, keyLen, or InfoBytes.Length passed to hkdf is zero or out of range; masterKey buffer shorter than keyLen; the salt buffer is null/undersized. These usually stem from a cipher whose EncryptorInfo declared wrong sizes, or from a salt that was not generated/padded correctly.

Common situations: A misconfigured AEAD cipher with incorrect KeySize/SaltSize in its EncryptorInfo; a custom method added without correct size metadata; buffer reuse bug leaving a salt shorter than declared.

Related errors

• Cannot initialize mbed TLS cipher context
• failed to set key
• failed to finish preparation
• ret is {0}
• method not found

AI-assisted analysis of shadowsocks/shadowsocks-windows@891d971682 (2026-08-13). Data as JSON: /api/errors/bc6c469ddb347119. Report an issue: GitHub.