shadowsocks/shadowsocks-windows · critical · System.Exception

openssl: fail to set AEAD nonce length

Error message

openssl: fail to set AEAD nonce length

What it means

Thrown from AEADOpenSSLEncryptor.InitCipher when EVP_CIPHER_CTX_ctrl with EVP_CTRL_AEAD_SET_IVLEN returns != 1. This ctrl call sets the nonce/IV length for the AEAD cipher; OpenSSL rejects it if the length is not allowed by the cipher (e.g. not the standard 12 for GCM) or if the cipher does not support variable IV length.

Source

Thrown at shadowsocks-csharp/Encryption/AEAD/AEADOpenSSLEncryptor.cs:75

            }
            else
            {
                _decryptCtx = ctx;
            }

            DeriveSessionKey(isEncrypt ? _encryptSalt : _decryptSalt, _Masterkey,
                isEncrypt ? _opensslEncSubkey : _opensslDecSubkey);

            var ret = OpenSSL.EVP_CipherInit_ex(ctx, _cipherInfoPtr, IntPtr.Zero, null, null,
                isEncrypt ? OpenSSL.OPENSSL_ENCRYPT : OpenSSL.OPENSSL_DECRYPT);
            if (ret != 1) throw new System.Exception("openssl: fail to init ctx");

            ret = OpenSSL.EVP_CIPHER_CTX_set_key_length(ctx, keyLen);
            if (ret != 1) throw new System.Exception("openssl: fail to set key length");

            ret = OpenSSL.EVP_CIPHER_CTX_ctrl(ctx, OpenSSL.EVP_CTRL_AEAD_SET_IVLEN,
                nonceLen, IntPtr.Zero);
            if (ret != 1) throw new System.Exception("openssl: fail to set AEAD nonce length");

            ret = OpenSSL.EVP_CipherInit_ex(ctx, IntPtr.Zero, IntPtr.Zero,
                isEncrypt ? _opensslEncSubkey : _opensslDecSubkey,
                null,
                isEncrypt ? OpenSSL.OPENSSL_ENCRYPT : OpenSSL.OPENSSL_DECRYPT);
            if (ret != 1) throw new System.Exception("openssl: cannot set key");
            OpenSSL.EVP_CIPHER_CTX_set_padding(ctx, 0);
        }

        public override void cipherEncrypt(byte[] plaintext, uint plen, byte[] ciphertext, ref uint clen)
        {
            OpenSSL.SetCtxNonce(_encryptCtx, _encNonce, true);
            // buf: all plaintext
            // outbuf: ciphertext + tag
            int ret;
            int tmpLen = 0;
            clen = 0;
            var tagBuf = new byte[tagLen];

View on GitHub (pinned to 891d971682)

Solutions

1. Confirm EncryptorInfo.NonceSize is 12 for GCM/CHACHA20-POLY1305.
2. Ensure the method is genuinely an AEAD cipher handled by this code path.
3. Cross-check nonceLen against the cipher specification before init.

Example fix

// before
ret = OpenSSL.EVP_CIPHER_CTX_ctrl(ctx, OpenSSL.EVP_CTRL_AEAD_SET_IVLEN, nonceLen, IntPtr.Zero);
if (ret != 1) throw new System.Exception("openssl: fail to set AEAD nonce length");

// after
if (ret != 1)
    throw new System.Exception($"openssl: fail to set AEAD nonce length (nonceLen={nonceLen}); must be 12 for GCM/CHACHA20");

// Validate nonce length before the ctrl call
if (nonceLen != 12) throw new ArgumentException($"AEAD nonce must be 12, got {nonceLen}");

bool IsStandardAeadNonceLen(int n) => n == 12;

try { OpenSSL.EVP_CIPHER_CTX_ctrl(ctx, EVP_CTRL_AEAD_SET_IVLEN, nonceLen, IntPtr.Zero); }
catch (Exception ex) when (ex.Message.Contains("nonce length"))
{ /* fix EncryptorInfo.NonceSize to 12 */ }

When it happens

Trigger: nonceLen is not 12 for AES-GCM (the spec length); the selected cipher is not actually an AEAD cipher; nonceLen computed wrong from EncryptorInfo.NonceSize.

Common situations: EncryptorInfo.NonceSize set to a non-standard value; method misclassified as AEAD in the cipher table; a CHACHA20 cipher expecting a different IV length.

Related errors

• openssl: fail to set key length
• method not found
• failed to set key
• openssl: cipher not found
• openssl: fail to create ctx

AI-assisted analysis of shadowsocks/shadowsocks-windows@891d971682 (2026-08-13). Data as JSON: /api/errors/e16baed5cb7e87af. Report an issue: GitHub.