shadowsocks/shadowsocks-windows · error · System.Exception

openssl: fail to finish AEAD

Error message

openssl: fail to finish AEAD

What it means

Thrown as a plain System.Exception when EVP_CipherFinal_ex writes more than 0 bytes during AEAD encryption. For AEAD stream ciphers (AES-GCM, ChaCha20-Poly1305) with padding disabled, all ciphertext is emitted by CipherUpdate and Final must produce zero bytes. A positive tmpLen means the cipher is behaving like a padded block cipher, i.e. it is not configured as a true AEAD stream.

Source

Thrown at shadowsocks-csharp/Encryption/AEAD/AEADOpenSSLEncryptor.cs:104

        {
            OpenSSL.SetCtxNonce(_encryptCtx, _encNonce, true);
            // buf: all plaintext
            // outbuf: ciphertext + tag
            int ret;
            int tmpLen = 0;
            clen = 0;
            var tagBuf = new byte[tagLen];

            ret = OpenSSL.EVP_CipherUpdate(_encryptCtx, ciphertext, out tmpLen,
                plaintext, (int) plen);
            if (ret != 1) throw new CryptoErrorException("openssl: fail to encrypt AEAD");
            clen += (uint) tmpLen;
            // For AEAD cipher, it should not output anything
            ret = OpenSSL.EVP_CipherFinal_ex(_encryptCtx, ciphertext, ref tmpLen);
            if (ret != 1) throw new CryptoErrorException("openssl: fail to finalize AEAD");
            if (tmpLen > 0)
            {
                throw new System.Exception("openssl: fail to finish AEAD");
            }

            OpenSSL.AEADGetTag(_encryptCtx, tagBuf, tagLen);
            Array.Copy(tagBuf, 0, ciphertext, clen, tagLen);
            clen += (uint) tagLen;
        }

        public override void cipherDecrypt(byte[] ciphertext, uint clen, byte[] plaintext, ref uint plen)
        {
            OpenSSL.SetCtxNonce(_decryptCtx, _decNonce, false);
            // buf: ciphertext + tag
            // outbuf: plaintext
            int ret;
            int tmpLen = 0;
            plen = 0;

            // split tag
            byte[] tagbuf = new byte[tagLen];

View on GitHub (pinned to 891d971682)

Solutions

1. Verify EVP_CIPHER_CTX_set_padding(_encryptCtx, 0) is executed in InitCipher after the key is set and returns 1.
2. Confirm the cipher is one of the AEAD types and that _cipherInfoPtr points to a GCM or ChaCha20-Poly1305 cipher (OpenSSL.GetCipherInfo must not return Zero).
3. Ensure a context is never reused after switching methods; always build a fresh encryptor.

Example fix

// before — padding never disabled
ret = OpenSSL.EVP_CipherInit_ex(ctx, _cipherInfoPtr, IntPtr.Zero, key, null, enc);

// after — explicitly disable padding after init and check the return code
ret = OpenSSL.EVP_CipherInit_ex(ctx, _cipherInfoPtr, IntPtr.Zero, key, null, enc);
if (ret != 1) throw new Exception("openssl: fail to init ctx");
int pad = OpenSSL.EVP_CIPHER_CTX_set_padding(ctx, 0);
if (pad != 1) throw new Exception("openssl: fail to disable padding");

// confirm padding is disabled — an AEAD cipher must stream with zero final output
int pad = OpenSSL.EVP_CIPHER_CTX_set_padding(ctx, 0);
if (pad != 1) throw new InvalidOperationException("cannot disable padding for AEAD");

try { enc.cipherEncrypt(plain, (uint)plain.Length, cipher, ref clen); }
catch (Exception ex) when (ex.Message.Contains("finish AEAD")) {
    // cipher is not behaving as an AEAD stream — this is a config/setup bug
    logger.Error("cipher not behaving as AEAD stream; check padding and method");
    throw;
}

When it happens

Trigger: InitCipher's EVP_CIPHER_CTX_set_padding(ctx, 0) call was skipped or returned non-1. The selected method resolved to a non-AEAD cipher object. The same context was reused after a method switch without re-init. An OpenSSL build whose GCM final incorrectly emits trailing data (rare bug).

Common situations: A typo in the method name causes fallback to a non-AEAD/block cipher. A modified InitCipher drops the set-padding call. The _ciphers dictionary is extended with a non-AEAD entry that shares this code path.

Related errors

• openssl: fail to encrypt AEAD
• openssl: fail to finalize AEAD
• openssl: cipher not found
• openssl: fail to create ctx
• openssl: fail to init ctx

AI-assisted analysis of shadowsocks/shadowsocks-windows@891d971682 (2026-08-13). Data as JSON: /api/errors/6344e381faf896df. Report an issue: GitHub.