shadowsocks/shadowsocks-windows · critical · CryptoErrorException

openssl: fail to finalize AEAD

Error message

openssl: fail to finalize AEAD

What it means

Thrown as CryptoErrorException when EVP_CipherFinal_ex returns non-1 during AEAD encryption. CipherFinal_ex flushes the cipher pipeline and, for GCM, finalises tag material. Because AEAD ciphers stream all ciphertext through CipherUpdate with padding disabled, Final succeeding is the expected steady state; failure after a successful Update signals an internally inconsistent or corrupted cipher context.

Source

Thrown at shadowsocks-csharp/Encryption/AEAD/AEADOpenSSLEncryptor.cs:101

        }

        public override void cipherEncrypt(byte[] plaintext, uint plen, byte[] ciphertext, ref uint clen)
        {
            OpenSSL.SetCtxNonce(_encryptCtx, _encNonce, true);
            // buf: all plaintext
            // outbuf: ciphertext + tag
            int ret;
            int tmpLen = 0;
            clen = 0;
            var tagBuf = new byte[tagLen];

            ret = OpenSSL.EVP_CipherUpdate(_encryptCtx, ciphertext, out tmpLen,
                plaintext, (int) plen);
            if (ret != 1) throw new CryptoErrorException("openssl: fail to encrypt AEAD");
            clen += (uint) tmpLen;
            // For AEAD cipher, it should not output anything
            ret = OpenSSL.EVP_CipherFinal_ex(_encryptCtx, ciphertext, ref tmpLen);
            if (ret != 1) throw new CryptoErrorException("openssl: fail to finalize AEAD");
            if (tmpLen > 0)
            {
                throw new System.Exception("openssl: fail to finish AEAD");
            }

            OpenSSL.AEADGetTag(_encryptCtx, tagBuf, tagLen);
            Array.Copy(tagBuf, 0, ciphertext, clen, tagLen);
            clen += (uint) tagLen;
        }

        public override void cipherDecrypt(byte[] ciphertext, uint clen, byte[] plaintext, ref uint plen)
        {
            OpenSSL.SetCtxNonce(_decryptCtx, _decNonce, false);
            // buf: ciphertext + tag
            // outbuf: plaintext
            int ret;
            int tmpLen = 0;
            plen = 0;

View on GitHub (pinned to 891d971682)

Solutions

  1. Confirm the method string on both peers is one of the supported AEAD ciphers (aes-128-gcm, aes-192-gcm, aes-256-gcm, chacha20-ietf-poly1305) and matches exactly.
  2. Re-derive the session subkey from a fresh salt and re-run InitCipher on a new encryptor instance.
  3. Audit InitCipher to confirm every EVP_CipherInit_ex / EVP_CIPHER_CTX_ctrl / set_key_length return code is checked, not swallowed.
  4. Verify native libcrypto integrity (reinstall or pin the OpenSSL build).

Example fix

// before
enc.cipherEncrypt(plain, (uint)plain.Length, cipher, ref clen);

// after — recreate the context once on a finalize failure
try {
    enc.cipherEncrypt(plain, (uint)plain.Length, cipher, ref clen);
} catch (CryptoErrorException ex) when (ex.Message.Contains("finalize")) {
    enc.Dispose();
    enc = new AEADOpenSSLEncryptor(method, password);
    enc.InitCipher(salt, true, isUdp);
    enc.cipherEncrypt(plain, (uint)plain.Length, cipher, ref clen);
}
Defensive patterns

Strategy: try-catch

Validate before calling

// nothing to validate at the call site; verify the init steps succeeded upstream
if (_cipherInfoPtr == IntPtr.Zero || _encryptCtx == IntPtr.Zero)
    throw new InvalidOperationException("AEAD context not initialised");

Try / catch

try {
    enc.cipherEncrypt(plain, (uint)plain.Length, cipher, ref clen);
} catch (CryptoErrorException ex) when (ex.Message.Contains("finalize")) {
    // re-key with a fresh salt once on a new encryptor
    enc.Dispose();
    enc = new AEADOpenSSLEncryptor(method, password);
    enc.InitCipher(salt, true, isUdp);
    enc.cipherEncrypt(plain, (uint)plain.Length, cipher, ref clen);
}

Prevention

When it happens

Trigger: The EVP context is corrupted by a memory overwrite or use-after-free between Update and Final. The padding/IV-length ctrl calls in InitCipher were skipped, reordered, or returned non-1. The cipher object behind _cipherInfoPtr is not actually an AEAD type despite the _ciphers entry. A FIPS/self-test abort or native libcrypto fault on the running context.

Common situations: Method strings differ between client and server (one side aes-256-gcm against a peer built on a mismatched lib). Corrupted key-derivation output from a malformed salt. Partial upgrade of the OpenSSL DLL leaving libcrypto internally inconsistent. Heap corruption from another thread stomping the context.

Related errors

AI-assisted analysis of shadowsocks/shadowsocks-windows@891d971682 (2026-08-13). Data as JSON: /api/errors/cae99c79b75003a6. Report an issue: GitHub.