shadowsocks/shadowsocks-windows · critical · CryptoErrorException

openssl: fail to encrypt AEAD

Error message

openssl: fail to encrypt AEAD

What it means

Thrown as CryptoErrorException when OpenSSL's EVP_CipherUpdate returns a value other than 1 during AEAD encryption. EVP_CipherUpdate feeds plaintext into the native cipher context (_encryptCtx) and writes ciphertext; a non-1 return means libcrypto rejected the operation. This wraps the EVP AEAD pipeline used for AES-GCM and ChaCha20-Poly1305 in Shadowsocks.

Source

Thrown at shadowsocks-csharp/Encryption/AEAD/AEADOpenSSLEncryptor.cs:97

                null,
                isEncrypt ? OpenSSL.OPENSSL_ENCRYPT : OpenSSL.OPENSSL_DECRYPT);
            if (ret != 1) throw new System.Exception("openssl: cannot set key");
            OpenSSL.EVP_CIPHER_CTX_set_padding(ctx, 0);
        }

        public override void cipherEncrypt(byte[] plaintext, uint plen, byte[] ciphertext, ref uint clen)
        {
            OpenSSL.SetCtxNonce(_encryptCtx, _encNonce, true);
            // buf: all plaintext
            // outbuf: ciphertext + tag
            int ret;
            int tmpLen = 0;
            clen = 0;
            var tagBuf = new byte[tagLen];

            ret = OpenSSL.EVP_CipherUpdate(_encryptCtx, ciphertext, out tmpLen,
                plaintext, (int) plen);
            if (ret != 1) throw new CryptoErrorException("openssl: fail to encrypt AEAD");
            clen += (uint) tmpLen;
            // For AEAD cipher, it should not output anything
            ret = OpenSSL.EVP_CipherFinal_ex(_encryptCtx, ciphertext, ref tmpLen);
            if (ret != 1) throw new CryptoErrorException("openssl: fail to finalize AEAD");
            if (tmpLen > 0)
            {
                throw new System.Exception("openssl: fail to finish AEAD");
            }

            OpenSSL.AEADGetTag(_encryptCtx, tagBuf, tagLen);
            Array.Copy(tagBuf, 0, ciphertext, clen, tagLen);
            clen += (uint) tagLen;
        }

        public override void cipherDecrypt(byte[] ciphertext, uint clen, byte[] plaintext, ref uint plen)
        {
            OpenSSL.SetCtxNonce(_decryptCtx, _decNonce, false);
            // buf: ciphertext + tag

View on GitHub (pinned to 891d971682)

Solutions

  1. Verify InitCipher(salt, isEncrypt: true, isUdp) was invoked exactly once before the first cipherEncrypt call and that none of its EVP_* calls returned non-1.
  2. Ensure each AEADOpenSSLEncryptor instance is bound to a single connection/thread; never share one across sockets.
  3. Confirm the bundled libcrypto matches the process architecture and the OpenSSL version expected by the wrapper, and that the native DLL loads at startup.
  4. Allocate the ciphertext buffer to at least plen + tagLen bytes before calling cipherEncrypt.

Example fix

// before
enc.cipherEncrypt(plain, (uint)plain.Length, cipher, ref clen); // _encryptCtx not initialised

// after
enc.InitCipher(salt, true, isUdp);                       // must init first
if (cipher.Length < plain.Length + enc.tagLen)
    throw new InvalidOperationException("ciphertext buffer too small");
enc.cipherEncrypt(plain, (uint)plain.Length, cipher, ref clen);
Defensive patterns

Strategy: try-catch

Validate before calling

// ensure the context is initialised and the buffer is large enough before encrypting
if (_encryptCtx == IntPtr.Zero)
    throw new InvalidOperationException("encryptor not initialised");
if (ciphertext == null || ciphertext.Length < plen + tagLen)
    throw new ArgumentException("ciphertext buffer too small");

Try / catch

try {
    enc.cipherEncrypt(plain, (uint)plain.Length, cipher, ref clen);
} catch (CryptoErrorException ex) {
    // the crypto context is now unusable — do not reuse it
    logger.Error(ex, "AEAD encrypt failed");
    enc.Dispose();
    throw;
}

Prevention

When it happens

Trigger: cipherEncrypt() is called before InitCipher(), so _encryptCtx is IntPtr.Zero or uninitialized when SetCtxNonce / EVP_CipherUpdate run. The native EVP context pointer was freed or corrupted (use-after-free, double Dispose). Two threads call cipherEncrypt() on the same encryptor instance simultaneously (EVP contexts are not thread-safe). The ciphertext buffer is smaller than plen + tagLen, or the subkey/nonce were derived from a bad salt.

Common situations: Bundled libcrypto is the wrong architecture (32 vs 64-bit) or an incompatible OpenSSL major version that silently fails CIPHER operations. An encryptor instance is reused across multiple sockets/connections without synchronization. A build where InitCipher's EVP_CipherInit_ex failed but the error was swallowed. Heap corruption from a concurrent native call clobbers the context.

Related errors

AI-assisted analysis of shadowsocks/shadowsocks-windows@891d971682 (2026-08-13). Data as JSON: /api/errors/efb942b8142ab2cc. Report an issue: GitHub.