shadowsocks/shadowsocks-windows · critical · CryptoErrorException

ret is {0}

Error message

ret is {0}

What it means

Thrown as CryptoErrorException from cipherEncrypt when MbedTLS.cipher_auth_encrypt (AES-GCM authenticated encryption) returns non-zero. The formatted message includes the native return code. Non-zero from an AEAD encrypt call means the operation could not complete — typically an internal GCM failure rather than bad input, since the inputs were validated earlier during setup.

Source

Thrown at shadowsocks-csharp/Encryption/AEAD/AEADMbedTLSEncryptor.cs:91

            // buf: all plaintext
            // outbuf: ciphertext + tag
            int ret;
            byte[] tagbuf = new byte[tagLen];
            uint olen = 0;
            switch (_cipher)
            {
                case CIPHER_AES:
                    ret = MbedTLS.cipher_auth_encrypt(_encryptCtx,
                        /* nonce */
                        _encNonce, (uint) nonceLen,
                        /* AD */
                        IntPtr.Zero, 0,
                        /* plain */
                        plaintext, plen,
                        /* cipher */
                        ciphertext, ref olen,
                        tagbuf, (uint) tagLen);
                    if (ret != 0) throw new CryptoErrorException(String.Format("ret is {0}", ret));
                    Debug.Assert(olen == plen);
                    // attach tag to ciphertext
                    Array.Copy(tagbuf, 0, ciphertext, (int) plen, tagLen);
                    clen = olen + (uint) tagLen;
                    break;
                default:
                    throw new System.Exception("not implemented");
            }
        }

        public override void cipherDecrypt(byte[] ciphertext, uint clen, byte[] plaintext, ref uint plen)
        {
            // buf: ciphertext + tag
            // outbuf: plaintext
            int ret;
            uint olen = 0;
            // split tag
            byte[] tagbuf = new byte[tagLen];

View on GitHub (pinned to 891d971682)

Solutions

1. Log the exact ret value and look it up in mbedTLS error codes (e.g. MBEDTLS_ERR_CIPHER_*).
2. Ensure the encrypt context is freshly initialized per session and not shared across threads.
3. Chunk very large plaintexts instead of passing them in one cipher_auth_encrypt call.
4. Confirm _encNonce is being incremented (IncrementNonce) and not reused.

Example fix

// before
if (ret != 0) throw new CryptoErrorException(String.Format("ret is {0}", ret));

// after
if (ret != 0)
    throw new CryptoErrorException($"mbedTLS cipher_auth_encrypt failed (ret={ret}, plen={plen})");

// Sanity-check plaintext length against a per-call cap
const int MaxChunk = 1 << 20;
if (plen > MaxChunk) /* chunk before calling cipher_auth_encrypt */

try { cipherEncrypt(plain, plen, cipher, ref clen); }
catch (CryptoErrorException ex)
{ /* log ret code; close the relay, the session key/nonce state is suspect */ }

When it happens

Trigger: Plaintext length exceeding the cipher's per-call limits; nonce reuse collision detected internally; the encrypt context was corrupted between setup and this call; plen/tagLen combination the GCM implementation rejects.

Common situations: Encrypting an unusually large buffer in a single call; a context being reused after a partial failure left it in a bad state; nonce counter overflow; native memory corruption.

Related errors

• failed to generate session key
• Cannot initialize mbed TLS cipher context
• failed to set key
• failed to finish preparation
• method not found

AI-assisted analysis of shadowsocks/shadowsocks-windows@891d971682 (2026-08-13). Data as JSON: /api/errors/dd95e45e1b5f9f24. Report an issue: GitHub.