microsoft/garnet · critical · Exception

Throttle count is negative

Error message

Throttle count is negative

What it means

ClientTcpNetworkSender.SendResponse increments a throttleCount via Interlocked.Increment and throws if the result is negative. Because throttleCount is only ever incremented on send and decremented on completion, a negative value means the 32-bit counter overflowed past int.MaxValue — i.e. more than ~2.1 billion outstanding sends were counted without the matching decrements, or the counter was corrupted. This is a self-consistency check indicating a serious accounting bug or a leaked/never-completed send path, not normal backpressure (that uses the separate throttle semaphore when cnt > ThrottleMax).

Source

Thrown at libs/client/ClientTcpNetworkSender.cs:47

            : base(socket, networkBufferSettings, networkPool, networkSendThrottleMax)
        {
            this.callback = callback;
            this.reusableSaea = new SimpleObjectPool<SocketAsyncEventArgs>(() =>
            {
                var s = new SocketAsyncEventArgs();
                s.Completed += SeaaBuffer_Completed;
                return s;
            });
        }

        /// <inheritdoc />
        public override void SendResponse(byte[] buffer, int offset, int count, object context)
        {
            var cnt = Interlocked.Increment(ref throttleCount);
            if (cnt < 0)
            {
                Interlocked.Decrement(ref throttleCount);
                throw new Exception("Throttle count is negative");
            }
            if (cnt > ThrottleMax)
                throttle.Wait();

            var s = reusableSaea.Checkout();
            s.SetBuffer(buffer, offset, count);
            s.UserToken = context;
            try
            {
                if (!socket.SendAsync(s))
                    SeaaBuffer_Completed(null, s);
            }
            catch
            {
                reusableSaea.Return(s);
                if (Interlocked.Decrement(ref throttleCount) >= ThrottleMax)
                    throttle.Release();
                // Rethrow exception as session is not usable

View on GitHub (pinned to 951b0fc683)

Solutions

1. Treat this as a defect: recycle/reconnect the ClientTcpNetworkSender periodically before the counter can overflow.
2. Ensure every SendResponse path has a guaranteed decrement (success, synchronous completion, exception, and dispose paths).
3. Report to the Garnet maintainers with a reproduction — a negative count implies an unmatched increment/decrement bug.

Example fix

// before — single long-lived sender under sustained load
// (counter eventually overflows after >2.1B sends)

// after — recycle the network sender/connection on a send-count budget
if (Interlocked.Read(ref sendsSinceReconnect) > SendBudget)
    await client.ReconnectAsync();

// No caller pre-check; recycle the network sender on a send-count budget to avoid overflow
if (Interlocked.Read(ref sendsSinceReconnect) > SendBudget)
    await client.ReconnectAsync();

try { networkSender.SendResponse(buf, off, count, ctx); }
catch (Exception ex) when (ex.Message.Contains("Throttle count is negative")) { await client.ReconnectAsync(); /* retry on fresh sender */ }

When it happens

Trigger: An extremely long-lived ClientTcpNetworkSender accumulating sends until throttleCount wraps to negative; or a path that increments throttleCount without a matching decrement on completion (a send-completion callback never firing, so the decrement in DisposeNetworkSender/SeaaBuffer_Completed is skipped).

Common situations: Long-running connections that never recycle the network sender under very high throughput; a bug where SocketAsyncEventArgs completion is lost; concurrent dispose racing with SendResponse corrupting the counter.

Related errors

• Getting handle in disposed device
• Exceeded maximum number of active LightEpoch instances {Acti
• LightEpoch
• Getting handle in disposed device
• Failed to connect at {EndPoint}

AI-assisted analysis of microsoft/garnet@951b0fc683 (2026-08-13). Data as JSON: /api/errors/7071e7dea8e21a50. Report an issue: GitHub.