microsoft/garnet · error · GarnetException

Disconnected

Error message

Disconnected

What it means

Thrown by LightClient.CompletePendingRequests while spin-waiting for outstanding async request tokens to be acknowledged by the server. The loop checks socket.Connected on each yield; if the underlying socket has dropped (RST, FIN, TCP reset, or server-side disconnect), this GarnetException is raised immediately rather than waiting for the deadline. It means the connection died while responses were still in flight.

Source

Thrown at libs/common/LightClient.cs:285

        /// <summary>
        /// Send len bytes from networkSender buffer.
        /// </summary>
        /// <param name="len"></param>
        /// <param name="numTokens"></param>
        public override void Send(int len, int numTokens = 1)
        {
            Interlocked.Add(ref numPendingRequests, numTokens);
            networkSender.SendResponse(0, len);
            networkSender.GetResponseObject();
        }

        public override bool CompletePendingRequests(int timeout = -1, CancellationToken token = default)
        {
            var deadline = timeout == -1 ? DateTime.MaxValue.Ticks : DateTime.Now.AddMilliseconds(timeout).Ticks;
            while (numPendingRequests > 0 && DateTime.Now.Ticks < deadline)
            {
                if (token.IsCancellationRequested) return false;
                if (!socket.Connected) throw new GarnetException("Disconnected");
                Thread.Yield();
            }

            // TODO: Re-enable to catch token counting errors.
            // Debug.Assert(numPendingRequests == 0, $"numPendingRequests cannot be nonzero, numPendingRequests = {numPendingRequests} | " +
            //    $"timeout = {timeout}, deadline: {deadline} > now: {DateTime.Now.Ticks}");
            return numPendingRequests == 0;
        }

        private static unsafe (int, int) DefaultLightReceiveUnsafe(byte* buf, int bytesRead, int opType) => (bytesRead, 1);

        /// <inheritdoc />
        public override void Dispose()
        {
            networkSender.ReturnResponseObject();
            networkHandler?.Dispose();
            networkPool?.Dispose();
        }

View on GitHub (pinned to 951b0fc683)

Solutions

1. Check server logs for crashes, OOM, or explicit client-disconnect events around the time of failure.
2. Verify there is no intermediate proxy/load balancer with an idle timeout shorter than your operation duration.
3. Recreate the LightClient and reconnect before retrying the request batch.
4. If the server crashed, identify and fix the root cause (memory pressure, assertion failure) before reconnecting.
5. Use a lower CompletePendingRequests timeout so dead connections are detected sooner rather than spinning.

Example fix

try
{
    client.Send(reqBuf, reqLen, numTokens: 3);
    client.CompletePendingRequests(timeout: 5000);
}
catch (GarnetException ex) when (ex.Message == "Disconnected")
{
    // recreate client and retry
    client.Dispose();
    client = new LightClient(host, port);
    client.Connect();
}

bool IsStillConnected(LightClient client)
{
    // Best-effort: check the underlying socket if exposed; otherwise wrap CompletePendingRequests.
    return client != null; // no public IsConnected; rely on try-catch below
}

try
{
    if (!client.CompletePendingRequests(timeout: 5000, token: cts.Token))
        throw new TimeoutException("Pending requests did not complete in time.");
}
catch (GarnetException ex) when (ex.Message == "Disconnected")
{
    // Recreate the client and retry the batch
    client.Dispose();
    client = new LightClient(host, port);
    client.Connect();
}

When it happens

Trigger: Calling CompletePendingRequests after Send/SendAsync when the server has closed, reset, or crashed between the request send and the response completion. Also occurs if the network drops mid-flight (e.g., a load balancer idle timeout killed the connection) or if the server is forcefully killed.

Common situations: Server process killed or crashed during a batch of pipelined requests; aggressive idle-connection timeout on a proxy/LB closes the socket; network partition between client and server; client reused a LightClient instance after a prior error that half-closed the socket; long-running operation exceeded server-side client timeout.

Related errors

• Failed to connect at {endpoint}
• Failed to connect at {EndPoint}
• Failed to connect at {EndPoint}
• Need to provide SslClientAuthenticationOptions when TLS is e
• Cannot provide SslClientAuthenticationOptions when TLS is di

AI-assisted analysis of microsoft/garnet@951b0fc683 (2026-08-13). Data as JSON: /api/errors/aeb817e5722ac73b. Report an issue: GitHub.