{"record":{"id":"aeb817e5722ac73b","repo":"microsoft/garnet","slug":"disconnected","errorCode":null,"errorMessage":"Disconnected","messagePattern":"Disconnected","errorType":"exception","errorClass":"GarnetException","httpStatus":null,"severity":"error","filePath":"libs/common/LightClient.cs","lineNumber":285,"sourceCode":"        /// <summary>\n        /// Send len bytes from networkSender buffer.\n        /// </summary>\n        /// <param name=\"len\"></param>\n        /// <param name=\"numTokens\"></param>\n        public override void Send(int len, int numTokens = 1)\n        {\n            Interlocked.Add(ref numPendingRequests, numTokens);\n            networkSender.SendResponse(0, len);\n            networkSender.GetResponseObject();\n        }\n\n        public override bool CompletePendingRequests(int timeout = -1, CancellationToken token = default)\n        {\n            var deadline = timeout == -1 ? DateTime.MaxValue.Ticks : DateTime.Now.AddMilliseconds(timeout).Ticks;\n            while (numPendingRequests > 0 && DateTime.Now.Ticks < deadline)\n            {\n                if (token.IsCancellationRequested) return false;\n                if (!socket.Connected) throw new GarnetException(\"Disconnected\");\n                Thread.Yield();\n            }\n\n            // TODO: Re-enable to catch token counting errors.\n            // Debug.Assert(numPendingRequests == 0, $\"numPendingRequests cannot be nonzero, numPendingRequests = {numPendingRequests} | \" +\n            //    $\"timeout = {timeout}, deadline: {deadline} > now: {DateTime.Now.Ticks}\");\n            return numPendingRequests == 0;\n        }\n\n        private static unsafe (int, int) DefaultLightReceiveUnsafe(byte* buf, int bytesRead, int opType) => (bytesRead, 1);\n\n        /// <inheritdoc />\n        public override void Dispose()\n        {\n            networkSender.ReturnResponseObject();\n            networkHandler?.Dispose();\n            networkPool?.Dispose();\n        }","sourceCodeStart":267,"sourceCodeEnd":303,"githubUrl":"https://github.com/microsoft/garnet/blob/951b0fc6838721f89d102c2bbe1b914e8d39d700/libs/common/LightClient.cs#L267-L303","documentation":"Thrown by LightClient.CompletePendingRequests while spin-waiting for outstanding async request tokens to be acknowledged by the server. The loop checks socket.Connected on each yield; if the underlying socket has dropped (RST, FIN, TCP reset, or server-side disconnect), this GarnetException is raised immediately rather than waiting for the deadline. It means the connection died while responses were still in flight.","triggerScenarios":"Calling CompletePendingRequests after Send/SendAsync when the server has closed, reset, or crashed between the request send and the response completion. Also occurs if the network drops mid-flight (e.g., a load balancer idle timeout killed the connection) or if the server is forcefully killed.","commonSituations":"Server process killed or crashed during a batch of pipelined requests; aggressive idle-connection timeout on a proxy/LB closes the socket; network partition between client and server; client reused a LightClient instance after a prior error that half-closed the socket; long-running operation exceeded server-side client timeout.","solutions":["Check server logs for crashes, OOM, or explicit client-disconnect events around the time of failure.","Verify there is no intermediate proxy/load balancer with an idle timeout shorter than your operation duration.","Recreate the LightClient and reconnect before retrying the request batch.","If the server crashed, identify and fix the root cause (memory pressure, assertion failure) before reconnecting.","Use a lower CompletePendingRequests timeout so dead connections are detected sooner rather than spinning."],"exampleFix":"try\n{\n    client.Send(reqBuf, reqLen, numTokens: 3);\n    client.CompletePendingRequests(timeout: 5000);\n}\ncatch (GarnetException ex) when (ex.Message == \"Disconnected\")\n{\n    // recreate client and retry\n    client.Dispose();\n    client = new LightClient(host, port);\n    client.Connect();\n}","handlingStrategy":"try-catch","validationCode":"bool IsStillConnected(LightClient client)\n{\n    // Best-effort: check the underlying socket if exposed; otherwise wrap CompletePendingRequests.\n    return client != null; // no public IsConnected; rely on try-catch below\n}","typeGuard":null,"tryCatchPattern":"try\n{\n    if (!client.CompletePendingRequests(timeout: 5000, token: cts.Token))\n        throw new TimeoutException(\"Pending requests did not complete in time.\");\n}\ncatch (GarnetException ex) when (ex.Message == \"Disconnected\")\n{\n    // Recreate the client and retry the batch\n    client.Dispose();\n    client = new LightClient(host, port);\n    client.Connect();\n}","preventionTips":["Always pair Send calls with CompletePendingRequests inside a try-catch for GarnetException(\"Disconnected\").","Use a finite timeout on CompletePendingRequests so dead connections are detected quickly.","Implement a reconnect-and-retry wrapper for batches that are safe to re-send.","Avoid reusing a LightClient after any GarnetException — recreate it instead."],"tags":["network","connection","client","lightclient","disconnect","spin-wait"],"backgroundTag":null,"analyzedSha":"951b0fc6838721f89d102c2bbe1b914e8d39d700","analyzedAt":"2026-08-13T19:01:32.939Z","schemaVersion":2},"datasetVersion":"2026-08-13T19:17:28.613Z"}