{"record":{"id":"65a1d7c9a26b3b5d","repo":"sgl-project/sglang","slug":"mooncake-transfer-sync-failed-for-req-id-sessio","errorCode":null,"errorMessage":"Mooncake transfer_sync failed for {req_id} (session={session_id}, nbytes={embedding.nbytes}, ret={xfer_ret})","messagePattern":"Mooncake transfer_sync failed for (.+?) \\(session=(.+?), nbytes=(.+?), ret=(.+?)\\)","errorType":"http","errorClass":"InternalError","httpStatus":500,"severity":"critical","filePath":"python/sglang/srt/disaggregation/encoder/server.py","lineNumber":1569,"sourceCode":"            if not mr_already_registered:\n                self.engine.register(embedding.data_ptr(), embedding.nbytes)\n            _t_xfer_start = time.monotonic()\n            xfer_ret = await asyncio.to_thread(\n                self.engine.transfer_sync,\n                session_id,\n                embedding.data_ptr(),\n                buffer_address,\n                embedding.nbytes,\n            )\n            xfer_ms = (time.monotonic() - _t_xfer_start) * 1000.0\n            if encoder_metrics_collector is not None:\n                encoder_metrics_collector.observe_transfer(\n                    xfer_ms / 1000.0, backend=\"mooncake\"\n                )\n            if not mr_already_registered:\n                self.engine.deregister(embedding.data_ptr())\n            if xfer_ret < 0:\n                raise InternalError(\n                    f\"Mooncake transfer_sync failed for {req_id} \"\n                    f\"(session={session_id}, nbytes={embedding.nbytes}, \"\n                    f\"ret={xfer_ret})\"\n                )\n            # Emit at INFO for slow transfers or per-send registrations.\n            if xfer_ms > 200.0 or not mr_already_registered:\n                logger.info(\n                    f\"[{req_id}] mooncake transfer_sync={xfer_ms:.1f}ms \"\n                    f\"nbytes={embedding.nbytes} shared_mr={mr_already_registered}\"\n                )\n\n            # Sibling ranks re-read mm_data here; meta_registry owns the release.\n\n        # Send ack/data\n        if url is not None:\n            endpoint = NetworkAddress.parse(url).to_tcp()\n        else:\n            endpoint = NetworkAddress(prefill_host, embedding_port).to_tcp()","sourceCodeStart":1551,"sourceCodeEnd":1587,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/srt/disaggregation/encoder/server.py#L1551-L1587","documentation":"The mooncake transfer engine's synchronous transfer (transfer_sync) returned a negative return code while pushing an embedding to the decode side. The message includes session id, byte count, and the engine return code for diagnosis.","triggerScenarios":"MooncakeStore.transfer_sync failing: broken RDMA connection, deregistered/expired session, fabric error, or wrong memory region — raised in _send after the bulk copy attempt.","commonSituations":"Decode/prefill instance restart mid-transfer, mooncake P2P fabric misconfiguration, network interruption, or buffer registration expiry under long uptime.","solutions":["Retry the request (transient fabric errors often clear)","Check mooncake engine logs for the meaning of ret and the session state","Verify RDMA devices/interfaces are healthy on both ends (ibstat, connection to mooncake metadata service)","Restart the prefill/decode pair to re-establish sessions if the session was invalidated"],"exampleFix":null,"handlingStrategy":"retry","validationCode":null,"typeGuard":null,"tryCatchPattern":"catch InternalError containing 'transfer_sync failed'; inspect ret code, retry once, then tear down the session and fail the room","preventionTips":["Keep-alive/health-check the mooncake session","Monitor transfer failure rate per session and recycle sessions proactively"],"tags":["mooncake","rdma","transfer","network"],"backgroundTag":"rdma-transfer-failed","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}