{"record":{"id":"5dce935a66dc9c6b","repo":"sgl-project/sglang","slug":"staging-bulk-rdma-transfer-failed-with-ret-ret","errorCode":null,"errorMessage":"[Staging] Bulk RDMA transfer failed with ret={ret}. src_ptr=0x{staging_buffer.get_ptr():x}, dst_ptr=0x{dst_staging_ptr + rank_offset:x}, size={local_bytes}. The decode staging buffer may not be properly registered.","messagePattern":"\\[Staging\\] Bulk RDMA transfer failed with ret=(.+?)\\. src_ptr=0x(.+?), dst_ptr=0x(.+?), size=(.+?)\\. The decode staging buffer may not be properly registered\\.","errorType":"exception","errorClass":"RuntimeError","httpStatus":null,"severity":"critical","filePath":"python/sglang/srt/disaggregation/mooncake/conn.py","lineNumber":600,"sourceCode":"            transfer_blocks = [\n                (\n                    staging_buffer.get_ptr(),\n                    dst_staging_ptr + rank_offset,\n                    local_bytes,\n                )\n            ]\n        else:\n            transfer_blocks = [\n                (\n                    staging_buffer.get_ptr() + src_idx * per_layer_bytes,\n                    dst_staging_ptr + rank_offset + dst_idx * per_layer_bytes,\n                    per_layer_bytes,\n                )\n                for src_idx, dst_idx in pairs\n            ]\n        ret = self._transfer_data(mooncake_session_id, transfer_blocks)\n        if ret != 0:\n            raise RuntimeError(\n                f\"[Staging] Bulk RDMA transfer failed with ret={ret}. \"\n                f\"src_ptr=0x{staging_buffer.get_ptr():x}, \"\n                f\"dst_ptr=0x{dst_staging_ptr + rank_offset:x}, size={local_bytes}. \"\n                f\"The decode staging buffer may not be properly registered.\"\n            )\n        return ret\n\n    def _transfer_data(self, mooncake_session_id, transfer_blocks):\n        if not transfer_blocks:\n            return 0\n\n        src_addrs, dst_addrs, lengths = zip(*transfer_blocks)\n        return self.engine.batch_transfer_sync(\n            mooncake_session_id, list(src_addrs), list(dst_addrs), list(lengths)\n        )\n\n    def _send_kvcache_generic(\n        self,","sourceCodeStart":582,"sourceCodeEnd":618,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/srt/disaggregation/mooncake/conn.py#L582-L618","documentation":"The mooncake connection's staged bulk RDMA transfer returned a non-zero code while moving decode-side KV from the staging buffer to destination slots. The message dumps src/dst pointers and size plus a hint about buffer registration.","triggerScenarios":"send_kvcache_staged failing in _transfer_data: staging buffer not memory-registered with the mooncake engine, destination MR expired, or fabric fault.","commonSituations":"Decode engine restart invalidating MRs, buffer reuse across allocator growth without re-registration, or RDMA driver issues.","solutions":["Ensure the decode staging buffer is registered with the mooncake engine before transfer (check registration logs)","Restart prefill/decode instances to re-register memory regions","Validate RDMA fabric health between the peers","Check for a version mismatch of mooncake engine between prefill and decode"],"exampleFix":null,"handlingStrategy":"retry","validationCode":null,"typeGuard":null,"tryCatchPattern":"catch RuntimeError matching 'Bulk RDMA transfer failed'; re-register staging buffer then retry once; else escalate room failure","preventionTips":["Register the decode staging buffer at allocation time and re-register on growth","Monitor mooncake engine registration events in logs"],"tags":["mooncake","rdma","kv-cache","staging"],"backgroundTag":"rdma-transfer-failed","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}