{"record":{"id":"8c2d5dfa65766ebc","repo":"sgl-project/sglang","slug":"failed-to-register-buffer-to-mooncake-store-error","errorCode":null,"errorMessage":"Failed to register buffer to Mooncake Store, error code: {ret_code}","messagePattern":"Failed to register buffer to Mooncake Store, error code: (.+?)","errorType":"exception","errorClass":"RuntimeError","httpStatus":null,"severity":"critical","filePath":"python/sglang/srt/mem_cache/storage/mooncake_store/mooncake_store.py","lineNumber":327,"sourceCode":"\n        else:\n            config = MooncakeStoreConfig.load_from_env()\n            logger.info(\"Mooncake Configuration loaded from env successfully.\")\n\n        return config\n\n    def register_buffer(self, tensor: torch.Tensor):\n        if self.store is None:\n            raise RuntimeError(\"Mooncake store is not initialized.\")\n        ptr = tensor.data_ptr()\n        size = tensor.numel() * tensor.element_size()\n        if size == 0:\n            # conv-only models have a 0-element ssm state; nothing to register\n            return\n        ret_code = self.store.register_buffer(ptr, size)\n        if ret_code != 0:\n            logger.error(f\"Failed to register buffer, error code: {ret_code}\")\n            raise RuntimeError(\n                f\"Failed to register buffer to Mooncake Store, error code: {ret_code}\"\n            )\n\n\nclass MooncakeStore(HiCacheStorage, MooncakeBaseStore):\n\n    @staticmethod\n    def _standalone_required_bytes(mem_pool: Any) -> int:\n        \"\"\"Compute total bytes of host buffers that must be visible to the real client.\n\n        In standalone (dummy client) mode, the real mooncake_client process needs\n        to map any host buffers we will later pass by pointer via register_buffer().\n        For hybrid models, that includes KV + sidecar pools (e.g. Mamba temporal/conv).\n        \"\"\"\n        # Prefer a generic \"hybrid pool\" accessor when present.\n        total = 0\n        seen_ptrs: set[int] = set()\n","sourceCodeStart":309,"sourceCodeEnd":345,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/srt/mem_cache/storage/mooncake_store/mooncake_store.py#L309-L345","documentation":"MooncakeDistributedStore.register_buffer(ptr, size) returned a non-zero code, so the transfer engine refused to register the host memory region for RDMA/DMA. This is logged then raised as RuntimeError with the code.","triggerScenarios":"Registering a host KV pool buffer whose memory is not pin-able (misaligned, hugepage memory with unsupported size, or already-registered region), or transfer-engine resource limits hit (too many registered regions / MRs).","commonSituations":"Hugepage-backed host pool with unsupported huge page size; exceeding RDMA MR limits on the NIC; memory from a different allocator than expected; standalone_storage=True with a non-MooncakeHostTensorAllocator pool.","solutions":["Check the ret_code against Mooncake transfer engine docs and earlier logger.error line","Pin/export the memory in a supported way (correct allocator, e.g. MooncakeHostTensorAllocator for standalone_storage)","Reduce registered pool size or number of buffers to stay under RDMA MR limits","Upgrade mooncake-transfer-engine"],"exampleFix":null,"handlingStrategy":"try-catch","validationCode":"# pre-allocate/limit pools so total registered bytes stay under RDMA MR budget\nassert pool_bytes <= MR_BUDGET_BYTES, 'host pool exceeds RDMA registration budget'","typeGuard":null,"tryCatchPattern":"try:\n    base.register_buffer(buf)\nexcept RuntimeError as e:\n    if 'Failed to register buffer to Mooncake Store' in str(e):\n        logger.error('registration failed code=%s; check allocator/RDMA limits', e)\n        raise","preventionTips":["Use the allocator required by your storage mode (MooncakeHostTensorAllocator for standalone)","Keep registered region count/size within NIC limits","Keep mooncake-transfer-engine updated"],"tags":["mooncake","rdma","memory-registration","transfer-engine"],"backgroundTag":"memory-registration-failed","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}