sgl-project/sglang · critical · RuntimeError

Failed to register buffer to Mooncake Store, error code: {re

Error message

Failed to register buffer to Mooncake Store, error code: {ret_code}

What it means

MooncakeDistributedStore.register_buffer(ptr, size) returned a non-zero code, so the transfer engine refused to register the host memory region for RDMA/DMA. This is logged then raised as RuntimeError with the code.

Source

Thrown at python/sglang/srt/mem_cache/storage/mooncake_store/mooncake_store.py:327

        else:
            config = MooncakeStoreConfig.load_from_env()
            logger.info("Mooncake Configuration loaded from env successfully.")

        return config

    def register_buffer(self, tensor: torch.Tensor):
        if self.store is None:
            raise RuntimeError("Mooncake store is not initialized.")
        ptr = tensor.data_ptr()
        size = tensor.numel() * tensor.element_size()
        if size == 0:
            # conv-only models have a 0-element ssm state; nothing to register
            return
        ret_code = self.store.register_buffer(ptr, size)
        if ret_code != 0:
            logger.error(f"Failed to register buffer, error code: {ret_code}")
            raise RuntimeError(
                f"Failed to register buffer to Mooncake Store, error code: {ret_code}"
            )


class MooncakeStore(HiCacheStorage, MooncakeBaseStore):

    @staticmethod
    def _standalone_required_bytes(mem_pool: Any) -> int:
        """Compute total bytes of host buffers that must be visible to the real client.

        In standalone (dummy client) mode, the real mooncake_client process needs
        to map any host buffers we will later pass by pointer via register_buffer().
        For hybrid models, that includes KV + sidecar pools (e.g. Mamba temporal/conv).
        """
        # Prefer a generic "hybrid pool" accessor when present.
        total = 0
        seen_ptrs: set[int] = set()

View on GitHub (pinned to 0132848349)

Solutions

1. Check the ret_code against Mooncake transfer engine docs and earlier logger.error line
2. Pin/export the memory in a supported way (correct allocator, e.g. MooncakeHostTensorAllocator for standalone_storage)
3. Reduce registered pool size or number of buffers to stay under RDMA MR limits
4. Upgrade mooncake-transfer-engine

# pre-allocate/limit pools so total registered bytes stay under RDMA MR budget
assert pool_bytes <= MR_BUDGET_BYTES, 'host pool exceeds RDMA registration budget'

try:
    base.register_buffer(buf)
except RuntimeError as e:
    if 'Failed to register buffer to Mooncake Store' in str(e):
        logger.error('registration failed code=%s; check allocator/RDMA limits', e)
        raise

When it happens

Trigger: Registering a host KV pool buffer whose memory is not pin-able (misaligned, hugepage memory with unsupported size, or already-registered region), or transfer-engine resource limits hit (too many registered regions / MRs).

Common situations: Hugepage-backed host pool with unsupported huge page size; exceeding RDMA MR limits on the NIC; memory from a different allocator than expected; standalone_storage=True with a non-MooncakeHostTensorAllocator pool.

Related errors

• mooncake encoder_transfer_backend requires HTTP encoders; us
• Mooncake transfer_sync failed for {req_id} (session={session
• [Staging] Bulk RDMA transfer failed with ret={ret}. src_ptr=
• NIXL memory registration failed for {mem_kind} kv tensors
• Mooncake Transfer Engine initialization failed.

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/8c2d5dfa65766ebc. Report an issue: GitHub.