vllm-project/vllm · critical · RuntimeError

Mooncake batch memory registration failed.

Error message

Mooncake batch memory registration failed.

What it means

RuntimeError raised when TransferEngine.batch_register_memory(kv_data_ptrs, kv_data_lens) returns nonzero after the KV cache storages were collected. The native engine failed to pin/register one or more memory regions for RDMA access — commonly permission issues, unsupported memory (CPU tensors when the device requires GPU memory, or CUDA unified memory), exceeding RDMA registration limits, or a device/driver mismatch with the allocated buffers.

Source

Thrown at vllm/distributed/kv_transfer/kv_connector/v1/mooncake/mooncake_connector.py:1727

                self.registered_group_indices.append(
                    self._layer_group_indices[layer_name]
                )
                storage = cache.untyped_storage()
                storage_addr = storage.data_ptr()
                if storage_addr not in seen_storage_ptrs:
                    seen_storage_ptrs.add(storage_addr)
                    kv_data_ptrs.append(storage_addr)
                    kv_data_lens.append(storage.nbytes())

        self.kv_caches_base_addr = region_base_addresses
        self.seen_base_addresses = kv_data_ptrs

        if not kv_data_ptrs:
            raise RuntimeError("No KV cache tensors were registered with Mooncake.")

        ret_value = self.engine.batch_register_memory(kv_data_ptrs, kv_data_lens)
        if ret_value != 0:
            raise RuntimeError("Mooncake batch memory registration failed.")

        self.device_kv_caches = kv_caches
        logger.debug(
            "registered block_lens=%s kv_block_lens=%s",
            self.block_len_per_layer,
            self.kv_block_len_per_layer,
        )

        # No need to launch server for D node.
        if self.is_kv_consumer:
            return

        ready_event = threading.Event()
        asyncio.run_coroutine_threadsafe(
            self._mooncake_sender_listener(ready_event), self.sender_loop
        )
        ready_event.wait()  # Wait for listener ZMQ socket to be ready.

View on GitHub (pinned to c794754062)

Solutions

1. Confirm the KV cache tensors are on the device the engine expects (GPU memory for GPU transfers) and the correct CUDA device is selected.
2. Check system RDMA limits (e.g. max locked memory ulimit -l, module parameters for memory registration) and raise them or shrink the KV cache.
3. Retry after verifying device health (nvidia-smi, ibv_devinfo) — a reset device can fail registrations.
4. If constraints can't be lifted, reduce gpu-memory-utilization / KV cache size so fewer bytes need registration.

total_bytes = sum(t.untyped_storage().nbytes() for layer in kv_caches.values() for t in layer)
import resource
locked_limit = resource.getrlimit(resource.RLIMIT_MEMLOCK)[0]
# warn if total_bytes approaches locked_limit before registration

Catch RuntimeError after batch_register_memory; verify device health (nvidia-smi/ibv_devinfo), check/raise memlock ulimit, optionally shrink the KV cache, then retry registration once.

When it happens

Trigger: Registering CPU-pinned or non-CUDA host memory while the engine was initialized for a GPU device; ulimit or RDMA max memory registration limits exceeded with very large KV caches; IOMMU/vGPU setups blocking DMA mappings; device lost or reset between engine init and registration.

Common situations: Very large KV cache (hundreds of GB) hitting system RDMA registration limits; running in virtualized environments (vGPU, SR-IOV constraints) where registration is restricted; mixing CUDA_VISIBLE_DEVICES changes after engine initialization.

Related errors

• JSON error: {0}
• Mooncake is not available
• Mooncake Transfer Engine initialization failed.
• Tokenizer error: {0}
• No KV cache tensors were registered with Mooncake.

AI-assisted analysis of vllm-project/vllm@c794754062 (2026-08-14). Data as JSON: /api/errors/7789835df4e21afc. Report an issue: GitHub.