{"record":{"id":"7789835df4e21afc","repo":"vllm-project/vllm","slug":"mooncake-batch-memory-registration-failed","errorCode":null,"errorMessage":"Mooncake batch memory registration failed.","messagePattern":"Mooncake batch memory registration failed\\.","errorType":"exception","errorClass":"RuntimeError","httpStatus":null,"severity":"critical","filePath":"vllm/distributed/kv_transfer/kv_connector/v1/mooncake/mooncake_connector.py","lineNumber":1727,"sourceCode":"                self.registered_group_indices.append(\n                    self._layer_group_indices[layer_name]\n                )\n                storage = cache.untyped_storage()\n                storage_addr = storage.data_ptr()\n                if storage_addr not in seen_storage_ptrs:\n                    seen_storage_ptrs.add(storage_addr)\n                    kv_data_ptrs.append(storage_addr)\n                    kv_data_lens.append(storage.nbytes())\n\n        self.kv_caches_base_addr = region_base_addresses\n        self.seen_base_addresses = kv_data_ptrs\n\n        if not kv_data_ptrs:\n            raise RuntimeError(\"No KV cache tensors were registered with Mooncake.\")\n\n        ret_value = self.engine.batch_register_memory(kv_data_ptrs, kv_data_lens)\n        if ret_value != 0:\n            raise RuntimeError(\"Mooncake batch memory registration failed.\")\n\n        self.device_kv_caches = kv_caches\n        logger.debug(\n            \"registered block_lens=%s kv_block_lens=%s\",\n            self.block_len_per_layer,\n            self.kv_block_len_per_layer,\n        )\n\n        # No need to launch server for D node.\n        if self.is_kv_consumer:\n            return\n\n        ready_event = threading.Event()\n        asyncio.run_coroutine_threadsafe(\n            self._mooncake_sender_listener(ready_event), self.sender_loop\n        )\n        ready_event.wait()  # Wait for listener ZMQ socket to be ready.\n","sourceCodeStart":1709,"sourceCodeEnd":1745,"githubUrl":"https://github.com/vllm-project/vllm/blob/c794754062d49a8fdb63ab3c5215b488b865030c/vllm/distributed/kv_transfer/kv_connector/v1/mooncake/mooncake_connector.py#L1709-L1745","documentation":"RuntimeError raised when TransferEngine.batch_register_memory(kv_data_ptrs, kv_data_lens) returns nonzero after the KV cache storages were collected. The native engine failed to pin/register one or more memory regions for RDMA access — commonly permission issues, unsupported memory (CPU tensors when the device requires GPU memory, or CUDA unified memory), exceeding RDMA registration limits, or a device/driver mismatch with the allocated buffers.","triggerScenarios":"Registering CPU-pinned or non-CUDA host memory while the engine was initialized for a GPU device; ulimit or RDMA max memory registration limits exceeded with very large KV caches; IOMMU/vGPU setups blocking DMA mappings; device lost or reset between engine init and registration.","commonSituations":"Very large KV cache (hundreds of GB) hitting system RDMA registration limits; running in virtualized environments (vGPU, SR-IOV constraints) where registration is restricted; mixing CUDA_VISIBLE_DEVICES changes after engine initialization.","solutions":["Confirm the KV cache tensors are on the device the engine expects (GPU memory for GPU transfers) and the correct CUDA device is selected.","Check system RDMA limits (e.g. max locked memory ulimit -l, module parameters for memory registration) and raise them or shrink the KV cache.","Retry after verifying device health (nvidia-smi, ibv_devinfo) — a reset device can fail registrations.","If constraints can't be lifted, reduce gpu-memory-utilization / KV cache size so fewer bytes need registration."],"exampleFix":null,"handlingStrategy":"retry","validationCode":"total_bytes = sum(t.untyped_storage().nbytes() for layer in kv_caches.values() for t in layer)\nimport resource\nlocked_limit = resource.getrlimit(resource.RLIMIT_MEMLOCK)[0]\n# warn if total_bytes approaches locked_limit before registration","typeGuard":null,"tryCatchPattern":"Catch RuntimeError after batch_register_memory; verify device health (nvidia-smi/ibv_devinfo), check/raise memlock ulimit, optionally shrink the KV cache, then retry registration once.","preventionTips":["Raise memlock limits (ulimit -l unlimited) on RDMA hosts","Confirm tensors live on the expected CUDA device before registration","Size the KV cache with RDMA registration limits in mind"],"tags":["mooncake","rdma","memory-registration","gpu","kv-transfer"],"backgroundTag":null,"analyzedSha":"c794754062d49a8fdb63ab3c5215b488b865030c","analyzedAt":"2026-08-14T21:17:39.825Z","schemaVersion":2},"datasetVersion":"2026-08-15T17:31:12.345Z"}