{"record":{"id":"1a07ca8ec778a7ad","repo":"vllm-project/vllm","slug":"kv-connector-self-kv-transfer-config-kv-connector","errorCode":null,"errorMessage":"KV connector {self.kv_transfer_config.kv_connector} is incompatible with PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True unless enable_cumem_allocator is also enabled. PyTorch's CUDA VMM allocator can remap KV cache virtual addresses to different physical pages, invalidating any pinned/registered KV memory (e.g. IB memory regions registered by NIXL or Mooncake). Either unset expandable_segments:True or enable the cumem allocator (sleep mode does this automatically and also routes KV allocations through CuMemAllocator's pool, where expandable_segments is automatically disabled).","messagePattern":"KV connector \\{self\\.kv_transfer_config\\.kv_connector\\} is incompatible with PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True unless enable_cumem_allocator is also enabled\\. PyTorch's CUDA VMM allocator can remap KV cache virtual addresses to different physical pages, invalidating any pinned/registered KV memory \\(e\\.g\\. IB memory regions registered by NIXL or Mooncake\\)\\. Either unset expandable_segments:True or enable the cumem allocator \\(sleep mode does this automatically and also routes KV allocations through CuMemAllocator's pool, where expandable_segments is automatically disabled\\)\\.","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"vllm/config/vllm.py","lineNumber":1013,"sourceCode":"        # registrations pointing at stale physical pages after any remap,\n        # producing RDMA failures like IBV_WC_REM_ACCESS_ERR /\n        # NIXL_ERR_REMOTE_DISCONNECT at the first inter-node KV transfer.\n        # We can't enumerate every in-tree and out-of-tree connector that\n        # pins memory, so we conservatively reject the combination whenever\n        # any KV connector is configured.\n        #\n        # CuMem allocator is exempt: CuMemAllocator.use_memory_pool toggles\n        # expandable_segments off around its pool (see #40812), so the KV\n        # cache allocated within that context lands on stable physical pages\n        # even when the env var is set.\n        if \"expandable_segments:True\" not in os.environ.get(\n            \"PYTORCH_CUDA_ALLOC_CONF\", \"\"\n        ):\n            return\n        if self.model_config is not None and (self.model_config.enable_cumem_allocator):\n            return\n\n        raise ValueError(\n            f\"KV connector {self.kv_transfer_config.kv_connector} is \"\n            \"incompatible with PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True \"\n            \"unless enable_cumem_allocator is also enabled. PyTorch's CUDA VMM \"\n            \"allocator can remap KV cache virtual addresses to different \"\n            \"physical pages, invalidating any pinned/registered KV memory \"\n            \"(e.g. IB memory regions registered by NIXL or Mooncake). Either \"\n            \"unset expandable_segments:True or enable the cumem allocator \"\n            \"(sleep mode does this automatically and also \"\n            \"routes KV allocations through CuMemAllocator's pool, where \"\n            \"expandable_segments is automatically disabled).\"\n        )\n\n    def _verify_sampling_replay_config(self) -> None:\n        model_config = self.model_config\n        if model_config is None or not model_config.return_sampling_mask:\n            return\n        if not self.use_v2_model_runner:\n            raise ValueError(\"sampling distribution replay requires Model Runner V2\")","sourceCodeStart":995,"sourceCodeEnd":1031,"githubUrl":"https://github.com/vllm-project/vllm/blob/c794754062d49a8fdb63ab3c5215b488b865030c/vllm/config/vllm.py#L995-L1031","documentation":"VllmConfig rejects KV-transfer connector setups when PYTORCH_CUDA_ALLOC_CONF contains expandable_segments:True and enable_cumem_allocator is off. With expandable segments, PyTorch's VMM allocator may remap KV-cache virtual addresses to different physical pages; connectors like NIXL or Mooncake register/pin those addresses as IB memory regions, which become silently invalid after remap. The CuMem allocator is exempt because its memory pool toggles expandable_segments off (see vLLM #40812), so enabling it (or removing the env var) satisfies the check.","triggerScenarios":"Running a P/D-disaggregated or KV-offload server (kv_transfer_config set with a connector like NIXLMetadata/HalfMooncakeTransferEngine) while the environment exports PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True and enable_cumem_allocator is not set on the model config. Commonly bitten by Docker images or cluster profiles that set this allocator flag for fragmentation reasons.","commonSituations":"Base images (NGC, internal GPU pods) that enable expandable_segments to fight fragmentation; adding KV-disagg (Mooncake/NIXL) to an existing serving stack that already tuned PYTORCH_CUDA_ALLOC_CONF; sleep-mode setups where cumem is auto-enabled (exempt) vs manual startups where it is not.","solutions":["Unset the allocator flag: remove expandable_segments:True from PYTORCH_CUDA_ALLOC_CONF (e.g. export PYTORCH_CUDA_ALLOC_CONF='') or drop the variable entirely","Or enable the cumem allocator: --enable-cumem-allocator (model_config.enable_cumem_allocator), which routes KV through CuMemAllocator's pool where expandable_segments is disabled","If using sleep mode, confirm it auto-enables cumem; otherwise set the flag explicitly","Bake the corrected env var into the deployment manifests so pods don't reintroduce it"],"exampleFix":"# before\nexport PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True\nvllm serve model --kv-transfer-config '{...NIXL...}'\n# after\nexport PYTORCH_CUDA_ALLOC_CONF=max_split_size_mb:128\nvllm serve model --kv-transfer-config '{...NIXL...}' --enable-cumem-allocator","handlingStrategy":"validation","validationCode":"import os\nexpandable = \"expandable_segments:True\" in os.environ.get(\"PYTORCH_CUDA_ALLOC_CONF\", \"\")\nuses_kv_connector = kv_transfer_config is not None\nif expandable and uses_kv_connector:\n    assert enable_cumem_allocator, \\\n        \"unset expandable_segments:True or enable cumem allocator\"","typeGuard":"def kv_alloc_safe(kv_cfg, cumem: bool) -> bool:\n    import os\n    if \"expandable_segments:True\" not in os.environ.get(\"PYTORCH_CUDA_ALLOC_CONF\", \"\"):\n        return True\n    return kv_cfg is None or cumem","tryCatchPattern":null,"preventionTips":["Audit PYTORCH_CUDA_ALLOC_CONF in base images before enabling KV connectors","Pair NIXL/Mooncake deployments with --enable-cumem-allocator by default","Add a startup preflight script that fails on expandable_segments + kv-transfer without cumem"],"tags":["kv-transfer","allocator","environment","disaggregation"],"backgroundTag":null,"analyzedSha":"c794754062d49a8fdb63ab3c5215b488b865030c","analyzedAt":"2026-08-14T21:17:39.825Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}