vllm-project/vllm · error · ValueError

KV connector {self.kv_transfer_config.kv_connector} is incom

Error message

KV connector {self.kv_transfer_config.kv_connector} is incompatible with PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True unless enable_cumem_allocator is also enabled. PyTorch's CUDA VMM allocator can remap KV cache virtual addresses to different physical pages, invalidating any pinned/registered KV memory (e.g. IB memory regions registered by NIXL or Mooncake). Either unset expandable_segments:True or enable the cumem allocator (sleep mode does this automatically and also routes KV allocations through CuMemAllocator's pool, where expandable_segments is automatically disabled).

What it means

VllmConfig rejects KV-transfer connector setups when PYTORCH_CUDA_ALLOC_CONF contains expandable_segments:True and enable_cumem_allocator is off. With expandable segments, PyTorch's VMM allocator may remap KV-cache virtual addresses to different physical pages; connectors like NIXL or Mooncake register/pin those addresses as IB memory regions, which become silently invalid after remap. The CuMem allocator is exempt because its memory pool toggles expandable_segments off (see vLLM #40812), so enabling it (or removing the env var) satisfies the check.

Source

Thrown at vllm/config/vllm.py:1013

        # registrations pointing at stale physical pages after any remap,
        # producing RDMA failures like IBV_WC_REM_ACCESS_ERR /
        # NIXL_ERR_REMOTE_DISCONNECT at the first inter-node KV transfer.
        # We can't enumerate every in-tree and out-of-tree connector that
        # pins memory, so we conservatively reject the combination whenever
        # any KV connector is configured.
        #
        # CuMem allocator is exempt: CuMemAllocator.use_memory_pool toggles
        # expandable_segments off around its pool (see #40812), so the KV
        # cache allocated within that context lands on stable physical pages
        # even when the env var is set.
        if "expandable_segments:True" not in os.environ.get(
            "PYTORCH_CUDA_ALLOC_CONF", ""
        ):
            return
        if self.model_config is not None and (self.model_config.enable_cumem_allocator):
            return

        raise ValueError(
            f"KV connector {self.kv_transfer_config.kv_connector} is "
            "incompatible with PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True "
            "unless enable_cumem_allocator is also enabled. PyTorch's CUDA VMM "
            "allocator can remap KV cache virtual addresses to different "
            "physical pages, invalidating any pinned/registered KV memory "
            "(e.g. IB memory regions registered by NIXL or Mooncake). Either "
            "unset expandable_segments:True or enable the cumem allocator "
            "(sleep mode does this automatically and also "
            "routes KV allocations through CuMemAllocator's pool, where "
            "expandable_segments is automatically disabled)."
        )

    def _verify_sampling_replay_config(self) -> None:
        model_config = self.model_config
        if model_config is None or not model_config.return_sampling_mask:
            return
        if not self.use_v2_model_runner:
            raise ValueError("sampling distribution replay requires Model Runner V2")

View on GitHub (pinned to c794754062)

Solutions

Unset the allocator flag: remove expandable_segments:True from PYTORCH_CUDA_ALLOC_CONF (e.g. export PYTORCH_CUDA_ALLOC_CONF='') or drop the variable entirely
Or enable the cumem allocator: --enable-cumem-allocator (model_config.enable_cumem_allocator), which routes KV through CuMemAllocator's pool where expandable_segments is disabled
If using sleep mode, confirm it auto-enables cumem; otherwise set the flag explicitly
Bake the corrected env var into the deployment manifests so pods don't reintroduce it

Example fix

# before
export PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True
vllm serve model --kv-transfer-config '{...NIXL...}'
# after
export PYTORCH_CUDA_ALLOC_CONF=max_split_size_mb:128
vllm serve model --kv-transfer-config '{...NIXL...}' --enable-cumem-allocator

Defensive patterns

Strategy: validation

Validate before calling

import os
expandable = "expandable_segments:True" in os.environ.get("PYTORCH_CUDA_ALLOC_CONF", "")
uses_kv_connector = kv_transfer_config is not None
if expandable and uses_kv_connector:
    assert enable_cumem_allocator, \
        "unset expandable_segments:True or enable cumem allocator"

Type guard

def kv_alloc_safe(kv_cfg, cumem: bool) -> bool:
    import os
    if "expandable_segments:True" not in os.environ.get("PYTORCH_CUDA_ALLOC_CONF", ""):
        return True
    return kv_cfg is None or cumem

Prevention

Audit PYTORCH_CUDA_ALLOC_CONF in base images before enabling KV connectors
Pair NIXL/Mooncake deployments with --enable-cumem-allocator by default
Add a startup preflight script that fails on expandable_segments + kv-transfer without cumem

When it happens

Trigger: Running a P/D-disaggregated or KV-offload server (kv_transfer_config set with a connector like NIXLMetadata/HalfMooncakeTransferEngine) while the environment exports PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True and enable_cumem_allocator is not set on the model config. Commonly bitten by Docker images or cluster profiles that set this allocator flag for fragmentation reasons.

Common situations: Base images (NGC, internal GPU pods) that enable expandable_segments to fight fragmentation; adding KV-disagg (Mooncake/NIXL) to an existing serving stack that already tuned PYTORCH_CUDA_ALLOC_CONF; sleep-mode setups where cumem is auto-enabled (exempt) vs manual startups where it is not.

Related errors

AI-assisted analysis of vllm-project/vllm@c794754062 (2026-08-14). Data as JSON: /api/errors/1a07ca8ec778a7ad. Report an issue: GitHub.