vllm-project/vllm · error · ValueError

sampling distribution replay requires Model Runner V2

Error message

sampling distribution replay requires Model Runner V2

What it means

VllmConfig._verify_sampling_replay_config guards the return_sampling_mask feature (replaying the exact sampling distribution, e.g. for distillation/audit pipelines). Returning the sampling mask is only implemented on Model Runner V2, so enabling it while the V1 runner is active fails validation at engine-config time. The whole validator only runs when model_config.return_sampling_mask is truthy.

Source

Thrown at vllm/config/vllm.py:1031

        raise ValueError(
            f"KV connector {self.kv_transfer_config.kv_connector} is "
            "incompatible with PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True "
            "unless enable_cumem_allocator is also enabled. PyTorch's CUDA VMM "
            "allocator can remap KV cache virtual addresses to different "
            "physical pages, invalidating any pinned/registered KV memory "
            "(e.g. IB memory regions registered by NIXL or Mooncake). Either "
            "unset expandable_segments:True or enable the cumem allocator "
            "(sleep mode does this automatically and also "
            "routes KV allocations through CuMemAllocator's pool, where "
            "expandable_segments is automatically disabled)."
        )

    def _verify_sampling_replay_config(self) -> None:
        model_config = self.model_config
        if model_config is None or not model_config.return_sampling_mask:
            return
        if not self.use_v2_model_runner:
            raise ValueError("sampling distribution replay requires Model Runner V2")
        if self.speculative_config is not None:
            raise ValueError(
                "sampling distribution replay does not support speculative decoding"
            )
        if model_config.is_diffusion:
            raise ValueError(
                "sampling distribution replay does not support diffusion models"
            )
        if model_config.logits_processors:
            raise ValueError(
                "sampling distribution replay does not support custom logits processors"
            )
        if model_config.logprobs_mode != "processed_logprobs":
            raise ValueError(
                "sampling distribution replay requires "
                "logprobs_mode='processed_logprobs' so that returned logprobs "
                "are normalized over the same nucleus as the sampling mask"
            )

View on GitHub (pinned to c794754062)

Solutions

Enable Model Runner V2 (V2 runner flag / env such as VLLM_USE_V2_MODEL_RUNNER, or the config field that sets use_v2_model_runner)
Remove/disable return_sampling_mask if you cannot move to V2
Upgrade vLLM to a release where V2 is default for your architecture

Example fix

# before
vllm serve model --return-sampling-mask
# after
VLLM_USE_V2_MODEL_RUNNER=1 vllm serve model --return-sampling-mask

Defensive patterns

Strategy: validation

Validate before calling

if model_cfg.get("return_sampling_mask"):
    assert use_v2_model_runner, "return_sampling_mask requires Model Runner V2"

Type guard

def sampling_replay_runner_ok(return_mask: bool, use_v2: bool) -> bool:
    return not return_mask or use_v2

Prevention

Gate return_sampling_mask deployments on V2 runner images/env
Feature-detect the runner version before enabling replay features in shared configs

When it happens

Trigger: Setting --return-sampling-mask (model_config.return_sampling_mask=True) without Model Runner V2 enabled — e.g. on a default V1 deployment or an architecture/flag combo that pins V1.

Common situations: Building a teacher-forcing / distillation data-collection server and enabling the mask flag on an older vLLM or with V1 forced via internal flags; enabling the flag fleet-wide while only part of the fleet uses V2.

Related errors

AI-assisted analysis of vllm-project/vllm@c794754062 (2026-08-14). Data as JSON: /api/errors/00412b4bc295b93c. Report an issue: GitHub.