{"record":{"id":"00412b4bc295b93c","repo":"vllm-project/vllm","slug":"sampling-distribution-replay-requires-model-runner","errorCode":null,"errorMessage":"sampling distribution replay requires Model Runner V2","messagePattern":"sampling distribution replay requires Model Runner V2","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"vllm/config/vllm.py","lineNumber":1031,"sourceCode":"        raise ValueError(\n            f\"KV connector {self.kv_transfer_config.kv_connector} is \"\n            \"incompatible with PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True \"\n            \"unless enable_cumem_allocator is also enabled. PyTorch's CUDA VMM \"\n            \"allocator can remap KV cache virtual addresses to different \"\n            \"physical pages, invalidating any pinned/registered KV memory \"\n            \"(e.g. IB memory regions registered by NIXL or Mooncake). Either \"\n            \"unset expandable_segments:True or enable the cumem allocator \"\n            \"(sleep mode does this automatically and also \"\n            \"routes KV allocations through CuMemAllocator's pool, where \"\n            \"expandable_segments is automatically disabled).\"\n        )\n\n    def _verify_sampling_replay_config(self) -> None:\n        model_config = self.model_config\n        if model_config is None or not model_config.return_sampling_mask:\n            return\n        if not self.use_v2_model_runner:\n            raise ValueError(\"sampling distribution replay requires Model Runner V2\")\n        if self.speculative_config is not None:\n            raise ValueError(\n                \"sampling distribution replay does not support speculative decoding\"\n            )\n        if model_config.is_diffusion:\n            raise ValueError(\n                \"sampling distribution replay does not support diffusion models\"\n            )\n        if model_config.logits_processors:\n            raise ValueError(\n                \"sampling distribution replay does not support custom logits processors\"\n            )\n        if model_config.logprobs_mode != \"processed_logprobs\":\n            raise ValueError(\n                \"sampling distribution replay requires \"\n                \"logprobs_mode='processed_logprobs' so that returned logprobs \"\n                \"are normalized over the same nucleus as the sampling mask\"\n            )","sourceCodeStart":1013,"sourceCodeEnd":1049,"githubUrl":"https://github.com/vllm-project/vllm/blob/c794754062d49a8fdb63ab3c5215b488b865030c/vllm/config/vllm.py#L1013-L1049","documentation":"VllmConfig._verify_sampling_replay_config guards the return_sampling_mask feature (replaying the exact sampling distribution, e.g. for distillation/audit pipelines). Returning the sampling mask is only implemented on Model Runner V2, so enabling it while the V1 runner is active fails validation at engine-config time. The whole validator only runs when model_config.return_sampling_mask is truthy.","triggerScenarios":"Setting --return-sampling-mask (model_config.return_sampling_mask=True) without Model Runner V2 enabled — e.g. on a default V1 deployment or an architecture/flag combo that pins V1.","commonSituations":"Building a teacher-forcing / distillation data-collection server and enabling the mask flag on an older vLLM or with V1 forced via internal flags; enabling the flag fleet-wide while only part of the fleet uses V2.","solutions":["Enable Model Runner V2 (V2 runner flag / env such as VLLM_USE_V2_MODEL_RUNNER, or the config field that sets use_v2_model_runner)","Remove/disable return_sampling_mask if you cannot move to V2","Upgrade vLLM to a release where V2 is default for your architecture"],"exampleFix":"# before\nvllm serve model --return-sampling-mask\n# after\nVLLM_USE_V2_MODEL_RUNNER=1 vllm serve model --return-sampling-mask","handlingStrategy":"validation","validationCode":"if model_cfg.get(\"return_sampling_mask\"):\n    assert use_v2_model_runner, \"return_sampling_mask requires Model Runner V2\"","typeGuard":"def sampling_replay_runner_ok(return_mask: bool, use_v2: bool) -> bool:\n    return not return_mask or use_v2","tryCatchPattern":null,"preventionTips":["Gate return_sampling_mask deployments on V2 runner images/env","Feature-detect the runner version before enabling replay features in shared configs"],"tags":["sampling","model-runner","config","validation"],"backgroundTag":null,"analyzedSha":"c794754062d49a8fdb63ab3c5215b488b865030c","analyzedAt":"2026-08-14T21:17:39.825Z","schemaVersion":2},"datasetVersion":"2026-08-15T17:31:12.345Z"}