{"record":{"id":"df39bb7a0cc0e92b","repo":"vllm-project/vllm","slug":"sampling-distribution-replay-does-not-support-spec","errorCode":null,"errorMessage":"sampling distribution replay does not support speculative decoding","messagePattern":"sampling distribution replay does not support speculative decoding","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"vllm/config/vllm.py","lineNumber":1033,"sourceCode":"            \"incompatible with PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True \"\n            \"unless enable_cumem_allocator is also enabled. PyTorch's CUDA VMM \"\n            \"allocator can remap KV cache virtual addresses to different \"\n            \"physical pages, invalidating any pinned/registered KV memory \"\n            \"(e.g. IB memory regions registered by NIXL or Mooncake). Either \"\n            \"unset expandable_segments:True or enable the cumem allocator \"\n            \"(sleep mode does this automatically and also \"\n            \"routes KV allocations through CuMemAllocator's pool, where \"\n            \"expandable_segments is automatically disabled).\"\n        )\n\n    def _verify_sampling_replay_config(self) -> None:\n        model_config = self.model_config\n        if model_config is None or not model_config.return_sampling_mask:\n            return\n        if not self.use_v2_model_runner:\n            raise ValueError(\"sampling distribution replay requires Model Runner V2\")\n        if self.speculative_config is not None:\n            raise ValueError(\n                \"sampling distribution replay does not support speculative decoding\"\n            )\n        if model_config.is_diffusion:\n            raise ValueError(\n                \"sampling distribution replay does not support diffusion models\"\n            )\n        if model_config.logits_processors:\n            raise ValueError(\n                \"sampling distribution replay does not support custom logits processors\"\n            )\n        if model_config.logprobs_mode != \"processed_logprobs\":\n            raise ValueError(\n                \"sampling distribution replay requires \"\n                \"logprobs_mode='processed_logprobs' so that returned logprobs \"\n                \"are normalized over the same nucleus as the sampling mask\"\n            )\n\n    def __post_init__(self):","sourceCodeStart":1015,"sourceCodeEnd":1051,"githubUrl":"https://github.com/vllm-project/vllm/blob/c794754062d49a8fdb63ab3c5215b488b865030c/vllm/config/vllm.py#L1015-L1051","documentation":"When return_sampling_mask is enabled, VllmConfig also rejects speculative decoding configurations. Sampling-distribution replay assumes a single forward's softmax/nucleus distribution; speculative decoding samples from multiple draft/target positions with accept-reject correction, so the returned mask would not correspond to a replayable per-token distribution. Raised when model_config.return_sampling_mask is truthy and speculative_config is not None.","triggerScenarios":"Launching a server with both --return-sampling-mask and any speculative config (e.g. --speculative-config '{\"method\":\"ngram\",...}' or -–speculative-model).","commonSituations":"Trying to accelerate a distillation teacher with ngram spec-decode while collecting sampling masks; config templates that layer every speed feature on one instance; enabling speculative decoding on a data-collection endpoint that also sets return_sampling_mask.","solutions":["Remove speculative decoding (drop --speculative-config / --speculative-model) from the instance that returns sampling masks","Run two deployments: a speculative one for serving and a non-speculative V2 one for mask collection","If you only need logprobs, not the mask, disable return_sampling_mask and use logprobs_mode='processed_logprobs' instead"],"exampleFix":"# before\nvllm serve model --return-sampling-mask \\\n  --speculative-config '{\"method\": \"ngram\", \"num_speculative_tokens\": 3}'\n# after\nvllm serve model --return-sampling-mask","handlingStrategy":"validation","validationCode":"if model_cfg.get(\"return_sampling_mask\"):\n    assert speculative_config is None, \\\n        \"sampling replay cannot be combined with speculative decoding\"","typeGuard":"def replay_spec_ok(return_mask: bool, spec_cfg) -> bool:\n    return not return_mask or spec_cfg is None","tryCatchPattern":null,"preventionTips":["Keep data-collection (mask-returning) instances free of speculative decoding","Split serving and collection into separate deployment profiles"],"tags":["sampling","speculative-decoding","config","incompatibility"],"backgroundTag":null,"analyzedSha":"c794754062d49a8fdb63ab3c5215b488b865030c","analyzedAt":"2026-08-14T21:17:39.825Z","schemaVersion":2},"datasetVersion":"2026-08-15T17:31:12.345Z"}