vllm-project/vllm · error · ValueError

sampling distribution replay does not support speculative de

Error message

sampling distribution replay does not support speculative decoding

What it means

When return_sampling_mask is enabled, VllmConfig also rejects speculative decoding configurations. Sampling-distribution replay assumes a single forward's softmax/nucleus distribution; speculative decoding samples from multiple draft/target positions with accept-reject correction, so the returned mask would not correspond to a replayable per-token distribution. Raised when model_config.return_sampling_mask is truthy and speculative_config is not None.

Source

Thrown at vllm/config/vllm.py:1033

            "incompatible with PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True "
            "unless enable_cumem_allocator is also enabled. PyTorch's CUDA VMM "
            "allocator can remap KV cache virtual addresses to different "
            "physical pages, invalidating any pinned/registered KV memory "
            "(e.g. IB memory regions registered by NIXL or Mooncake). Either "
            "unset expandable_segments:True or enable the cumem allocator "
            "(sleep mode does this automatically and also "
            "routes KV allocations through CuMemAllocator's pool, where "
            "expandable_segments is automatically disabled)."
        )

    def _verify_sampling_replay_config(self) -> None:
        model_config = self.model_config
        if model_config is None or not model_config.return_sampling_mask:
            return
        if not self.use_v2_model_runner:
            raise ValueError("sampling distribution replay requires Model Runner V2")
        if self.speculative_config is not None:
            raise ValueError(
                "sampling distribution replay does not support speculative decoding"
            )
        if model_config.is_diffusion:
            raise ValueError(
                "sampling distribution replay does not support diffusion models"
            )
        if model_config.logits_processors:
            raise ValueError(
                "sampling distribution replay does not support custom logits processors"
            )
        if model_config.logprobs_mode != "processed_logprobs":
            raise ValueError(
                "sampling distribution replay requires "
                "logprobs_mode='processed_logprobs' so that returned logprobs "
                "are normalized over the same nucleus as the sampling mask"
            )

    def __post_init__(self):

View on GitHub (pinned to c794754062)

Solutions

1. Remove speculative decoding (drop --speculative-config / --speculative-model) from the instance that returns sampling masks
2. Run two deployments: a speculative one for serving and a non-speculative V2 one for mask collection
3. If you only need logprobs, not the mask, disable return_sampling_mask and use logprobs_mode='processed_logprobs' instead

Example fix

# before
vllm serve model --return-sampling-mask \
  --speculative-config '{"method": "ngram", "num_speculative_tokens": 3}'
# after
vllm serve model --return-sampling-mask

if model_cfg.get("return_sampling_mask"):
    assert speculative_config is None, \
        "sampling replay cannot be combined with speculative decoding"

def replay_spec_ok(return_mask: bool, spec_cfg) -> bool:
    return not return_mask or spec_cfg is None

When it happens

Trigger: Launching a server with both --return-sampling-mask and any speculative config (e.g. --speculative-config '{"method":"ngram",...}' or -–speculative-model).

Common situations: Trying to accelerate a distillation teacher with ngram spec-decode while collecting sampling masks; config templates that layer every speed feature on one instance; enabling speculative decoding on a data-collection endpoint that also sets return_sampling_mask.

Related errors

• use_heterogeneous_vocab currently only supports greedy draft
• sampling distribution replay does not support diffusion mode
• sampling distribution replay does not support custom logits
• rejection_sample_method='synthetic' requires exactly one of
• synthetic_acceptance_rates must have length {n}, got {rates}

AI-assisted analysis of vllm-project/vllm@c794754062 (2026-08-14). Data as JSON: /api/errors/df39bb7a0cc0e92b. Report an issue: GitHub.