vllm-project/vllm · error · ValueError

use_heterogeneous_vocab currently only supports greedy draft

Error message

use_heterogeneous_vocab currently only supports greedy draft sampling. Set draft_sample_method='greedy' (the default) or omit it.

What it means

When use_heterogeneous_vocab=True, vLLM requires draft_sample_method='greedy'. Heterogeneous-vocab speculative decoding aligns draft and target distributions across different vocabularies, and that alignment is currently only implemented for greedy (argmax) draft sampling. Stochastic draft sampling would require translating probability distributions between vocabularies, which is not supported yet.

Source

Thrown at vllm/config/speculative.py:1393

            or self.synthetic_acceptance_length is not None
        ):
            raise ValueError(
                "synthetic_acceptance_rates / synthetic_acceptance_length "
                "are only valid with rejection_sample_method='synthetic'."
            )

        if self.draft_model_config:
            self.draft_model_config.verify_with_parallel_config(
                self.draft_parallel_config
            )

        if self.use_heterogeneous_vocab and not self.uses_draft_model():
            raise ValueError(
                "use_heterogeneous_vocab only works with method='draft_model'"
            )

        if self.use_heterogeneous_vocab and self.draft_sample_method != "greedy":
            raise ValueError(
                "use_heterogeneous_vocab currently only supports greedy draft "
                "sampling. Set draft_sample_method='greedy' (the default) or "
                "omit it."
            )

        if not self.use_heterogeneous_vocab:
            self.verify_equal_vocab_size_if_draft_model()
        return self

    def verify_equal_vocab_size_if_draft_model(self):
        if (
            self.method == "draft_model"
            and self.target_model_config is not None
            and self.draft_model_config is not None
        ):
            target_vocab_size = self.target_model_config.get_vocab_size()
            draft_vocab_size = self.draft_model_config.get_vocab_size()
            if target_vocab_size != draft_vocab_size:

View on GitHub (pinned to c794754062)

Solutions

1. Remove draft_sample_method from the config (greedy is the default) so the check passes
2. Explicitly set draft_sample_method='greedy' alongside use_heterogeneous_vocab=True
3. If you need stochastic draft sampling, disable use_heterogeneous_vocab and use models with matching vocabularies

Example fix

# before
speculative_config = {
    "method": "draft_model",
    "model": "draft",
    "use_heterogeneous_vocab": True,
    "draft_sample_method": "sample",
}
# after
speculative_config = {
    "method": "draft_model",
    "model": "draft",
    "use_heterogeneous_vocab": True,
    "draft_sample_method": "greedy",
}

spec = {"method": "draft_model", "use_heterogeneous_vocab": True,
        "draft_sample_method": "sample"}
if spec.get("use_heterogeneous_vocab"):
    assert spec.get("draft_sample_method", "greedy") == "greedy", \
        "heterogeneous vocab requires greedy draft sampling"

def valid_draft_sampling(spec: dict) -> bool:
    return spec.get("draft_sample_method", "greedy") == "greedy"

When it happens

Trigger: Setting use_heterogeneous_vocab=True together with draft_sample_method set to anything other than 'greedy' (the default), e.g. draft_sample_method='sample' or a temperature-based sampler, in the speculative config.

Common situations: Enabling randomized draft sampling to improve acceptance diversity and then turning on heterogeneous vocab; carrying over draft_sample_method from a same-vocab setup where sampling worked; misreading the default and explicitly setting a non-greedy value.

Related errors

• rejection_sample_method='synthetic' requires exactly one of
• synthetic_acceptance_rates must have length {n}, got {rates}
• synthetic_acceptance_rates entries must be in [0, 1], got {r
• synthetic_acceptance_rates must be non-increasing, got {rate
• synthetic_acceptance_length must be in [1, {n + 1}], got {le

AI-assisted analysis of vllm-project/vllm@c794754062 (2026-08-14). Data as JSON: /api/errors/ceb5bac57036b185. Report an issue: GitHub.