vllm-project/vllm · error · ValueError

Target and draft model should have the same vocabulary size.

Error message

Target and draft model should have the same vocabulary size. Target model vocab_size={target_vocab_size}. Draft model vocab_size={draft_vocab_size}. Using models with different tokenizers can cause out-of-bounds errors during speculative decoding.

What it means

For method='draft_model' with both target and draft configs loaded, vLLM compares target_model_config.get_vocab_size() and draft_model_config.get_vocab_size() and refuses to run when they differ. Speculative decoding verifies draft tokens against target logits indexed by token id, so mismatched vocabularies cause out-of-bounds indexing or silently wrong accept/reject decisions. This guard is skipped only when use_heterogeneous_vocab=True, which opts into the cross-vocab code path.

Source

Thrown at vllm/config/speculative.py:1412

                "use_heterogeneous_vocab currently only supports greedy draft "
                "sampling. Set draft_sample_method='greedy' (the default) or "
                "omit it."
            )

        if not self.use_heterogeneous_vocab:
            self.verify_equal_vocab_size_if_draft_model()
        return self

    def verify_equal_vocab_size_if_draft_model(self):
        if (
            self.method == "draft_model"
            and self.target_model_config is not None
            and self.draft_model_config is not None
        ):
            target_vocab_size = self.target_model_config.get_vocab_size()
            draft_vocab_size = self.draft_model_config.get_vocab_size()
            if target_vocab_size != draft_vocab_size:
                raise ValueError(
                    f"Target and draft model should have the same vocabulary size. "
                    f"Target model vocab_size={target_vocab_size}. "
                    f"Draft model vocab_size={draft_vocab_size}. "
                    f"Using models with different tokenizers can cause out-of-bounds "
                    f"errors during speculative decoding."
                )

    @property
    def max_num_new_slots_for_drafting(self) -> int:
        """Return the maximum additional drafting slots per request.

        The scheduler budget already includes one query slot per decoding request.
        Let K be ``num_speculative_tokens``. Standard configurations require:

        ==================== ============= ======== ================
        Algorithm            Method        Parallel Additional slots
        ==================== ============= ======== ================
        EAGLE3               eagle3        No       0

View on GitHub (pinned to c794754062)

Solutions

1. Pick a draft model that shares the target's tokenizer (same vocab_size), e.g. the official small variant of the target family
2. Set use_heterogeneous_vocab=True with method='draft_model' and draft_sample_method='greedy' to use the heterogeneous-vocabulary code path
3. Re-tokenize/save the draft model with the target's tokenizer so both report identical vocab size
4. Verify sizes up front: tokenizer.vocab_size (plus added tokens) for both checkpoints before launching

Example fix

# before
llm = LLM(model="meta-llama/Llama-3.1-70B",
          speculative_config={"method": "draft_model", "model": "some-other-draft"})
# after
llm = LLM(model="meta-llama/Llama-3.1-70B",
          speculative_config={
              "method": "draft_model",
              "model": "meta-llama/Llama-3.2-1B",
              "use_heterogeneous_vocab": True,
              "draft_sample_method": "greedy",
          })

from transformers import AutoConfig
t = AutoConfig.from_pretrained(target).vocab_size
d = AutoConfig.from_pretrained(draft).vocab_size
if t != d:
    print(f"vocab mismatch: target={t} draft={d}; "
          "need use_heterogeneous_vocab or a matching-tokenizer draft")

def same_vocab(target_cfg, draft_cfg) -> bool:
    return target_cfg.get_vocab_size() == draft_cfg.get_vocab_size()

When it happens

Trigger: Running vLLM with a speculative draft model whose tokenizer/vocabulary size differs from the target (e.g. --speculative-model pointing at a model fine-tuned with a different tokenizer), method='draft_model', and use_heterogeneous_vocab not enabled (the verify_equal_vocab_size_if_draft_model call runs because use_heterogeneous_vocab is False).

Common situations: Using a small draft model (e.g. a different model family) whose tokenizer differs from the target; tokenizer version drift where the draft was saved with added/removed special tokens; trying to pair Llama-family target with a non-Llama draft; forgetting that use_heterogeneous_vocab exists as the supported escape hatch.

Related errors

• use_heterogeneous_vocab only works with method='draft_model'
• ReasoningConfig: failed to tokenize reasoning strings: reaso
• rejection_sample_method='synthetic' requires exactly one of
• synthetic_acceptance_rates must have length {n}, got {rates}
• synthetic_acceptance_rates entries must be in [0, 1], got {r

AI-assisted analysis of vllm-project/vllm@c794754062 (2026-08-14). Data as JSON: /api/errors/74d3fdb0bc7f5759. Report an issue: GitHub.