{"record":{"id":"74d3fdb0bc7f5759","repo":"vllm-project/vllm","slug":"target-and-draft-model-should-have-the-same-vocabu","errorCode":null,"errorMessage":"Target and draft model should have the same vocabulary size. Target model vocab_size={target_vocab_size}. Draft model vocab_size={draft_vocab_size}. Using models with different tokenizers can cause out-of-bounds errors during speculative decoding.","messagePattern":"Target and draft model should have the same vocabulary size\\. Target model vocab_size=(.+?)\\. Draft model vocab_size=(.+?)\\. Using models with different tokenizers can cause out-of-bounds errors during speculative decoding\\.","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"vllm/config/speculative.py","lineNumber":1412,"sourceCode":"                \"use_heterogeneous_vocab currently only supports greedy draft \"\n                \"sampling. Set draft_sample_method='greedy' (the default) or \"\n                \"omit it.\"\n            )\n\n        if not self.use_heterogeneous_vocab:\n            self.verify_equal_vocab_size_if_draft_model()\n        return self\n\n    def verify_equal_vocab_size_if_draft_model(self):\n        if (\n            self.method == \"draft_model\"\n            and self.target_model_config is not None\n            and self.draft_model_config is not None\n        ):\n            target_vocab_size = self.target_model_config.get_vocab_size()\n            draft_vocab_size = self.draft_model_config.get_vocab_size()\n            if target_vocab_size != draft_vocab_size:\n                raise ValueError(\n                    f\"Target and draft model should have the same vocabulary size. \"\n                    f\"Target model vocab_size={target_vocab_size}. \"\n                    f\"Draft model vocab_size={draft_vocab_size}. \"\n                    f\"Using models with different tokenizers can cause out-of-bounds \"\n                    f\"errors during speculative decoding.\"\n                )\n\n    @property\n    def max_num_new_slots_for_drafting(self) -> int:\n        \"\"\"Return the maximum additional drafting slots per request.\n\n        The scheduler budget already includes one query slot per decoding request.\n        Let K be ``num_speculative_tokens``. Standard configurations require:\n\n        ==================== ============= ======== ================\n        Algorithm            Method        Parallel Additional slots\n        ==================== ============= ======== ================\n        EAGLE3               eagle3        No       0","sourceCodeStart":1394,"sourceCodeEnd":1430,"githubUrl":"https://github.com/vllm-project/vllm/blob/c794754062d49a8fdb63ab3c5215b488b865030c/vllm/config/speculative.py#L1394-L1430","documentation":"For method='draft_model' with both target and draft configs loaded, vLLM compares target_model_config.get_vocab_size() and draft_model_config.get_vocab_size() and refuses to run when they differ. Speculative decoding verifies draft tokens against target logits indexed by token id, so mismatched vocabularies cause out-of-bounds indexing or silently wrong accept/reject decisions. This guard is skipped only when use_heterogeneous_vocab=True, which opts into the cross-vocab code path.","triggerScenarios":"Running vLLM with a speculative draft model whose tokenizer/vocabulary size differs from the target (e.g. --speculative-model pointing at a model fine-tuned with a different tokenizer), method='draft_model', and use_heterogeneous_vocab not enabled (the verify_equal_vocab_size_if_draft_model call runs because use_heterogeneous_vocab is False).","commonSituations":"Using a small draft model (e.g. a different model family) whose tokenizer differs from the target; tokenizer version drift where the draft was saved with added/removed special tokens; trying to pair Llama-family target with a non-Llama draft; forgetting that use_heterogeneous_vocab exists as the supported escape hatch.","solutions":["Pick a draft model that shares the target's tokenizer (same vocab_size), e.g. the official small variant of the target family","Set use_heterogeneous_vocab=True with method='draft_model' and draft_sample_method='greedy' to use the heterogeneous-vocabulary code path","Re-tokenize/save the draft model with the target's tokenizer so both report identical vocab size","Verify sizes up front: tokenizer.vocab_size (plus added tokens) for both checkpoints before launching"],"exampleFix":"# before\nllm = LLM(model=\"meta-llama/Llama-3.1-70B\",\n          speculative_config={\"method\": \"draft_model\", \"model\": \"some-other-draft\"})\n# after\nllm = LLM(model=\"meta-llama/Llama-3.1-70B\",\n          speculative_config={\n              \"method\": \"draft_model\",\n              \"model\": \"meta-llama/Llama-3.2-1B\",\n              \"use_heterogeneous_vocab\": True,\n              \"draft_sample_method\": \"greedy\",\n          })","handlingStrategy":"validation","validationCode":"from transformers import AutoConfig\nt = AutoConfig.from_pretrained(target).vocab_size\nd = AutoConfig.from_pretrained(draft).vocab_size\nif t != d:\n    print(f\"vocab mismatch: target={t} draft={d}; \"\n          \"need use_heterogeneous_vocab or a matching-tokenizer draft\")","typeGuard":"def same_vocab(target_cfg, draft_cfg) -> bool:\n    return target_cfg.get_vocab_size() == draft_cfg.get_vocab_size()","tryCatchPattern":null,"preventionTips":["Compare get_vocab_size() (vocab + added tokens) of both checkpoints before launching speculative serving","Prefer drafts from the same model family/tokenizer lineage","Wrap engine startup in try/except ValueError and print the message — it names both sizes for quick diagnosis"],"tags":["speculative-decoding","tokenizer","config","validation"],"backgroundTag":null,"analyzedSha":"c794754062d49a8fdb63ab3c5215b488b865030c","analyzedAt":"2026-08-14T21:17:39.825Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}