sgl-project/sglang · critical · ValueError

STANDALONE speculative decoding requires the draft model to

Error message

STANDALONE speculative decoding requires the draft model to share the same vocabulary as the target model, but got target vocab_size={target_vocab_size} and draft vocab_size={draft_vocab_size}. Use a draft model with a matching vocabulary, or a speculative algorithm that supports heterogeneous vocabularies.

What it means

STANDALONE speculative decoding (separate draft model) requires draft and target to share vocab size; _validate_vocab_compatibility raises ValueError when target vocab_size != draft vocab_size, since logits/argmax comparison across mismatched vocabularies is invalid.

Source

Thrown at python/sglang/srt/speculative/standalone_worker_v2.py:211

            (), dtype=torch.int64, device=self.device
        )
        self.extend_lens = torch.empty((), dtype=torch.int64, device=self.device)

        self.plan_stream, self.plan_stream_ctx = get_plan_stream(self.device)

        # TODO: Adaptive speculative
        self.adaptive_controller: Optional[AdaptiveController] = None

    def _validate_vocab_compatibility(
        self,
        target_vocab_size: int,
        target_tokenizer,
    ) -> None:
        """Raise ValueError if the draft and target vocabularies are incompatible."""
        draft_vocab_size = self._draft_worker.draft_runner.model_config.vocab_size
        draft_tokenizer = self._draft_worker.draft_worker.tokenizer
        if target_vocab_size != draft_vocab_size:
            raise ValueError(
                f"STANDALONE speculative decoding requires the draft model to share the "
                f"same vocabulary as the target model, but got "
                f"target vocab_size={target_vocab_size} and "
                f"draft vocab_size={draft_vocab_size}. "
                f"Use a draft model with a matching vocabulary, or a speculative "
                f"algorithm that supports heterogeneous vocabularies."
            )
        if (
            target_tokenizer is not None
            and draft_tokenizer is not None
            and hasattr(target_tokenizer, "get_vocab")
            and hasattr(draft_tokenizer, "get_vocab")
            and target_tokenizer.get_vocab() != draft_tokenizer.get_vocab()
        ):
            raise ValueError(
                "STANDALONE speculative decoding requires the draft model to share the "
                "same vocabulary as the target model, but the two tokenizers have "
                "different token-to-id mappings even though their vocab sizes match. "

View on GitHub (pinned to 0132848349)

Solutions

1. Use a draft model with identical vocab size/tokenizer family as the target (e.g. same-family small model)
2. Verify both configs: compare target.config.vocab_size vs draft config vocab_size before launch
3. If vocabularies truly differ, switch to an algorithm supporting heterogeneous vocabularies (e.g. EAGLE-style with adapter) or drop spec decoding

Example fix

# before
--speculative-algorithm STANDALONE --speculative-draft-model-path tiny_random bert
# after
--speculative-algorithm STANDALONE --speculative-draft-model-path <same-family small model>

from transformers import AutoConfig
t = AutoConfig.from_pretrained(target_path).vocab_size
d = AutoConfig.from_pretrained(draft_path).vocab_size
assert t == d, f'vocab mismatch: {t} vs {d}'

When it happens

Trigger: Launching with --speculative-algorithm STANDALONE (V2) where the draft model's config vocab_size differs from the target's, e.g. pairing an Llama-68M draft with a non-Llama target.

Common situations: Choosing a mismatched draft model (different tokenizer family); off-by-config wrong draft path; draft config with added special tokens changing vocab_size.

Related errors

• STANDALONE speculative decoding requires the draft model to
• This browser cannot encode H.264 MP4
• Helion KDA decode requires power-of-two key and value head d
• `A_log` must have {HV} elements (got {A_log.numel()}).
• `dt_bias` must have {HV * K} elements (got {dt_bias.numel()}

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/99f0a919c2376bf6. Report an issue: GitHub.