vllm-project/vllm · error · ValueError

A speculative model was provided, but `num_speculative_token

Error message

A speculative model was provided, but `num_speculative_tokens` was not provided

What it means

Raised when a speculative/draft model is configured but num_speculative_tokens is None after all defaulting: the user did not pass it and the draft model's hf_config has no n_predict attribute to fall back on. Without this count vLLM cannot size the draft loop or the verification buffers, so it fails fast during SpeculativeConfig.__post_init__.

Source

Thrown at vllm/config/speculative.py:1084

                n_predict = getattr(
                    self.draft_model_config.hf_config, "n_predict", None
                )
                if n_predict is not None:
                    if self.num_speculative_tokens is None:
                        # Default to max value defined in draft model config.
                        self.num_speculative_tokens = n_predict
                    elif (
                        self.num_speculative_tokens > n_predict
                        and self.num_speculative_tokens % n_predict != 0
                    ):
                        # Ensure divisibility for MTP module reuse.
                        raise ValueError(
                            f"num_speculative_tokens:{self.num_speculative_tokens}"
                            f" must be divisible by {n_predict=}"
                        )

                if self.num_speculative_tokens is None:
                    raise ValueError(
                        "A speculative model was provided, but "
                        "`num_speculative_tokens` was not provided"
                    )

                if self.dspark_draft_topk is not None and self.method != "dspark":
                    raise ValueError("dspark_draft_topk is only supported by DSpark")

                dspark_draft_topk = None
                if self.method == "dspark":
                    hf_config = self.draft_model_config.hf_config
                    dspark_draft_topk = self.dspark_draft_topk
                    if dspark_draft_topk is None:
                        dspark_draft_topk = getattr(
                            hf_config, "dspark_draft_topk", None
                        )
                    if dspark_draft_topk is not None:
                        draft_vocab_size = (
                            getattr(hf_config, "draft_vocab_size", None)

View on GitHub (pinned to c794754062)

Solutions

1. Add num_speculative_tokens to the speculative_config (typical values 1-5, e.g. 3 for EAGLE)
2. Use a draft checkpoint whose config declares n_predict so it can be defaulted
3. Pass --num-speculative-tokens N on the serve/serve CLI equivalent

Example fix

# before
speculative_config={"method": "eagle", "model": "yuhuili/EAGLE-LLaMA3.1-Instruct-8B"}
# after
speculative_config={"method": "eagle", "model": "yuhuili/EAGLE-LLaMA3.1-Instruct-8B", "num_speculative_tokens": 3}

if spec_cfg.get("model") and "num_speculative_tokens" not in spec_cfg:
    from transformers import AutoConfig
    if not hasattr(AutoConfig.from_pretrained(spec_cfg["model"]), "n_predict"):
        spec_cfg["num_speculative_tokens"] = 3  # sensible default for eagle-style drafts

def needs_explicit_token_count(spec_cfg: dict) -> bool:
    if "num_speculative_tokens" in spec_cfg:
        return False
    from transformers import AutoConfig
    return not hasattr(AutoConfig.from_pretrained(spec_cfg["model"]), "n_predict")

When it happens

Trigger: speculative_config={'method': 'eagle', 'model': '<draft>'} with no num_speculative_tokens key, where the draft checkpoint config lacks n_predict. Common with plain eagle/eagle3 draft heads and draft_model method.

Common situations: Following an EAGLE quick-start that omits num_speculative_tokens; assuming the draft checkpoint always carries n_predict (only MTP-style heads usually do).

Related errors

• num_speculative_tokens must be provided with speculative mod
• rejection_sample_method='synthetic' requires exactly one of
• synthetic_acceptance_rates must have length {n}, got {rates}
• synthetic_acceptance_rates entries must be in [0, 1], got {r
• synthetic_acceptance_rates must be non-increasing, got {rate

AI-assisted analysis of vllm-project/vllm@c794754062 (2026-08-14). Data as JSON: /api/errors/0ba60a6fb2376a2e. Report an issue: GitHub.