vllm-project/vllm · error · ValueError

num_speculative_tokens must be provided with speculative mod

Error message

num_speculative_tokens must be provided with speculative model unless the draft model config contains an n_predict parameter.

What it means

Raised by the SpeculativeConfig._verify_args model_validator when num_speculative_tokens is None at field-validation time. This is the schema-level backstop: the constructor-level defaulting (e.g. from the draft's n_predict, covered elsewhere in __post_init__) has not run or was unavailable, so the value must be supplied explicitly unless the draft config carries n_predict.

Source

Thrown at vllm/config/speculative.py:1353

    @field_validator("attention_backend", mode="before")
    @classmethod
    def _parse_attention_backend(cls, value: Any) -> Any:
        if isinstance(value, str):
            if value.lower() == "auto":
                return None
            return AttentionBackendEnum[value.upper()]
        return value

    @model_validator(mode="after")
    def _verify_args(self) -> Self:
        if self.tensor_parallel_size is not None:
            raise ValueError(
                "'tensor_parallel_size' is not a valid argument in the "
                "speculative_config. Please pass 'draft_tensor_parallel_size' instead."
            )

        if self.num_speculative_tokens is None:
            raise ValueError(
                "num_speculative_tokens must be provided with "
                "speculative model unless the draft model config contains an "
                "n_predict parameter."
            )

        if self.num_speculative_tokens <= 0:
            raise ValueError(
                "Expected num_speculative_tokens to be greater "
                f"than zero ({self.num_speculative_tokens})."
            )

        if self.rejection_sample_method == "synthetic":
            # Consolidate to per-position rates
            self.synthetic_acceptance_rates = self._resolve_synthetic_acceptance_rates(
                self.num_speculative_tokens,
                self.synthetic_acceptance_rates,
                self.synthetic_acceptance_length,
            )

View on GitHub (pinned to c794754062)

Solutions

1. Add num_speculative_tokens (positive int, typically 1-5) to the speculative_config
2. Use a draft checkpoint whose hf_config declares n_predict so the default applies
3. If using ngram, ensure the method branch you intend is actually selected so its defaulting logic runs

Example fix

# before
SpeculativeConfig.from_kwargs(method="ngram", prompt_lookup_max=4)  # missing count
# after
SpeculativeConfig.from_kwargs(method="ngram", prompt_lookup_max=4, num_speculative_tokens=3)

REQUIRED_UNLESS_NPREDICT = {"num_speculative_tokens"}
if "num_speculative_tokens" not in spec_cfg:
    from transformers import AutoConfig
    if not hasattr(AutoConfig.from_pretrained(spec_cfg["model"]), "n_predict"):
        spec_cfg.setdefault("num_speculative_tokens", 3)

def spec_cfg_is_complete(spec_cfg: dict) -> bool:
    if "num_speculative_tokens" in spec_cfg:
        return True
    from transformers import AutoConfig
    return hasattr(AutoConfig.from_pretrained(spec_cfg["model"]), "n_predict")

When it happens

Trigger: Constructing SpeculativeConfig (directly or via speculative_config dict) with no num_speculative_tokens where the draft does not expose n_predict; commonly with method='ngram' plus explicit prompt_lookup settings, or draft_model-style configs.

Common situations: Minimal configs copied from docs examples that relied on a draft checkpoint default; direct programmatic use of SpeculativeConfig.from_kwargs where the key was dropped.

Related errors

• A speculative model was provided, but `num_speculative_token
• Expected num_speculative_tokens to be greater than zero ({se
• num_speculative_tokens:{self.num_speculative_tokens} must be
• expected str or QuantKey, got {type(v).__name__}
• unknown quantization name {v!r}; expected one of {sorted(QUA

AI-assisted analysis of vllm-project/vllm@c794754062 (2026-08-14). Data as JSON: /api/errors/a559117c3c0e76fa. Report an issue: GitHub.