vllm-project/vllm · error · ValueError

VllmConfig does not have enough slots to schedule a token an

Error message

VllmConfig does not have enough slots to schedule a token and support the speculative decoding settings. Got {max_num_batched_tokens=} and {scheduled_token_delta=}.

What it means

A stricter companion check to [391]: even when max_num_scheduled_tokens is positive, the config must leave at least one slot for a real token after reserving `scheduled_token_delta` draft slots. If `max_num_batched_tokens <= max_num_new_slots_for_drafting`, validation fails with both values printed.

Source

Thrown at vllm/config/vllm.py:1873

                    "max_num_scheduled_tokens is set to"
                    f" {self.scheduler_config.max_num_scheduled_tokens} based on"
                    " the speculative decoding settings, which does not allow"
                    " any tokens to be scheduled. Increase max_num_batched_tokens"
                    " to accommodate the additional draft token slots, or decrease"
                    " num_speculative_tokens."
                )
            if self.scheduler_config.max_num_scheduled_tokens < 8192:
                logger.warning_once(
                    "max_num_scheduled_tokens is set to"
                    f" {self.scheduler_config.max_num_scheduled_tokens} based on"
                    " the speculative decoding settings. This may lead to suboptimal"
                    " performance. Consider increasing max_num_batched_tokens to"
                    " accommodate the additional draft token slots, or decrease"
                    " num_speculative_tokens.",
                )

            if max_num_batched_tokens <= scheduled_token_delta:
                raise ValueError(
                    "VllmConfig does not have enough slots to schedule a token and"
                    " support the speculative decoding settings."
                    f" Got {max_num_batched_tokens=} and {scheduled_token_delta=}."
                )

    def _set_cudagraph_sizes(self):
        """
        vLLM defines the default candidate list of batch sizes for CUDA graph
        capture as:

        ```python
        default_max_graph_size = 1024 if is_data_center_blackwell else 512
        max_graph_size = min(max_num_seqs * decode_query_len * 2,
                             default_max_graph_size)
        # 1, 2, 4, then multiples of 8 up to 256 and then multiples of 16
        # up to max_graph_size
        cudagraph_capture_sizes = [1, 2, 4] + list(range(8, 256, 8)) + list(
            range(256, max_graph_size + 1, 16))

View on GitHub (pinned to c794754062)

Solutions

  1. Raise `--max-num-batched-tokens` above the reported scheduled_token_delta value.
  2. Or reduce `num_speculative_tokens` (or ngram k / lookahead) to shrink the reservation.
  3. Leave max_num_batched_tokens at default unless memory forces a reduction.

Example fix

# before (error reports e.g. max_num_batched_tokens=256, scheduled_token_delta=512)
vllm serve model --max-num-batched-tokens 256 \
  --speculative-config '{"method":"ngram","num_speculative_tokens":56,...}'

# after
vllm serve model --max-num-batched-tokens 4096 \
  --speculative-config '{"method":"ngram","num_speculative_tokens":56,...}'
Defensive patterns

Strategy: validation

Validate before calling

from vllm.config import SpeculativeConfig
# before engine start
sc = SpeculativeConfig(**spec_kwargs)
if max_num_batched_tokens <= sc.max_num_new_slots_for_drafting:
    raise SystemExit(f"max_num_batched_tokens must be > {sc.max_num_new_slots_for_drafting}")

Try / catch

try:
    LLM(**args)
except ValueError as e:
    if "not enough slots to schedule" in str(e):
        args["max_num_batched_tokens"] *= 4  # then retry
    else:
        raise

Prevention

When it happens

Trigger: Any speculative-decode launch where `--max-num-batched-tokens` is less than or equal to `speculative_config.max_num_new_slots_for_drafting` (a function of num_speculative_tokens and lookahead/k values, computed in vllm/config/speculative.py:1421).

Common situations: Aggressive speculative settings (large num_speculative_tokens or ngram k) on top of a small max_num_batched_tokens; auto-derived batched tokens on memory-constrained GPUs.

Related errors

AI-assisted analysis of vllm-project/vllm@c794754062 (2026-08-14). Data as JSON: /api/errors/2c83327a6916923b. Report an issue: GitHub.