vllm-project/vllm · error · ValueError

max_num_scheduled_tokens is set to {self.scheduler_config.ma

Error message

max_num_scheduled_tokens is set to {self.scheduler_config.max_num_scheduled_tokens} based on the speculative decoding settings, which does not allow any tokens to be scheduled. Increase max_num_batched_tokens to accommodate the additional draft token slots, or decrease num_speculative_tokens.

What it means

With speculative decoding, vLLM reserves extra scheduling slots for draft tokens: `max_num_scheduled_tokens` is derived from `max_num_batched_tokens` minus `max_num_new_slots_for_drafting`. If that subtraction lands at zero or below, no token can ever be scheduled, so validation raises. See `_set_max_num_scheduled_tokens` in vllm/config/vllm.py:1840.

Source

Thrown at vllm/config/vllm.py:1854

            for size in possible_sizes
            if size % self.parallel_config.tensor_parallel_size == 0
        ]

    def _set_max_num_scheduled_tokens(self):
        """
        In most cases, the scheduler may schedule a batch with as many tokens as the
        worker is configured to handle.
        """
        if self.speculative_config is not None:
            scheduled_token_delta = (
                self.speculative_config.max_num_new_slots_for_drafting
            )
            max_num_batched_tokens = self.scheduler_config.max_num_batched_tokens
            if self.scheduler_config.max_num_scheduled_tokens is None:
                self.scheduler_config.max_num_scheduled_tokens = max_num_batched_tokens

            if self.scheduler_config.max_num_scheduled_tokens <= 0:
                raise ValueError(
                    "max_num_scheduled_tokens is set to"
                    f" {self.scheduler_config.max_num_scheduled_tokens} based on"
                    " the speculative decoding settings, which does not allow"
                    " any tokens to be scheduled. Increase max_num_batched_tokens"
                    " to accommodate the additional draft token slots, or decrease"
                    " num_speculative_tokens."
                )
            if self.scheduler_config.max_num_scheduled_tokens < 8192:
                logger.warning_once(
                    "max_num_scheduled_tokens is set to"
                    f" {self.scheduler_config.max_num_scheduled_tokens} based on"
                    " the speculative decoding settings. This may lead to suboptimal"
                    " performance. Consider increasing max_num_batched_tokens to"
                    " accommodate the additional draft token slots, or decrease"
                    " num_speculative_tokens.",
                )

            if max_num_batched_tokens <= scheduled_token_delta:

View on GitHub (pinned to c794754062)

Solutions

1. Increase `--max-num-batched-tokens` so it comfortably exceeds the draft-token reservation (e.g. 8192+delta).
2. Or decrease `num_speculative_tokens` in the speculative config.
3. Recompute: ensure max_num_batched_tokens > speculative_config.max_num_new_slots_for_drafting.

Example fix

# before
vllm serve model --max-num-batched-tokens 64 \
  --speculative-config '{"method":"eagle","num_speculative_tokens":8}'

# after
vllm serve model --max-num-batched-tokens 8192 \
  --speculative-config '{"method":"eagle","num_speculative_tokens":8}'

from vllm.config.vllm import VllmConfig  # or recompute delta
# rule of thumb: max_num_batched_tokens must exceed draft reservation
if spec_config and max_num_batched_tokens <= spec_config.num_speculative_tokens + 2:
    raise SystemExit("increase --max-num-batched-tokens or lower num_speculative_tokens")

try:
    LLM(**args)
except ValueError as e:
    if "max_num_scheduled_tokens" in str(e):
        args["max_num_batched_tokens"] = max(8192, args["max_num_batched_tokens"] * 2)
    else:
        raise

When it happens

Trigger: Launching with a small `--max-num-batched-tokens` (e.g. equal to or smaller than the draft slot reservation, such as num_speculative_tokens+1 per request) together with a speculative config.

Common situations: Tuning batched tokens down for memory on small GPUs while running EAGLE/NGRAM drafting; raising num_speculative_tokens without scaling max_num_batched_tokens.

Related errors

• VllmConfig does not have enough slots to schedule a token an
• max_num_batched_tokens ({self.max_num_batched_tokens}) is sm
• max_num_batched_tokens ({self.max_num_batched_tokens}) must
• long_prefill_token_threshold ({self.long_prefill_token_thres
• rejection_sample_method='synthetic' requires exactly one of

AI-assisted analysis of vllm-project/vllm@c794754062 (2026-08-14). Data as JSON: /api/errors/f5eb2fb0db7e5cea. Report an issue: GitHub.