vllm-project/vllm · error · ValueError

max_num_batched_tokens ({self.max_num_batched_tokens}) is sm

Error message

max_num_batched_tokens ({self.max_num_batched_tokens}) is smaller than max_model_len ({max_model_len}). This effectively limits the maximum sequence length to max_num_batched_tokens and makes vLLM reject longer sequences. Please increase max_num_batched_tokens or decrease max_model_len.

What it means

SchedulerConfig.verify_max_model_len rejects configs where max_num_batched_tokens < max_model_len and chunked prefill is off. Without chunking, a single prefill must fit in one batched-tokens budget, so any prompt longer than max_num_batched_tokens could never be scheduled.

Source

Thrown at vllm/config/scheduler.py:254

            )

        self.max_num_encoder_input_tokens = self.max_num_batched_tokens
        self.encoder_cache_size = self.max_num_batched_tokens

        if self.enable_chunked_prefill:
            logger.info_once(
                "Chunked prefill is enabled with max_num_batched_tokens=%d.",
                self.max_num_batched_tokens,
            )

        self.verify_max_model_len(max_model_len)

    def verify_max_model_len(self, max_model_len: int) -> Self:
        if (
            self.max_num_batched_tokens < max_model_len
            and not self.enable_chunked_prefill
        ):
            raise ValueError(
                f"max_num_batched_tokens ({self.max_num_batched_tokens}) is "
                f"smaller than max_model_len ({max_model_len}). "
                "This effectively limits the maximum sequence length to "
                "max_num_batched_tokens and makes vLLM reject longer "
                "sequences. Please increase max_num_batched_tokens or "
                "decrease max_model_len."
            )

        if self.max_num_batched_tokens < self.max_num_seqs:
            raise ValueError(
                f"max_num_batched_tokens ({self.max_num_batched_tokens}) must "
                "be greater than or equal to max_num_seqs "
                f"({self.max_num_seqs})."
            )

        if self.max_num_batched_tokens > self.max_num_seqs * max_model_len:
            logger.warning(
                "max_num_batched_tokens (%d) exceeds max_num_seqs "

View on GitHub (pinned to c794754062)

Solutions

1. Enable chunked prefill (--enable-chunked-prefill) so long prompts are split across steps
2. Raise --max-num-batched-tokens to at least max_model_len
3. Lower --max-model-len to fit within the current budget
4. Set a VLLM_* default/env or config that keeps the two consistent at deploy time

Example fix

# before
--max-num-batched-tokens 2048 --max-model-len 32768

# after
--max-num-batched-tokens 2048 --max-model-len 32768 --enable-chunked-prefill

def scheduler_ok(mnbt: int, max_model_len: int, chunked: bool) -> bool:
    return chunked or mnbt >= max_model_len

null

null

When it happens

Trigger: Small --max-num-batched-tokens (e.g. 2048) with a model whose max_model_len is 8192+ and --enable-chunked-prefill absent/false; deriving max_num_batched_tokens from a default while loading a long-context model; API server startup with explicit small budget.

Common situations: Tuning batched tokens down for memory on GPUs; enabling long-context models (32k/128k) while keeping old scheduler settings; chunked prefill disabled for cudagraph/compat reasons on older versions.

Related errors

• max_num_batched_tokens ({self.max_num_batched_tokens}) must
• long_prefill_token_threshold ({self.long_prefill_token_thres
• this model's maximum context length is {max_model_len} token
• Unknown dtype: {dtype!r}
• {msg} To allow overriding this maximum, set the env var VLLM

AI-assisted analysis of vllm-project/vllm@c794754062 (2026-08-14). Data as JSON: /api/errors/ed187b46b5afb7f3. Report an issue: GitHub.