vllm-project/vllm · error · ValueError

long_prefill_token_threshold ({self.long_prefill_token_thres

Error message

long_prefill_token_threshold ({self.long_prefill_token_threshold}) cannot be greater than the max_model_len ({max_model_len}).

What it means

verify_max_model_len caps long_prefill_token_threshold at max_model_len: the threshold decides when a prefill is treated as 'long', and a threshold beyond the longest possible sequence is meaningless and would misroute every prefill into the long path.

Source

Thrown at vllm/config/scheduler.py:279

            )

        if self.max_num_batched_tokens < self.max_num_seqs:
            raise ValueError(
                f"max_num_batched_tokens ({self.max_num_batched_tokens}) must "
                "be greater than or equal to max_num_seqs "
                f"({self.max_num_seqs})."
            )

        if self.max_num_batched_tokens > self.max_num_seqs * max_model_len:
            logger.warning(
                "max_num_batched_tokens (%d) exceeds max_num_seqs "
                "* max_model_len (%d). This may lead to unexpected behavior.",
                self.max_num_batched_tokens,
                self.max_num_seqs * max_model_len,
            )

        if self.long_prefill_token_threshold > max_model_len:
            raise ValueError(
                "long_prefill_token_threshold "
                f"({self.long_prefill_token_threshold}) cannot be greater "
                f"than the max_model_len ({max_model_len})."
            )

        return self

View on GitHub (pinned to c794754062)

Solutions

1. Lower long_prefill_token_threshold to <= max_model_len (typically a fraction of it)
2. Or raise --max-model-len if the long context is actually needed
3. Add a startup assertion in deploy scripts: threshold <= max_model_len

Example fix

# before
--max-model-len 8192 --long-prefill-token-threshold 32768

# after
--max-model-len 8192 --long-prefill-token-threshold 4096

def threshold_ok(threshold: int, max_model_len: int) -> bool:
    return threshold <= max_model_len

null

null

When it happens

Trigger: Setting --long-prefill-token-threshold 65536 with --max-model-len 8192; keeping a tuned threshold after lowering max_model_len for memory; defaults from a benchmark script applied to a shorter-context model.

Common situations: Performance tuning carried over between models; shrinking context windows after threshold tuning; config files shared across heterogeneous deployments.

Related errors

• max_num_batched_tokens ({self.max_num_batched_tokens}) is sm
• max_num_batched_tokens ({self.max_num_batched_tokens}) must
• Unknown dtype: {dtype!r}
• 'mm_shm_cache_max_object_size_mb' should only be set when 'm
• 'mm_encoder_fp8_scale_path' and 'mm_encoder_fp8_scale_save_p

AI-assisted analysis of vllm-project/vllm@c794754062 (2026-08-14). Data as JSON: /api/errors/aafc1cc3f0a18096. Report an issue: GitHub.