vllm-project/vllm · error · Error

this model's maximum context length is {max_model_len} token

Error message

this model's maximum context length is {max_model_len} tokens, but the prompt contains {prompt_len} input tokens

What it means

Classic context-length rejection: the tokenized prompt (`prompt_len`) exceeds the model's configured `max_model_len` (u32 token counts carried in the error). Thrown by the chat layer during request validation before scheduling; `error.rs:94` classifies it in `is_request_validation_error()` as a user-request error, so it maps to an HTTP 4xx at the API boundary rather than a server fault.

Source

Thrown at rust/src/chat/src/error.rs:65

    ParserInitialization {
        kind: &'static str,
        name: String,
        #[source]
        error: BoxedError,
    },
    #[error(
        "gpt_oss uses native Harmony output parsing; generic {kind} parser override `{selection}` is not supported"
    )]
    HarmonyParserOverrideUnsupported {
        kind: &'static str,
        selection: String,
    },
    #[error("harmony output parsing failed")]
    HarmonyOutputParsing {
        #[source]
        error: BoxedError,
    },
    #[error(
        "this model's maximum context length is {max_model_len} tokens, \
         but the prompt contains {prompt_len} input tokens"
    )]
    PromptTooLong { max_model_len: u32, prompt_len: u32 },
    #[error("chat request stream `{request_id}` closed before terminal output")]
    StreamClosedBeforeTerminalOutput { request_id: String },
    #[error("tool call stream state is inconsistent: {message}")]
    ToolCallStreamInvariant { message: String },
    #[error("duplicate tool name `{name}`")]
    DuplicateToolName { name: String },
    #[error("tool_choice requires at least one available tool")]
    ToolChoiceRequiresTools,
    #[error("tool_choice function `{name}` was not found in the available tools")]
    ToolChoiceFunctionNotFound { name: String },
    #[error("failed to build structural tag: {message}")]
    StructuralTag { message: String },
    #[error(transparent)]
    Text(#[from] vllm_text::Error),

View on GitHub (pinned to c794754062)

Solutions

1. Client-side: truncate or summarize the conversation so tokenized prompt < max_model_len (leave headroom for output tokens).
2. Count tokens with the same tokenizer the server uses before sending (including chat-template overhead).
3. Server-side: raise `max_model_len` in the engine config if the model's native context supports it.
4. Enable/keep automatic prefix caching so repeated long prefixes are not re-billed, letting you trim history safely.

Example fix

// before
let messages = full_history; // 200k tokens, max_model_len = 128k

// after
let messages = trim_to_token_budget(full_history, max_model_len - reserved_output);

let prompt_len = tokenizer.count_chat_tokens(&request)?;
if prompt_len + request.max_tokens.unwrap_or(16) > max_model_len as usize {
    return Err(TrimHistory(prompt_len, max_model_len));
}

match result {
    Err(vllm_chat::Error::PromptTooLong { max_model_len, prompt_len }) => {
        // 4xx to the client, never a 500
        respond_json(StatusCode::BAD_REQUEST, json!({
            "error": "prompt too long",
            "max_model_len": max_model_len,
            "prompt_len": prompt_len
        }))
    }
    other => other?,
}

When it happens

Trigger: Sending a chat request whose messages tokenize to more tokens than `max_model_len`; includes tokenizer-added special tokens and chat template overhead, so a prompt sized exactly to the limit can still fail.

Common situations: Large pasted documents or long conversation histories; retrieval-augmented prompts stuffing too many chunks; images/multimodal content consuming the budget; `--max-model-len` set lower than what client code assumes.

Related errors

• duplicate tool name `{name}`
• tool_choice requires at least one available tool
• tool_choice function `{name}` was not found in the available
• this model's maximum context length is {max_model_len} token
• token_id(s) {token_ids:?} in {parameter} are out of vocabula

AI-assisted analysis of vllm-project/vllm@c794754062 (2026-08-14). Data as JSON: /api/errors/e55e78f31e71872f. Report an issue: GitHub.