vllm-project/vllm · error · Error

chat request stream `{request_id}` closed before terminal ou

Error message

chat request stream `{request_id}` closed before terminal output

What it means

The chat request's response stream ended (client disconnect or stream terminator) before any terminal output event (e.g. finish/stop) was produced — see `rust/src/chat/src/stream.rs:114` and `:158`. The `request_id` field identifies which request died. It signals a protocol/lifecycle violation: a stream must terminate with a terminal message.

Source

Thrown at rust/src/chat/src/error.rs:70

    },
    #[error(
        "gpt_oss uses native Harmony output parsing; generic {kind} parser override `{selection}` is not supported"
    )]
    HarmonyParserOverrideUnsupported {
        kind: &'static str,
        selection: String,
    },
    #[error("harmony output parsing failed")]
    HarmonyOutputParsing {
        #[source]
        error: BoxedError,
    },
    #[error(
        "this model's maximum context length is {max_model_len} tokens, \
         but the prompt contains {prompt_len} input tokens"
    )]
    PromptTooLong { max_model_len: u32, prompt_len: u32 },
    #[error("chat request stream `{request_id}` closed before terminal output")]
    StreamClosedBeforeTerminalOutput { request_id: String },
    #[error("tool call stream state is inconsistent: {message}")]
    ToolCallStreamInvariant { message: String },
    #[error("duplicate tool name `{name}`")]
    DuplicateToolName { name: String },
    #[error("tool_choice requires at least one available tool")]
    ToolChoiceRequiresTools,
    #[error("tool_choice function `{name}` was not found in the available tools")]
    ToolChoiceFunctionNotFound { name: String },
    #[error("failed to build structural tag: {message}")]
    StructuralTag { message: String },
    #[error(transparent)]
    Text(#[from] vllm_text::Error),
    #[error(transparent)]
    Tokenizer(#[from] vllm_tokenizer::TokenizerError),
}

pub type Result<T> = std::result::Result<T, Error>;

View on GitHub (pinned to c794754062)

Solutions

Match the `request_id` to server logs to find which request/stream aborted.
Raise client and intermediary (proxy/LB) read timeouts for streaming endpoints.
If your client intentionally aborts streams, treat this error as expected and ignore/log it rather than failing.
Check for panics or early returns in custom stream middleware that skip the terminal event.

Example fix

// before
let stream = client.chat_stream(req); // client drops stream early

// after
let mut stream = client.chat_stream(req)?;
while let Some(ev) = stream.next().await {
    handle(ev)?; // consume until the terminal event before dropping
}

Defensive patterns

Strategy: try-catch

Try / catch

if matches!(err, vllm_chat::Error::StreamClosedBeforeTerminalOutput { .. }) {
    // client disconnect or upstream abort: log at info, do not alert
    tracing::info!(?err, "stream aborted before terminal output");
    return Ok(());
}

Prevention

Consume streams to the terminal event before dropping them in client code.
Set proxy/LB idle timeouts above your longest expected inter-chunk gap for SSE.
Correlate request_id from the error with server logs instead of guessing.

When it happens

Trigger: SSE/streaming chat response where the underlying transport closes or the generator is dropped before emitting the terminal chunk; client aborts the HTTP request mid-stream; internal task handling the stream panics or is cancelled.

Common situations: Client timeouts (proxy idle timeouts killing SSE connections); load balancer cutting long streams; bugs where the stream handler returns early without a finish event; disconnect-on-first-token patterns.

Related errors

AI-assisted analysis of vllm-project/vllm@c794754062 (2026-08-14). Data as JSON: /api/errors/18dfc97cbb3db09e. Report an issue: GitHub.