vllm-project/vllm · error · Error

tool call stream state is inconsistent: {message}

Error message

tool call stream state is inconsistent: {message}

What it means

Internal invariant violation in tool-call stream assembly: while folding deltas into tool calls, the state machine reached a combination it cannot represent (raised at `output/structured.rs:134` and `output/default/unified.rs:177/185`). This is a defect-level error — either a parser bug or non-conforming model output — not a user input error (not listed in `is_request_validation_error`).

Source

Thrown at rust/src/chat/src/error.rs:72

        "gpt_oss uses native Harmony output parsing; generic {kind} parser override `{selection}` is not supported"
    )]
    HarmonyParserOverrideUnsupported {
        kind: &'static str,
        selection: String,
    },
    #[error("harmony output parsing failed")]
    HarmonyOutputParsing {
        #[source]
        error: BoxedError,
    },
    #[error(
        "this model's maximum context length is {max_model_len} tokens, \
         but the prompt contains {prompt_len} input tokens"
    )]
    PromptTooLong { max_model_len: u32, prompt_len: u32 },
    #[error("chat request stream `{request_id}` closed before terminal output")]
    StreamClosedBeforeTerminalOutput { request_id: String },
    #[error("tool call stream state is inconsistent: {message}")]
    ToolCallStreamInvariant { message: String },
    #[error("duplicate tool name `{name}`")]
    DuplicateToolName { name: String },
    #[error("tool_choice requires at least one available tool")]
    ToolChoiceRequiresTools,
    #[error("tool_choice function `{name}` was not found in the available tools")]
    ToolChoiceFunctionNotFound { name: String },
    #[error("failed to build structural tag: {message}")]
    StructuralTag { message: String },
    #[error(transparent)]
    Text(#[from] vllm_text::Error),
    #[error(transparent)]
    Tokenizer(#[from] vllm_tokenizer::TokenizerError),
}

pub type Result<T> = std::result::Result<T, Error>;

impl Error {

View on GitHub (pinned to c794754062)

Solutions

1. Capture the full raw delta sequence for the failing request to see which invariant (index/name/arguments ordering) broke.
2. Try a different tool-call parser that matches the model's trained format.
3. Retry the request non-streaming — the full-text parser path avoids incremental state entirely.
4. If it reproduces with stock models/parsers, file a bug with the `message` field content and delta trace (it names the exact invariant).

Example fix

// before
parser: "hermes" // model emits mistral-style [TOOL_CALLS] tags

// after
parser: "mistral" // parser format matches model training

match stream_result {
    Err(e @ vllm_chat::Error::ToolCallStreamInvariant { .. }) => {
        tracing::warn!(%e, "incremental tool-call decode failed; retrying non-streaming");
        client.chat(request.with_stream(false)).await // full-text parse path
    }
    other => other,
}

When it happens

Trigger: Streaming with a tool-call parser where deltas arrive out of the expected order (e.g. argument delta for a tool index that was never opened, or two `name` deltas for one index); model emitting malformed partial tool-call JSON that the incremental decoder cannot merge.

Common situations: Custom or newly added tool parser with a buggy incremental mode; a model fine-tune that emits interleaved/duplicated tool-call tags; switching parsers (e.g. from hermes to mistral style) against a model trained for the other format.

Related errors

• harmony output parsing failed
• chat request stream `{request_id}` closed before terminal ou
• request output stream for `{request_id}` closed unexpectedly
• tool parser parsing failed: {message}
• text request stream `{request_id}` closed before terminal ou

AI-assisted analysis of vllm-project/vllm@c794754062 (2026-08-14). Data as JSON: /api/errors/1682fa493f5807c3. Report an issue: GitHub.