vllm-project/vllm · error · Error

harmony output parsing failed

Error message

harmony output parsing failed

What it means

Top-level wrapper for failures while decoding gpt-oss model output with the native Harmony parser (the harmony message format with analysis/channel tags). The concrete failure lives in the chained `#[source] BoxedError` (the harmony crate's own error). It means the token stream did not conform to the Harmony grammar the parser expects.

Source

Thrown at rust/src/chat/src/error.rs:60

        kind: &'static str,
        name: String,
        available_names: Vec<String>,
    },
    #[error("failed to initialize {kind} parser `{name}`")]
    ParserInitialization {
        kind: &'static str,
        name: String,
        #[source]
        error: BoxedError,
    },
    #[error(
        "gpt_oss uses native Harmony output parsing; generic {kind} parser override `{selection}` is not supported"
    )]
    HarmonyParserOverrideUnsupported {
        kind: &'static str,
        selection: String,
    },
    #[error("harmony output parsing failed")]
    HarmonyOutputParsing {
        #[source]
        error: BoxedError,
    },
    #[error(
        "this model's maximum context length is {max_model_len} tokens, \
         but the prompt contains {prompt_len} input tokens"
    )]
    PromptTooLong { max_model_len: u32, prompt_len: u32 },
    #[error("chat request stream `{request_id}` closed before terminal output")]
    StreamClosedBeforeTerminalOutput { request_id: String },
    #[error("tool call stream state is inconsistent: {message}")]
    ToolCallStreamInvariant { message: String },
    #[error("duplicate tool name `{name}`")]
    DuplicateToolName { name: String },
    #[error("tool_choice requires at least one available tool")]
    ToolChoiceRequiresTools,
    #[error("tool_choice function `{name}` was not found in the available tools")]

View on GitHub (pinned to c794754062)

Solutions

1. Read the chained source error to identify which harmony stage failed.
2. Increase `max_tokens` / max_model_len so harmony payloads are not truncated mid-stream.
3. Reset sampling parameters to the gpt-oss recommended defaults and re-test.
4. If persistent on every request, verify the model weights/config actually correspond to a harmony-format gpt-oss checkpoint.

Example fix

// before
SamplingParams { temperature: 1.5, max_tokens: Some(16), .. } // truncates harmony output

// after
SamplingParams { temperature: 1.0, max_tokens: Some(1024), .. }

match result {
    Err(vllm_chat::Error::HarmonyOutputParsing { error }) => {
        tracing::warn!(?error, "harmony parse failed; retrying once with lower temperature");
        retry_with_conservative_sampling().await
    }
    other => other,
}

When it happens

Trigger: Streaming or non-streaming completion from a gpt-oss model where the generated text cannot be parsed into Harmony events — malformed channel tags, truncated harmony payloads, or a reasoning/assistant segment that fails harmony decoding.

Common situations: Output truncated by max_tokens mid-harmony-payload; sampling parameters (very high temperature, bad grammar constraints) producing degenerate output; a fine-tune of gpt-oss that drifted from the harmony format.

Related errors

• gpt_oss uses native Harmony output parsing; generic {kind} p
• chat request stream `{request_id}` closed before terminal ou
• tool call stream state is inconsistent: {message}
• request output stream for `{request_id}` closed unexpectedly
• tool parser parsing failed: {message}

AI-assisted analysis of vllm-project/vllm@c794754062 (2026-08-14). Data as JSON: /api/errors/fbfc97fb0b762734. Report an issue: GitHub.