vllm-project/vllm · error · Error

harmony output parsing failed

Error message

harmony output parsing failed

What it means

Top-level wrapper for failures while decoding gpt-oss model output with the native Harmony parser (the harmony message format with analysis/channel tags). The concrete failure lives in the chained `#[source] BoxedError` (the harmony crate's own error). It means the token stream did not conform to the Harmony grammar the parser expects.

Solutions

1. Read the chained source error to identify which harmony stage failed.
2. Increase `max_tokens` / max_model_len so harmony payloads are not truncated mid-stream.
3. Reset sampling parameters to the gpt-oss recommended defaults and re-test.
4. If persistent on every request, verify the model weights/config actually correspond to a harmony-format gpt-oss checkpoint.

Example fix

// before
SamplingParams { temperature: 1.5, max_tokens: Some(16), .. } // truncates harmony output

// after
SamplingParams { temperature: 1.0, max_tokens: Some(1024), .. }

match result {
    Err(vllm_chat::Error::HarmonyOutputParsing { error }) => {
        tracing::warn!(?error, "harmony parse failed; retrying once with lower temperature");
        retry_with_conservative_sampling().await
    }
    other => other,
}

When it happens

Trigger: Streaming or non-streaming completion from a gpt-oss model where the generated text cannot be parsed into Harmony events — malformed channel tags, truncated harmony payloads, or a reasoning/assistant segment that fails harmony decoding.

Common situations: Output truncated by max_tokens mid-harmony-payload; sampling parameters (very high temperature, bad grammar constraints) producing degenerate output; a fine-tune of gpt-oss that drifted from the harmony format.

Related errors

• gpt_oss uses native Harmony output parsing; generic
• chat request stream ` ` closed before terminal output
• request output stream for
• text request stream ` ` closed before terminal output
• tool call stream state is inconsistent

AI-assisted analysis of vllm-project/vllm@c794754062 (2026-08-14). Data as JSON: /api/errors/fbfc97fb0b762734. Report an issue: GitHub.

Appendix: source

Thrown at rust/src/chat/src/error.rs:60

        kind: &'static str,
        name: String,
        available_names: Vec<String>,
    },
    #[error("failed to initialize {kind} parser `{name}`")]
    ParserInitialization {
        kind: &'static str,
        name: String,
        #[source]
        error: BoxedError,
    },
    #[error(
        "gpt_oss uses native Harmony output parsing; generic {kind} parser override `{selection}` is not supported"
    )]
    HarmonyParserOverrideUnsupported {
        kind: &'static str,
        selection: String,
    },
    #[error("harmony output parsing failed")]
    HarmonyOutputParsing {
        #[source]
        error: BoxedError,
    },
    #[error(
        "this model's maximum context length is {max_model_len} tokens, \
         but the prompt contains {prompt_len} input tokens"
    )]
    PromptTooLong { max_model_len: u32, prompt_len: u32 },
    #[error("chat request stream `{request_id}` closed before terminal output")]
    StreamClosedBeforeTerminalOutput { request_id: String },
    #[error("tool call stream state is inconsistent: {message}")]
    ToolCallStreamInvariant { message: String },
    #[error("duplicate tool name `{name}`")]
    DuplicateToolName { name: String },
    #[error("tool_choice requires at least one available tool")]
    ToolChoiceRequiresTools,
    #[error("tool_choice function `{name}` was not found in the available tools")]

View on GitHub (pinned to c794754062)