vllm-project/vllm · error · UnifiedParserError

tokenizer is missing unified parser token `{token}`

Error message

tokenizer is missing unified parser token `{token}`

What it means

UnifiedParserError::MissingToken is raised by the token_id helper when the tokenizer has no ID for a token the unified parser requires (start/end delimiters for reasoning sections or tool-call boundaries, e.g. <|tool_call|>-style markers). The parser cannot detect section boundaries without these IDs.

Source

Thrown at rust/src/parser/src/unified/mod.rs:206

    /// Flush any buffered parser state at end of stream.
    fn finish(&mut self) -> Result<UnifiedParserOutput> {
        Ok(UnifiedParserOutput::default())
    }

    /// Clear parser state and return currently uncommitted buffered text.
    fn reset(&mut self) -> String {
        String::new()
    }
}

/// Errors produced while creating or running unified parsers.
#[derive(Debug, Error, Macro)]
#[thiserror_ext(macro(path = "crate::unified", mangle))]
pub enum UnifiedParserError {
    #[error("combined parser is constructed from split parser instances")]
    CombinedParserConstructor,
    #[error("tokenizer is missing unified parser token `{token}`")]
    MissingToken { token: String },
    #[error("unified parser parsing failed: {message}")]
    ParsingFailed { message: String },
    #[error(transparent)]
    Reasoning(#[from] ReasoningError),
    #[error(transparent)]
    Tool(#[from] ToolParserError),
}

/// Returns the ID for the given token, or an error if it's not found.
fn token_id(tokenizer: &dyn vllm_tokenizer::Tokenizer, token: &str) -> Result<u32> {
    tokenizer.token_to_id(token).ok_or_else(|| UnifiedParserError::MissingToken {
        token: token.to_string(),
    })
}

View on GitHub (pinned to c794754062)

Solutions

1. Pair the unified parser with the tokenizer/model it was designed for
2. Pre-check tokenizer.token_to_id() for the token named in the error before enabling the parser
3. Add the missing special tokens back to the tokenizer vocabulary if using a custom merge

Example fix

# before
--model some/model --tool-parser granite --reasoning-parser granite

# after
--model ibm-granite/granite-3.x --tool-parser granite --reasoning-parser granite

for token in required_tokens_for_family(&family) {
    tokenizer.token_to_id(token)
        .ok_or_else(|| anyhow::anyhow!("tokenizer missing {token}; unified parser {family} unusable"))?;
}

fn is_unified_missing_token(e: &UnifiedParserError) -> bool {
    matches!(e, UnifiedParserError::MissingToken { .. })
}

if let Err(UnifiedParserError::MissingToken { token }) = build_unified(family, &tokenizer) {
    return Err(config_err!("add {token} to the tokenizer or disable unified parser {family}"));
}

When it happens

Trigger: Enabling a unified parser whose special tokens are absent from the served tokenizer's vocabulary — token_to_id() returns None and token_id() maps that to this error.

Common situations: Unified parser family (e.g. granite) used with a tokenizer from another model; stripped-down or re-exported tokenizers missing special tokens; parser/tokenizer version mismatch after upgrades.

Related errors

• tokenizer is missing reasoning delimiter token `{token}`
• `{name}` only provides a unified parser; the same reasoning
• `{name}` only provides a unified parser; the same reasoning
• {kind} parser `{name}` is not registered{}
• gpt_oss uses native Harmony output parsing; generic {kind} p

AI-assisted analysis of vllm-project/vllm@c794754062 (2026-08-14). Data as JSON: /api/errors/e6d7e181092b7ea7. Report an issue: GitHub.