{"record":{"id":"e6d7e181092b7ea7","repo":"vllm-project/vllm","slug":"tokenizer-is-missing-unified-parser-token-token","errorCode":null,"errorMessage":"tokenizer is missing unified parser token `{token}`","messagePattern":"tokenizer is missing unified parser token `(.+?)`","errorType":"exception","errorClass":"UnifiedParserError","httpStatus":null,"severity":"error","filePath":"rust/src/parser/src/unified/mod.rs","lineNumber":206,"sourceCode":"\n    /// Flush any buffered parser state at end of stream.\n    fn finish(&mut self) -> Result<UnifiedParserOutput> {\n        Ok(UnifiedParserOutput::default())\n    }\n\n    /// Clear parser state and return currently uncommitted buffered text.\n    fn reset(&mut self) -> String {\n        String::new()\n    }\n}\n\n/// Errors produced while creating or running unified parsers.\n#[derive(Debug, Error, Macro)]\n#[thiserror_ext(macro(path = \"crate::unified\", mangle))]\npub enum UnifiedParserError {\n    #[error(\"combined parser is constructed from split parser instances\")]\n    CombinedParserConstructor,\n    #[error(\"tokenizer is missing unified parser token `{token}`\")]\n    MissingToken { token: String },\n    #[error(\"unified parser parsing failed: {message}\")]\n    ParsingFailed { message: String },\n    #[error(transparent)]\n    Reasoning(#[from] ReasoningError),\n    #[error(transparent)]\n    Tool(#[from] ToolParserError),\n}\n\n/// Returns the ID for the given token, or an error if it's not found.\nfn token_id(tokenizer: &dyn vllm_tokenizer::Tokenizer, token: &str) -> Result<u32> {\n    tokenizer.token_to_id(token).ok_or_else(|| UnifiedParserError::MissingToken {\n        token: token.to_string(),\n    })\n}\n","sourceCodeStart":188,"sourceCodeEnd":222,"githubUrl":"https://github.com/vllm-project/vllm/blob/c794754062d49a8fdb63ab3c5215b488b865030c/rust/src/parser/src/unified/mod.rs#L188-L222","documentation":"UnifiedParserError::MissingToken is raised by the token_id helper when the tokenizer has no ID for a token the unified parser requires (start/end delimiters for reasoning sections or tool-call boundaries, e.g. <|tool_call|>-style markers). The parser cannot detect section boundaries without these IDs.","triggerScenarios":"Enabling a unified parser whose special tokens are absent from the served tokenizer's vocabulary — token_to_id() returns None and token_id() maps that to this error.","commonSituations":"Unified parser family (e.g. granite) used with a tokenizer from another model; stripped-down or re-exported tokenizers missing special tokens; parser/tokenizer version mismatch after upgrades.","solutions":["Pair the unified parser with the tokenizer/model it was designed for","Pre-check tokenizer.token_to_id() for the token named in the error before enabling the parser","Add the missing special tokens back to the tokenizer vocabulary if using a custom merge"],"exampleFix":"# before\n--model some/model --tool-parser granite --reasoning-parser granite\n\n# after\n--model ibm-granite/granite-3.x --tool-parser granite --reasoning-parser granite","handlingStrategy":"validation","validationCode":"for token in required_tokens_for_family(&family) {\n    tokenizer.token_to_id(token)\n        .ok_or_else(|| anyhow::anyhow!(\"tokenizer missing {token}; unified parser {family} unusable\"))?;\n}","typeGuard":"fn is_unified_missing_token(e: &UnifiedParserError) -> bool {\n    matches!(e, UnifiedParserError::MissingToken { .. })\n}","tryCatchPattern":"if let Err(UnifiedParserError::MissingToken { token }) = build_unified(family, &tokenizer) {\n    return Err(config_err!(\"add {token} to the tokenizer or disable unified parser {family}\"));\n}","preventionTips":["Use the model family the unified parser targets","Add startup assertions for all required special tokens","After tokenizer edits/merges, verify special-token coverage before enabling unified parsing"],"tags":["tokenizer","unified-parser","configuration","rust"],"backgroundTag":null,"analyzedSha":"c794754062d49a8fdb63ab3c5215b488b865030c","analyzedAt":"2026-08-14T21:17:39.825Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}