{"record":{"id":"3fa88f26f3402a31","repo":"vllm-project/vllm","slug":"tokenizer-is-missing-reasoning-delimiter-token-t","errorCode":null,"errorMessage":"tokenizer is missing reasoning delimiter token `{token}`","messagePattern":"tokenizer is missing reasoning delimiter token `(.+?)`","errorType":"exception","errorClass":"ReasoningError","httpStatus":null,"severity":"error","filePath":"rust/src/parser/src/reasoning/mod.rs","lineNumber":131,"sourceCode":"    /// Some model families emit reasoning sentinels as special tokens. Those\n    /// parsers need `skip_special_tokens = false` while parsing is enabled.\n    fn preserve_special_tokens(&self) -> bool {\n        false\n    }\n\n    /// Feed one decoded text delta into the parser.\n    fn push(&mut self, delta: &str) -> Result<ReasoningDelta>;\n\n    /// Flush any buffered partial delimiter state at end of stream.\n    fn finish(&mut self) -> Result<ReasoningDelta> {\n        Ok(ReasoningDelta::default())\n    }\n}\n\n/// Errors produced while creating or running reasoning parsers.\n#[derive(Debug, Error)]\npub enum ReasoningError {\n    #[error(\"tokenizer is missing reasoning delimiter token `{token}`\")]\n    MissingToken { token: String },\n    #[error(\n        \"`{name}` only provides a unified parser; the same reasoning parser and tool parser should be specified together\"\n    )]\n    DummyUnifiedParser { name: String },\n}\n\n#[cfg(test)]\nmod tests;\n","sourceCodeStart":113,"sourceCodeEnd":141,"githubUrl":"https://github.com/vllm-project/vllm/blob/c794754062d49a8fdb63ab3c5215b488b865030c/rust/src/parser/src/reasoning/mod.rs#L113-L141","documentation":"ReasoningError::MissingToken is thrown when constructing a reasoning parser whose configured delimiter token (e.g. </think> or a similar end-of-reasoning marker) does not exist in the tokenizer's vocabulary. The parser needs the token's ID to detect reasoning boundaries, so token_to_id returning None is fatal.","triggerScenarios":"Selecting a reasoning parser (e.g. deepseek-r1 style) with a tokenizer that lacks the parser's delimiter token — typically a tokenizer for a different model family, or a custom/converted tokenizer missing special tokens.","commonSituations":"Using --reasoning-parser with a base model tokenizer that has no <think>/</think> tokens; fine-tuned or merged tokenizers that dropped special tokens; parser name not matching the served model.","solutions":["Use the reasoning parser that matches the model family (e.g. deepseek_r1 parsers with DeepSeek-R1 tokenizers)","Verify the delimiter token exists: tokenizer.token_to_id(\"</think>\") before enabling the parser","If using a custom tokenizer, re-add the required special tokens to the vocab"],"exampleFix":"# before (wrong pairing)\n--model meta-llama/Llama-3-8B --reasoning-parser deepseek_r1\n\n# after\n--model deepseek-ai/DeepSeek-R1 --reasoning-parser deepseek_r1","handlingStrategy":"validation","validationCode":"// Verify delimiter token exists before enabling the parser\nlet tok = tokenizer.token_to_id(\"</think>\")\n    .ok_or_else(|| anyhow::anyhow!(\"tokenizer lacks </think>; pick a matching reasoning parser\"))?;","typeGuard":"fn is_missing_reasoning_token(e: &ReasoningError) -> bool {\n    matches!(e, ReasoningError::MissingToken { .. })\n}","tryCatchPattern":"if let Err(ReasoningError::MissingToken { token }) = parser_result {\n    return Err(config_err!(\"reasoning parser needs token {token} not in this tokenizer; disable it or switch parser\"));\n}","preventionTips":["Pair parser family with model family (deepseek_r1 parser with R1 tokenizers)","Smoke-test token_to_id for the delimiter token in your serving startup checks","After tokenizer merges/conversions, assert special tokens survived"],"tags":["tokenizer","reasoning-parser","configuration","rust"],"backgroundTag":null,"analyzedSha":"c794754062d49a8fdb63ab3c5215b488b865030c","analyzedAt":"2026-08-14T21:17:39.825Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}