{"record":{"id":"5aaf461dcd5170db","repo":"sgl-project/sglang","slug":"think-end-token-reasoning-parser-detector-think","errorCode":null,"errorMessage":"think_end_token '{reasoning_parser.detector.think_end_token}' could not be encoded by the tokenizer.","messagePattern":"think_end_token '(.+?)' could not be encoded by the tokenizer\\.","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"python/sglang/srt/constrained/reasoner_grammar_backend.py","lineNumber":257,"sourceCode":"                old_output_ids, new_output_ids, next_state\n            )\n\n\nclass ReasonerGrammarBackend(BaseGrammarBackend):\n    def __init__(\n        self,\n        grammar_backend: BaseGrammarBackend,\n        reasoning_parser: ReasoningParser,\n        tokenizer: Union[PreTrainedTokenizer, PreTrainedTokenizerFast],\n        enable_strict_thinking: bool = False,\n    ):\n        super().__init__()\n        self.grammar_backend = grammar_backend\n        think_end_ids = tokenizer.encode(\n            reasoning_parser.detector.think_end_token, add_special_tokens=False\n        )\n        if not think_end_ids:\n            raise ValueError(\n                f\"think_end_token '{reasoning_parser.detector.think_end_token}' \"\n                f\"could not be encoded by the tokenizer.\"\n            )\n        self.think_end_ids = think_end_ids\n        self._enable_strict_thinking = enable_strict_thinking\n        self.think_excluded_token_ids = self._get_think_excluded_token_ids(\n            reasoning_parser, tokenizer\n        )\n        self.max_think_tokens = envs.SGLANG_MAX_THINK_TOKENS.get()\n        self.enable_token_filter = self.enable_strict_thinking and (\n            self.think_excluded_token_ids is not None or self.max_think_tokens >= 0\n        )\n        if (\n            self.enable_token_filter\n            and not self.grammar_backend.is_support_token_filter\n        ):\n            raise ValueError(\n                \"Strict reasoning format requested but the grammar backend does not \"","sourceCodeStart":239,"sourceCodeEnd":275,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/srt/constrained/reasoner_grammar_backend.py#L239-L275","documentation":"ReasonerGrammarBackend encodes the reasoning parser's think-end marker (e.g. '</think>') with the model tokenizer; if encoding yields no token IDs, it raises ValueError because strict reasoning enforcement cannot detect the end of thinking.","triggerScenarios":"Constructing ReasonerGrammarBackend where reasoning_parser.detector.think_end_token is empty or not representable by the tokenizer (yields zero tokens with add_special_tokens=False).","commonSituations":"Custom reasoning parsers with a misconfigured think_end_token; tokenizers whose vocab genuinely lacks pieces for the marker; empty-string detector fields on new model integrations.","solutions":["Verify reasoning_parser.detector.think_end_token is set to the correct literal (e.g. '</think>') for the model","Test tokenizer.encode(think_end_token, add_special_tokens=False) manually; if empty, fix the tokenizer/parser config","Use a different --reasoning-parser value matching the model"],"exampleFix":null,"handlingStrategy":"validation","validationCode":"ids = tokenizer.encode(parser.detector.think_end_token, add_special_tokens=False)\nassert ids, 'think_end_token not encodable by this tokenizer'","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Add a startup assertion for reasoning-parser markers on new model integrations","Unit-test detector marker tokens against the model tokenizer in CI"],"tags":["reasoning","tokenizer","grammar","startup"],"backgroundTag":"token-encoding-failed","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}