{"record":{"id":"09255a5f3b137542","repo":"sgl-project/sglang","slug":"think-excluded-token-token-could-not-be-encode","errorCode":null,"errorMessage":"think_excluded_token '{token}' could not be encoded by the tokenizer. All excluded tokens must be encodable for strict reasoning mode to function correctly.","messagePattern":"think_excluded_token '(.+?)' could not be encoded by the tokenizer\\. All excluded tokens must be encodable for strict reasoning mode to function correctly\\.","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"python/sglang/srt/constrained/reasoner_grammar_backend.py","lineNumber":296,"sourceCode":"            )\n        self._token_filter_fn = (\n            self.grammar_backend.set_token_filter if self.enable_token_filter else None\n        )\n\n    def _get_think_excluded_token_ids(\n        self,\n        reasoning_parser: ReasoningParser,\n        tokenizer: Union[PreTrainedTokenizer, PreTrainedTokenizerFast],\n    ) -> Optional[List[int]]:\n        excluded_ids = []\n        if (not self.enable_strict_thinking) or (\n            not reasoning_parser.detector.think_excluded_tokens\n        ):\n            return None\n        for token in reasoning_parser.detector.think_excluded_tokens:\n            new_ids = tokenizer.encode(token, add_special_tokens=False)\n            if not new_ids:\n                raise ValueError(\n                    f\"think_excluded_token '{token}' could not be encoded by the \"\n                    f\"tokenizer. All excluded tokens must be encodable for strict \"\n                    f\"reasoning mode to function correctly.\"\n                )\n            excluded_ids += new_ids\n        return excluded_ids\n\n    def _make_grammar_object(\n        self, grammar: Optional[BaseGrammarObject], reasoning: bool\n    ) -> ReasonerGrammarObject:\n        obj = ReasonerGrammarObject(\n            grammar=grammar,\n            think_end_ids=self.think_end_ids,\n            think_excluded_token_ids=self.think_excluded_token_ids,\n            max_think_tokens=self.max_think_tokens,\n            enable_token_filter=self.enable_token_filter,\n            token_filter_fn=self._token_filter_fn,\n            allocate_vocab_mask_fn=self.grammar_backend.allocate_vocab_mask,","sourceCodeStart":278,"sourceCodeEnd":314,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/srt/constrained/reasoner_grammar_backend.py#L278-L314","documentation":"Each token in reasoning_parser.detector.think_excluded_tokens must encode to at least one tokenizer ID for strict reasoning mode; _get_think_excluded_token_ids raises ValueError for any token that encodes to nothing.","triggerScenarios":"A think_excluded_tokens entry that is an empty string or not representable in the tokenizer vocab, encountered while building ReasonerGrammarBackend.","commonSituations":"Custom/edited reasoning parser configs with placeholder or whitespace-only excluded tokens; tokenizer vocab missing special markers used by the parser.","solutions":["Print reasoning_parser.detector.think_excluded_tokens and tokenizer.encode each entry to find the offender","Remove or correct empty/invalid entries in the parser config","Ensure the tokenizer actually contains the excluded marker tokens"],"exampleFix":null,"handlingStrategy":"validation","validationCode":"for tok in parser.detector.think_excluded_tokens or []:\n    assert tokenizer.encode(tok, add_special_tokens=False), f'unencodable excluded token: {tok!r}'","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Validate excluded-token lists as part of reasoning-parser config loading","Avoid empty-string entries in token lists"],"tags":["reasoning","tokenizer","grammar"],"backgroundTag":"token-encoding-failed","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}