sgl-project/sglang · error · ValueError

think_excluded_token '{token}' could not be encoded by the t

Error message

think_excluded_token '{token}' could not be encoded by the tokenizer. All excluded tokens must be encodable for strict reasoning mode to function correctly.

What it means

Each token in reasoning_parser.detector.think_excluded_tokens must encode to at least one tokenizer ID for strict reasoning mode; _get_think_excluded_token_ids raises ValueError for any token that encodes to nothing.

Source

Thrown at python/sglang/srt/constrained/reasoner_grammar_backend.py:296

            )
        self._token_filter_fn = (
            self.grammar_backend.set_token_filter if self.enable_token_filter else None
        )

    def _get_think_excluded_token_ids(
        self,
        reasoning_parser: ReasoningParser,
        tokenizer: Union[PreTrainedTokenizer, PreTrainedTokenizerFast],
    ) -> Optional[List[int]]:
        excluded_ids = []
        if (not self.enable_strict_thinking) or (
            not reasoning_parser.detector.think_excluded_tokens
        ):
            return None
        for token in reasoning_parser.detector.think_excluded_tokens:
            new_ids = tokenizer.encode(token, add_special_tokens=False)
            if not new_ids:
                raise ValueError(
                    f"think_excluded_token '{token}' could not be encoded by the "
                    f"tokenizer. All excluded tokens must be encodable for strict "
                    f"reasoning mode to function correctly."
                )
            excluded_ids += new_ids
        return excluded_ids

    def _make_grammar_object(
        self, grammar: Optional[BaseGrammarObject], reasoning: bool
    ) -> ReasonerGrammarObject:
        obj = ReasonerGrammarObject(
            grammar=grammar,
            think_end_ids=self.think_end_ids,
            think_excluded_token_ids=self.think_excluded_token_ids,
            max_think_tokens=self.max_think_tokens,
            enable_token_filter=self.enable_token_filter,
            token_filter_fn=self._token_filter_fn,
            allocate_vocab_mask_fn=self.grammar_backend.allocate_vocab_mask,

View on GitHub (pinned to 0132848349)

Solutions

1. Print reasoning_parser.detector.think_excluded_tokens and tokenizer.encode each entry to find the offender
2. Remove or correct empty/invalid entries in the parser config
3. Ensure the tokenizer actually contains the excluded marker tokens

for tok in parser.detector.think_excluded_tokens or []:
    assert tokenizer.encode(tok, add_special_tokens=False), f'unencodable excluded token: {tok!r}'

When it happens

Trigger: A think_excluded_tokens entry that is an empty string or not representable in the tokenizer vocab, encountered while building ReasonerGrammarBackend.

Common situations: Custom/edited reasoning parser configs with placeholder or whitespace-only excluded tokens; tokenizer vocab missing special markers used by the parser.

Related errors

• think_end_token '{reasoning_parser.detector.think_end_token}
• Strict reasoning format requested but the grammar backend do
• name must be provided
• Cannot load PE model: 'model_max_length' not found in {os.pa
• Cosmos3TokenizationStage requires a tokenizer; expected the

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/09255a5f3b137542. Report an issue: GitHub.