{"record":{"id":"fd6791998660215c","repo":"huggingface/transformers","slug":"stop-string-preprocessing-was-unable-to-identify-t","errorCode":null,"errorMessage":"Stop string preprocessing was unable to identify tokens matching one or more of the supplied stop string(s). This is most often caused by the stop strings containing unusual characters that are not in the tokenizer vocabulary.","messagePattern":"Stop string preprocessing was unable to identify tokens matching one or more of the supplied stop string\\(s\\)\\. This is most often caused by the stop strings containing unusual characters that are not in the tokenizer vocabulary\\.","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"src/transformers/generation/stopping_criteria.py","lineNumber":436,"sourceCode":"                    token_end_overlaps[stop_string][tok_idx] = possible_end_lengths\n        return token_valid_positions, token_end_overlaps\n\n    @staticmethod\n    def _stop_string_create_embedding_vec(token_list, token_indices, stop_strings) -> dict[str, torch.Tensor]:\n        \"\"\"This function precomputes everything needed for the run-time checks in StopStringCriteria, and packs\n        them into an embedding tensor that can be accessed with pure tensor operations. For the specifics of the values\n        that are precomputed and what they are used for, please refer to the StopStringCriteria docstring!\"\"\"\n        token_valid_positions, token_end_overlaps = StopStringCriteria._stop_string_get_matching_positions(\n            token_list, token_indices, stop_strings\n        )\n        all_valid_positions = [len(val) for positions in token_valid_positions.values() for val in positions.values()]\n        # In some cases, tokens may have no valid internal positions (such as single-character stop strings), so\n        # we need a fallback to handle this case\n        max_valid_positions = max(all_valid_positions) if all_valid_positions else 1\n        # There should always be at least one valid end_len, however, so no fallback needed here\n        valid_end_lens = [len(val) for positions in token_end_overlaps.values() for val in positions.values()]\n        if not valid_end_lens:\n            raise ValueError(\n                \"Stop string preprocessing was unable to identify tokens matching one or more of the \"\n                \"supplied stop string(s). This is most often caused by the stop \"\n                \"strings containing unusual characters that are not in the tokenizer vocabulary.\"\n            )\n        max_valid_end_lens = max(valid_end_lens)\n        vec_size = len(stop_strings) * (max_valid_positions + max_valid_end_lens) + 1\n        # We use +2 instead of +1 so we can have a dummy entry at the end. We will clamp all token values\n        # over the max to this, ensuring they do not contribute to stop string matching.\n        gather_vec = np.full((max(token_indices) + 2, vec_size), dtype=np.int32, fill_value=-1)\n\n        for i, stop_string in enumerate(stop_strings):\n            positions = token_valid_positions[stop_string]\n            end_lens = token_end_overlaps[stop_string]\n\n            # Since this is lots of very small assignments of lists, we build it with numpy rather\n            # than torch for speed + simplicity, then convert to torch at the end\n            for token_idx, valid_positions in positions.items():\n                gather_vec[token_idx, max_valid_positions * i : max_valid_positions * i + len(valid_positions)] = (","sourceCodeStart":418,"sourceCodeEnd":454,"githubUrl":"https://github.com/huggingface/transformers/blob/a597f974857b3d92939971296bc0deb93d33d780/src/transformers/generation/stopping_criteria.py#L418-L454","documentation":"Raised during StopStringCriteria preprocessing when no token in the vocabulary overlaps the end of any supplied stop string. The criteria precomputes, per token, which character positions could contribute to a stop-string suffix; if that map is empty, tensor-based matching is impossible and generation setup aborts. Typical root cause: stop strings contain characters (or unicode/normalization variants) that never appear in any token of the tokenizer vocabulary.","triggerScenarios":"Calling model.generate(..., stop_strings=\"→\") with a tokenizer whose tokens never contain that character; mixing NFC/NFD unicode normalization between stop string and tokenizer; whitespace-only stop strings for byte-level BPE where the pretokenizer splits them away; using stop strings in a language script the tokenizer does not cover.","commonSituations":"Multilingual apps stopping on CJK or emoji strings with an English tokenizer; copy-pasted smart quotes/en-dashes as stop strings; switching tokenizer without updating stop strings.","solutions":["Verify each stop string's characters actually occur in decoded vocabulary tokens; drop or rewrite strings that do not (e.g. replace smart quotes with ASCII).","Normalize the stop string the same way the tokenizer does (unicodedata.normalize('NFC', s)) before passing it.","Prefer simple ASCII phrases that are guaranteed tokenizable.","As a fallback use StoppingCriteria on decoded text via a custom subclass instead of stop_strings."],"exampleFix":"# before\nout = model.generate(**inputs, stop_strings=[\"\\u2014done\\u2014\"])  # em-dashes not in vocab\n\n# after\nout = model.generate(**inputs, stop_strings=[\"--done--\"])  # characters present in vocab","handlingStrategy":"validation","validationCode":"import unicodedata\n\ndef validate_stop_strings(stop_strings, tokenizer):\n    vocab_text = \"\".join(tokenizer.get_vocab().keys())\n    for s in stop_strings:\n        s = unicodedata.normalize(\"NFC\", s)\n        if not any(ch in vocab_text for ch in s):\n            raise ValueError(f\"stop string {s!r} has no characters present in the tokenizer vocabulary\")","typeGuard":null,"tryCatchPattern":"try:\n    model.generate(**inputs, stop_strings=stop_strings)\nexcept ValueError as e:\n    if \"stop string\" in str(e).lower():\n        stop_strings = [s for s in stop_strings if s.isascii()]\n        return model.generate(**inputs, stop_strings=stop_strings)\n    raise","preventionTips":["Keep stop strings ASCII/simple unless you verified the tokenizer covers the script.","Normalize unicode (NFC) on both stop strings and inputs.","Smoke-test stop_strings against tokenizer.get_vocab() at app startup."],"tags":["stop-strings","tokenizer-vocabulary","unicode","stopping-criteria","generation"],"backgroundTag":null,"analyzedSha":"a597f974857b3d92939971296bc0deb93d33d780","analyzedAt":"2026-08-14T18:24:08.354Z","schemaVersion":2},"datasetVersion":"2026-08-15T17:31:12.345Z"}