huggingface/transformers · error · ValueError

Stop string preprocessing was unable to identify tokens matc

Error message

Stop string preprocessing was unable to identify tokens matching one or more of the supplied stop string(s). This is most often caused by the stop strings containing unusual characters that are not in the tokenizer vocabulary.

What it means

Raised during StopStringCriteria preprocessing when no token in the vocabulary overlaps the end of any supplied stop string. The criteria precomputes, per token, which character positions could contribute to a stop-string suffix; if that map is empty, tensor-based matching is impossible and generation setup aborts. Typical root cause: stop strings contain characters (or unicode/normalization variants) that never appear in any token of the tokenizer vocabulary.

Source

Thrown at src/transformers/generation/stopping_criteria.py:436

                    token_end_overlaps[stop_string][tok_idx] = possible_end_lengths
        return token_valid_positions, token_end_overlaps

    @staticmethod
    def _stop_string_create_embedding_vec(token_list, token_indices, stop_strings) -> dict[str, torch.Tensor]:
        """This function precomputes everything needed for the run-time checks in StopStringCriteria, and packs
        them into an embedding tensor that can be accessed with pure tensor operations. For the specifics of the values
        that are precomputed and what they are used for, please refer to the StopStringCriteria docstring!"""
        token_valid_positions, token_end_overlaps = StopStringCriteria._stop_string_get_matching_positions(
            token_list, token_indices, stop_strings
        )
        all_valid_positions = [len(val) for positions in token_valid_positions.values() for val in positions.values()]
        # In some cases, tokens may have no valid internal positions (such as single-character stop strings), so
        # we need a fallback to handle this case
        max_valid_positions = max(all_valid_positions) if all_valid_positions else 1
        # There should always be at least one valid end_len, however, so no fallback needed here
        valid_end_lens = [len(val) for positions in token_end_overlaps.values() for val in positions.values()]
        if not valid_end_lens:
            raise ValueError(
                "Stop string preprocessing was unable to identify tokens matching one or more of the "
                "supplied stop string(s). This is most often caused by the stop "
                "strings containing unusual characters that are not in the tokenizer vocabulary."
            )
        max_valid_end_lens = max(valid_end_lens)
        vec_size = len(stop_strings) * (max_valid_positions + max_valid_end_lens) + 1
        # We use +2 instead of +1 so we can have a dummy entry at the end. We will clamp all token values
        # over the max to this, ensuring they do not contribute to stop string matching.
        gather_vec = np.full((max(token_indices) + 2, vec_size), dtype=np.int32, fill_value=-1)

        for i, stop_string in enumerate(stop_strings):
            positions = token_valid_positions[stop_string]
            end_lens = token_end_overlaps[stop_string]

            # Since this is lots of very small assignments of lists, we build it with numpy rather
            # than torch for speed + simplicity, then convert to torch at the end
            for token_idx, valid_positions in positions.items():
                gather_vec[token_idx, max_valid_positions * i : max_valid_positions * i + len(valid_positions)] = (

View on GitHub (pinned to a597f97485)

Solutions

1. Verify each stop string's characters actually occur in decoded vocabulary tokens; drop or rewrite strings that do not (e.g. replace smart quotes with ASCII).
2. Normalize the stop string the same way the tokenizer does (unicodedata.normalize('NFC', s)) before passing it.
3. Prefer simple ASCII phrases that are guaranteed tokenizable.
4. As a fallback use StoppingCriteria on decoded text via a custom subclass instead of stop_strings.

Example fix

# before
out = model.generate(**inputs, stop_strings=["\u2014done\u2014"])  # em-dashes not in vocab

# after
out = model.generate(**inputs, stop_strings=["--done--"])  # characters present in vocab

import unicodedata

def validate_stop_strings(stop_strings, tokenizer):
    vocab_text = "".join(tokenizer.get_vocab().keys())
    for s in stop_strings:
        s = unicodedata.normalize("NFC", s)
        if not any(ch in vocab_text for ch in s):
            raise ValueError(f"stop string {s!r} has no characters present in the tokenizer vocabulary")

try:
    model.generate(**inputs, stop_strings=stop_strings)
except ValueError as e:
    if "stop string" in str(e).lower():
        stop_strings = [s for s in stop_strings if s.isascii()]
        return model.generate(**inputs, stop_strings=stop_strings)
    raise

When it happens

Trigger: Calling model.generate(..., stop_strings="→") with a tokenizer whose tokens never contain that character; mixing NFC/NFD unicode normalization between stop string and tokenizer; whitespace-only stop strings for byte-level BPE where the pretokenizer splits them away; using stop strings in a language script the tokenizer does not cover.

Common situations: Multilingual apps stopping on CJK or emoji strings with an English tokenizer; copy-pasted smart quotes/en-dashes as stop strings; switching tokenizer without updating stop strings.

Related errors

• There are one or more stop strings, either in the arguments
• StoppingCriteria needs to be subclassed
• stopping_criteria is not supported for continuous batching.
• `crop` was called, but the current layer does not track past
• Once the sliding window size has been reached, `DynamicSlidi

AI-assisted analysis of huggingface/transformers@a597f97485 (2026-08-14). Data as JSON: /api/errors/fd6791998660215c. Report an issue: GitHub.