huggingface/tokenizers · error

AddedVocabulary bad split

Error message

AddedVocabulary bad split

What it means

split_with_indices splits the input sentence on added-token matches found via the split trie, slicing the NormalizedString at the match byte offsets. It .expect("AddedVocabulary bad split") panics when those offsets don't form a valid slice of the NormalizedString. This is an internal invariant: offsets come from find_matches over the same string, so a mismatch means offsets and string content are out of sync.

Solutions

1. Register added tokens with add_tokens/add_special_tokens so normalization handling is set correctly (especially normalized: false for special tokens), and re-register if the normalizer changed afterwards.
2. Avoid adding tokens that contain characters your normalizer removes or rewrites; align token contents with the normalized form.
3. Minimize the failing input and report upstream — a mismatch between find_matches offsets and the string is a library bug if the standard API is used.

Example fix

// before — token content altered by normalizer
tokenizer.add_tokens([AddedToken("ＦＵＬＬＷＩＤＴＨ")]); // normalizer NFKC-strips it -> offsets desync

// after — use add_special_tokens / normalized:false or matching content
tokenizer.add_special_tokens(["[SPECIAL]"]);

// ensure added tokens won't be rewritten by the normalizer
for (const t of addedTokens) {
  if (normalizerStrips(t.content)) throw new Error(`added token '${t.content}' conflicts with normalizer; use add_special_tokens or normalized:false`);
}

try { return tokenizer.encode(text); } catch (e) { if (String(e).includes('bad split')) { rebuildAddedVocabulary(); return tokenizer.encode(text); } throw e; }

When it happens

Trigger: extract_and_normalize processing text where the added-token trie match byte offsets fall outside the NormalizedString or on invalid boundaries — e.g. after added special tokens were registered containing characters altered/removed by normalization, so matched offsets in the original string no longer slice cleanly.

Common situations: Adding tokens whose content includes characters that the tokenizer's normalizer strips or changes (then encoding text containing them); registering added tokens after building offsets incorrectly; custom added-vocabulary manipulation in a fork.

Understand the failure class

Background: "This is a bug, please report it": internal invariant violations, unreachable panics, and SNH errors explained — this error's family across 47 libraries.

Related errors

• NormalizedString bad split
• Helper
• sep_token not found in the vocabulary
• cls_token not found in the vocabulary
• likelihood is NAN. Input sentence may be too long.

AI-assisted analysis of huggingface/tokenizers@6cfd9d385c (2026-09-09). Data as JSON: /api/errors/ab5542ea8db05681. Report an issue: GitHub.

Appendix: source

Thrown at tokenizers/src/tokenizer/added_vocabulary.rs:505

        }

        splits
    }

    /// Split the input sentence to extract anything we found from the `MatchingSet`, as well as
    /// the list of corresponding IDs
    /// The list of IDs have the exact same number of elements than the Iterator.
    fn split_with_indices(
        &self,
        sentence: NormalizedString,
        split_re: &MatchingSet,
    ) -> Vec<(NormalizedString, Option<Vec<Token>>)> {
        self.find_matches(sentence.get(), split_re)
            .into_iter()
            .map(|(id, byte_offsets)| {
                let slice = sentence
                    .slice(Range::Normalized(byte_offsets.0..byte_offsets.1))
                    .expect("AddedVocabulary bad split");
                if let Some(id) = id {
                    let value = slice.get().to_owned();
                    let len = value.len();
                    (slice, Some(vec![Token::new(id, value, (0, len))]))
                } else {
                    (slice, None)
                }
            })
            .collect()
    }

    /// Extract the additional vocabulary from the given sentence, normalizing it along the way.
    ///
    /// Some tokens should match against their normalized representation, as well as the
    /// non-normalized one. For example, when we expect to extract the token `yesterday` in the
    /// input sentence `I read a book Yesterday`, if the normalizer is supposed to lowercase
    /// everything, we expect a match.
    pub fn extract_and_normalize<N: Normalizer>(

View on GitHub (pinned to 6cfd9d385c)