{"record":{"id":"ab5542ea8db05681","repo":"huggingface/tokenizers","slug":"addedvocabulary-bad-split","errorCode":null,"errorMessage":"AddedVocabulary bad split","messagePattern":"AddedVocabulary bad split","errorType":"panic","errorClass":null,"httpStatus":null,"severity":"error","filePath":"tokenizers/src/tokenizer/added_vocabulary.rs","lineNumber":505,"sourceCode":"        }\n\n        splits\n    }\n\n    /// Split the input sentence to extract anything we found from the `MatchingSet`, as well as\n    /// the list of corresponding IDs\n    /// The list of IDs have the exact same number of elements than the Iterator.\n    fn split_with_indices(\n        &self,\n        sentence: NormalizedString,\n        split_re: &MatchingSet,\n    ) -> Vec<(NormalizedString, Option<Vec<Token>>)> {\n        self.find_matches(sentence.get(), split_re)\n            .into_iter()\n            .map(|(id, byte_offsets)| {\n                let slice = sentence\n                    .slice(Range::Normalized(byte_offsets.0..byte_offsets.1))\n                    .expect(\"AddedVocabulary bad split\");\n                if let Some(id) = id {\n                    let value = slice.get().to_owned();\n                    let len = value.len();\n                    (slice, Some(vec![Token::new(id, value, (0, len))]))\n                } else {\n                    (slice, None)\n                }\n            })\n            .collect()\n    }\n\n    /// Extract the additional vocabulary from the given sentence, normalizing it along the way.\n    ///\n    /// Some tokens should match against their normalized representation, as well as the\n    /// non-normalized one. For example, when we expect to extract the token `yesterday` in the\n    /// input sentence `I read a book Yesterday`, if the normalizer is supposed to lowercase\n    /// everything, we expect a match.\n    pub fn extract_and_normalize<N: Normalizer>(","sourceCodeStart":487,"sourceCodeEnd":523,"githubUrl":"https://github.com/huggingface/tokenizers/blob/6cfd9d385ca0ed91c10b49f0ce97d02cfde1b607/tokenizers/src/tokenizer/added_vocabulary.rs#L487-L523","documentation":"split_with_indices splits the input sentence on added-token matches found via the split trie, slicing the NormalizedString at the match byte offsets. It .expect(\"AddedVocabulary bad split\") panics when those offsets don't form a valid slice of the NormalizedString. This is an internal invariant: offsets come from find_matches over the same string, so a mismatch means offsets and string content are out of sync.","triggerScenarios":"extract_and_normalize processing text where the added-token trie match byte offsets fall outside the NormalizedString or on invalid boundaries — e.g. after added special tokens were registered containing characters altered/removed by normalization, so matched offsets in the original string no longer slice cleanly.","commonSituations":"Adding tokens whose content includes characters that the tokenizer's normalizer strips or changes (then encoding text containing them); registering added tokens after building offsets incorrectly; custom added-vocabulary manipulation in a fork.","solutions":["Register added tokens with add_tokens/add_special_tokens so normalization handling is set correctly (especially normalized: false for special tokens), and re-register if the normalizer changed afterwards.","Avoid adding tokens that contain characters your normalizer removes or rewrites; align token contents with the normalized form.","Minimize the failing input and report upstream — a mismatch between find_matches offsets and the string is a library bug if the standard API is used."],"exampleFix":"// before — token content altered by normalizer\ntokenizer.add_tokens([AddedToken(\"ＦＵＬＬＷＩＤＴＨ\")]); // normalizer NFKC-strips it -> offsets desync\n\n// after — use add_special_tokens / normalized:false or matching content\ntokenizer.add_special_tokens([\"[SPECIAL]\"]);","handlingStrategy":"validation","validationCode":"// ensure added tokens won't be rewritten by the normalizer\nfor (const t of addedTokens) {\n  if (normalizerStrips(t.content)) throw new Error(`added token '${t.content}' conflicts with normalizer; use add_special_tokens or normalized:false`);\n}","typeGuard":null,"tryCatchPattern":"try { return tokenizer.encode(text); } catch (e) { if (String(e).includes('bad split')) { rebuildAddedVocabulary(); return tokenizer.encode(text); } throw e; }","preventionTips":["Register added tokens via add_tokens/add_special_tokens with correct normalized flags","Avoid added tokens containing characters your normalizer alters","Re-check the added vocabulary after any normalizer change"],"tags":["rust","added-tokens","normalization","internal-invariant","tokenizer"],"backgroundTag":"internal-invariant-violation","analyzedSha":"6cfd9d385ca0ed91c10b49f0ce97d02cfde1b607","analyzedAt":"2026-09-09T11:43:25.027Z","contentChangedAt":"2026-09-09T11:43:25.027Z","schemaVersion":2},"datasetVersion":"2026-09-16T04:17:20.429Z"}