{"record":{"id":"e1ee5f041e305cd9","repo":"huggingface/tokenizers","slug":"normalizedstring-bad-split","errorCode":null,"errorMessage":"NormalizedString bad split","messagePattern":"NormalizedString bad split","errorType":"panic","errorClass":null,"httpStatus":null,"severity":"error","filePath":"tokenizers/src/pre_tokenizers/unicode_scripts/pre_tokenizer.rs","lineNumber":72,"sourceCode":"                        Some(offset)\n                    } else {\n                        None\n                    };\n                    offset += c.len_utf8();\n                    if script != Some(Script::Any) {\n                        last_script = script;\n                    }\n\n                    result\n                })\n                .collect();\n            ranges.push(normalized.get().len());\n            Ok(ranges\n                .windows(2)\n                .map(|item| {\n                    normalized\n                        .slice(Range::Normalized(item[0]..item[1]))\n                        .expect(\"NormalizedString bad split\")\n                })\n                .collect::<Vec<_>>())\n        })\n    }\n}\n\n#[cfg(test)]\nmod tests {\n    use super::*;\n    use crate::OffsetReferential;\n    use crate::OffsetType;\n\n    #[test]\n    fn basic() {\n        let pretok = UnicodeScripts {};\n        let mut pretokenized = PreTokenizedString::from(\"どこで生れ。Yes\");\n        pretok.pre_tokenize(&mut pretokenized).unwrap();\n        assert_eq!(","sourceCodeStart":54,"sourceCodeEnd":90,"githubUrl":"https://github.com/huggingface/tokenizers/blob/6cfd9d385ca0ed91c10b49f0ce97d02cfde1b607/tokenizers/src/pre_tokenizers/unicode_scripts/pre_tokenizer.rs#L54-L90","documentation":"UnicodeScripts pre-tokenization splits the NormalizedString at script-boundary offsets computed from the script segmentation. It then slices the string at those ranges and .expect(\"NormalizedString bad split\") panics if any slice is invalid (offsets not on valid boundaries or out of order). The library throws it as an internal invariant: the ranges it just computed from the string's own script map should always be valid slice boundaries.","triggerScenarios":"Pre-tokenizing (basic or spaces_are_included_in_every_script path) a normalized string whose byte offsets from the script ranges don't align with NormalizedString slice boundaries — typically indicates corrupted string state or an offset computation bug, e.g. when earlier transformations shifted offsets.","commonSituations":"Encoding text with mixed scripts after a custom normalization or pre-tokenization step that invalidated offset tracking; using a modified/forked pipeline; rare Unicode edge cases (unassigned or unusual script characters) triggering segmentation edge behavior.","solutions":["Use the standard pipeline untouched: ensure text goes through the tokenizer's own normalization before unicode_scripts pre-tokenization rather than injecting pre-transformed strings.","Reproduce with a minimal input to identify the offending characters; if specific Unicode characters trigger it, report the bug upstream with the exact input.","As a workaround, remove the UnicodeScripts pre-tokenizer or replace it with a whitespace Metaspace/Whitespace pre-tokenizer if script splitting isn't required."],"exampleFix":"// before (custom pipeline)\nlet norm = custom_normalizer.normalize(...);\nscripts.pre_tokenize(norm); // may panic: bad split\n\n// after — let the tokenizer own normalization/offsets\nlet enc = tokenizer.with_pre_tokenizer(pre_tokenizers.unicode_scripts()).encode(text);","handlingStrategy":"fallback","validationCode":null,"typeGuard":null,"tryCatchPattern":"// wrap tokenization of suspect mixed-script input\ntry { return tokenizer.encode(text); } catch (e) { if (String(e).includes('bad split')) { return fallbackWhitespaceTokenizer.encode(text); } throw e; }","preventionTips":["Do not inject externally normalized strings into the pipeline; let the tokenizer own offsets","Keep the pre-tokenizer/normalizer chain unmodified unless you understand offset tracking","Report minimal reproductions of Unicode edge cases upstream"],"tags":["rust","unicode","pre-tokenization","internal-invariant","tokenizer"],"backgroundTag":"internal-invariant-violation","analyzedSha":"6cfd9d385ca0ed91c10b49f0ce97d02cfde1b607","analyzedAt":"2026-09-09T11:43:25.027Z","contentChangedAt":"2026-09-09T11:43:25.027Z","schemaVersion":2},"datasetVersion":"2026-09-16T04:17:20.429Z"}