{"record":{"id":"2b2455a83094d260","repo":"huggingface/tokenizers","slug":"normalizedstring-bad-split-2b2455","errorCode":null,"errorMessage":"NormalizedString bad split","messagePattern":"NormalizedString bad split","errorType":"panic","errorClass":null,"httpStatus":null,"severity":"error","filePath":"tokenizers/src/tokenizer/normalizer.rs","lineNumber":776,"sourceCode":"                            } else {\n                                acc.push((offsets, false));\n                            }\n                            previous_match = is_match;\n                            acc\n                        });\n                matches.reverse();\n                matches\n            }\n        };\n\n        // Then we split according to the computed splits\n        Ok(splits\n            .into_iter()\n            .filter_map(|(offsets, remove)| {\n                if !remove {\n                    Some(\n                        self.slice(Range::Normalized(offsets.0..offsets.1))\n                            .expect(\"NormalizedString bad split\"),\n                    )\n                } else {\n                    None\n                }\n            })\n            .collect())\n    }\n\n    /// Remove any leading space(s) of the normalized string\n    pub fn lstrip(&mut self) -> &mut Self {\n        self.lrstrip(true, false)\n    }\n\n    /// Remove any trailing space(s) of the normalized string\n    pub fn rstrip(&mut self) -> &mut Self {\n        self.lrstrip(false, true)\n    }\n","sourceCodeStart":758,"sourceCodeEnd":794,"githubUrl":"https://github.com/huggingface/tokenizers/blob/6cfd9d385ca0ed91c10b49f0ce97d02cfde1b607/tokenizers/src/tokenizer/normalizer.rs#L758-L794","documentation":"An internal `expect()` panic in `NormalizedString::split`. After computing the split offsets, the library slices each kept segment out of the normalized string; each slice is expected to be a valid range within that string by construction. If the slicing fails, the split offsets were inconsistent with the string's internal transformation alignment, so the library deliberately panics with this message.","triggerScenarios":"Calling `NormalizedString::split` (the public `split` wrapper) with a split function returning offsets that do not lie within the normalized string — normally only through a buggy custom split function passed to the normalizer, or an upstream bug in a built-in splitter.","commonSituations":"Seen when users implement custom pre-tokenizers/splitters whose returned offsets exceed the string length or are out of order, or when upgrading tokenizers and a released version has an alignment bug with a particular normalizer (e.g. byte-level + added tokens).","solutions":["Check your custom split/splitter implementation: every returned offset pair must be monotonically increasing and within `0..normalized_string.len()`; clamp and sort offsets before returning.","Reinstall/upgrade tokenizers to the latest patch release to rule out a known alignment bug.","Bisect the normalizer chain: temporarily remove custom or byte-level normalizers to identify which transform desynchronizes the offsets.","Report a minimal reproducer (normalizer + splitter + input) to the tokenizers repository if it reproduces on stock components."],"exampleFix":"// before: custom splitter may return offsets beyond the string\nOk(vec![((start, end + 1), false)])\n\n// after: clamp offsets to the normalized string bounds\nlet end = end.min(input.get().len());\nOk(vec![((start.min(end), end), false)])","handlingStrategy":"validation","validationCode":"def offsets_valid(offsets, length):\n    prev = 0\n    for (start, end), _ in offsets:\n        if not (prev <= start <= end <= length):\n            return False\n        prev = start\n    return True","typeGuard":null,"tryCatchPattern":"try:\n    splits = normalized_string.split(my_split_fn)\nexcept Exception as e:\n    raise ValueError(\"split returned out-of-range offsets; check custom splitter\") from e","preventionTips":["Ensure custom split functions return sorted, non-overlapping offsets bounded by the normalized string length.","Account for byte-length vs char-length when the string contains multi-byte characters.","Write unit tests for your splitter against strings with unicode, emoji, and empty segments.","Keep the tokenizers package updated; some offset-alignment bugs are fixed in patch releases."],"tags":["rust","tokenizers","panic","offsets"],"backgroundTag":"internal-invariant-violation","analyzedSha":"6cfd9d385ca0ed91c10b49f0ce97d02cfde1b607","analyzedAt":"2026-09-09T11:43:25.027Z","contentChangedAt":"2026-09-09T11:43:25.027Z","schemaVersion":2},"datasetVersion":"2026-09-16T04:17:20.429Z"}