huggingface/tokenizers · error

NormalizedString bad split

Error message

NormalizedString bad split

What it means

An internal `expect()` panic in `NormalizedString::split`. After computing the split offsets, the library slices each kept segment out of the normalized string; each slice is expected to be a valid range within that string by construction. If the slicing fails, the split offsets were inconsistent with the string's internal transformation alignment, so the library deliberately panics with this message.

Solutions

1. Check your custom split/splitter implementation: every returned offset pair must be monotonically increasing and within `0..normalized_string.len()`; clamp and sort offsets before returning.
2. Reinstall/upgrade tokenizers to the latest patch release to rule out a known alignment bug.
3. Bisect the normalizer chain: temporarily remove custom or byte-level normalizers to identify which transform desynchronizes the offsets.
4. Report a minimal reproducer (normalizer + splitter + input) to the tokenizers repository if it reproduces on stock components.

Example fix

// before: custom splitter may return offsets beyond the string
Ok(vec![((start, end + 1), false)])

// after: clamp offsets to the normalized string bounds
let end = end.min(input.get().len());
Ok(vec![((start.min(end), end), false)])

def offsets_valid(offsets, length):
    prev = 0
    for (start, end), _ in offsets:
        if not (prev <= start <= end <= length):
            return False
        prev = start
    return True

try:
    splits = normalized_string.split(my_split_fn)
except Exception as e:
    raise ValueError("split returned out-of-range offsets; check custom splitter") from e

When it happens

Trigger: Calling `NormalizedString::split` (the public `split` wrapper) with a split function returning offsets that do not lie within the normalized string — normally only through a buggy custom split function passed to the normalizer, or an upstream bug in a built-in splitter.

Common situations: Seen when users implement custom pre-tokenizers/splitters whose returned offsets exceed the string length or are out of order, or when upgrading tokenizers and a released version has an alignment bug with a particular normalizer (e.g. byte-level + added tokens).

Understand the failure class

Background: "This is a bug, please report it": internal invariant violations, unreachable panics, and SNH errors explained — this error's family across 47 libraries.

Related errors

• likelihood is NAN. Input sentence may be too long.
• Uninitialized Encoding
• Cannot build Piece from string
• encode: `sequence` can't be `None`
• encode_batch: `inputs` can't be `None`

AI-assisted analysis of huggingface/tokenizers@6cfd9d385c (2026-09-09). Data as JSON: /api/errors/2b2455a83094d260. Report an issue: GitHub.

Appendix: source

Thrown at tokenizers/src/tokenizer/normalizer.rs:776

                            } else {
                                acc.push((offsets, false));
                            }
                            previous_match = is_match;
                            acc
                        });
                matches.reverse();
                matches
            }
        };

        // Then we split according to the computed splits
        Ok(splits
            .into_iter()
            .filter_map(|(offsets, remove)| {
                if !remove {
                    Some(
                        self.slice(Range::Normalized(offsets.0..offsets.1))
                            .expect("NormalizedString bad split"),
                    )
                } else {
                    None
                }
            })
            .collect())
    }

    /// Remove any leading space(s) of the normalized string
    pub fn lstrip(&mut self) -> &mut Self {
        self.lrstrip(true, false)
    }

    /// Remove any trailing space(s) of the normalized string
    pub fn rstrip(&mut self) -> &mut Self {
        self.lrstrip(false, true)
    }

View on GitHub (pinned to 6cfd9d385c)