huggingface/tokenizers · error

NormalizedString bad split

Error message

NormalizedString bad split

What it means

UnicodeScripts pre-tokenization splits the NormalizedString at script-boundary offsets computed from the script segmentation. It then slices the string at those ranges and .expect("NormalizedString bad split") panics if any slice is invalid (offsets not on valid boundaries or out of order). The library throws it as an internal invariant: the ranges it just computed from the string's own script map should always be valid slice boundaries.

Solutions

1. Use the standard pipeline untouched: ensure text goes through the tokenizer's own normalization before unicode_scripts pre-tokenization rather than injecting pre-transformed strings.
2. Reproduce with a minimal input to identify the offending characters; if specific Unicode characters trigger it, report the bug upstream with the exact input.
3. As a workaround, remove the UnicodeScripts pre-tokenizer or replace it with a whitespace Metaspace/Whitespace pre-tokenizer if script splitting isn't required.

Example fix

// before (custom pipeline)
let norm = custom_normalizer.normalize(...);
scripts.pre_tokenize(norm); // may panic: bad split

// after — let the tokenizer own normalization/offsets
let enc = tokenizer.with_pre_tokenizer(pre_tokenizers.unicode_scripts()).encode(text);

// wrap tokenization of suspect mixed-script input
try { return tokenizer.encode(text); } catch (e) { if (String(e).includes('bad split')) { return fallbackWhitespaceTokenizer.encode(text); } throw e; }

When it happens

Trigger: Pre-tokenizing (basic or spaces_are_included_in_every_script path) a normalized string whose byte offsets from the script ranges don't align with NormalizedString slice boundaries — typically indicates corrupted string state or an offset computation bug, e.g. when earlier transformations shifted offsets.

Common situations: Encoding text with mixed scripts after a custom normalization or pre-tokenization step that invalidated offset tracking; using a modified/forked pipeline; rare Unicode edge cases (unassigned or unusual script characters) triggering segmentation edge behavior.

Understand the failure class

Background: "This is a bug, please report it": internal invariant violations, unreachable panics, and SNH errors explained — this error's family across 47 libraries.

Related errors

• AddedVocabulary bad split
• Helper
• sep_token not found in the vocabulary
• cls_token not found in the vocabulary
• likelihood is NAN. Input sentence may be too long.

AI-assisted analysis of huggingface/tokenizers@6cfd9d385c (2026-09-09). Data as JSON: /api/errors/e1ee5f041e305cd9. Report an issue: GitHub.

Appendix: source

Thrown at tokenizers/src/pre_tokenizers/unicode_scripts/pre_tokenizer.rs:72

                        Some(offset)
                    } else {
                        None
                    };
                    offset += c.len_utf8();
                    if script != Some(Script::Any) {
                        last_script = script;
                    }

                    result
                })
                .collect();
            ranges.push(normalized.get().len());
            Ok(ranges
                .windows(2)
                .map(|item| {
                    normalized
                        .slice(Range::Normalized(item[0]..item[1]))
                        .expect("NormalizedString bad split")
                })
                .collect::<Vec<_>>())
        })
    }
}

#[cfg(test)]
mod tests {
    use super::*;
    use crate::OffsetReferential;
    use crate::OffsetType;

    #[test]
    fn basic() {
        let pretok = UnicodeScripts {};
        let mut pretokenized = PreTokenizedString::from("どこで生れ。Yes");
        pretok.pre_tokenize(&mut pretokenized).unwrap();
        assert_eq!(

View on GitHub (pinned to 6cfd9d385c)