huggingface/tokenizers · error

likelihood is NAN. Input sentence may be too long.

Error message

likelihood is NAN. Input sentence may be too long.

What it means

During Unigram model training, run_e_step computes the marginal likelihood z of each sentence's lattice via populate_marginal. If z is NaN — a numerical overflow/underflow caused by extremely long sentences producing probabilities that exceed f64's usable range — training panics with this message rather than continuing with corrupted statistics.

Solutions

1. Pre-split the corpus into shorter sentences (e.g. split on newlines, sentence tokenization, or fixed-length chunks) before training.
2. Cap input length by truncating or chunking long lines to a few thousand characters before passing sentences to the trainer.
3. Inspect the corpus for pathological entries (no whitespace/punctuation for megabytes) and clean or filter them.
4. If it persists, retry with a smaller seed sentence size / limit exhaustive training parameters that increase lattice size.

Example fix

// before
trainer = UnigramTrainer(vocab_size=80000, special_tokens=["<unk>"])
trainer.train(files=["raw_dump.txt"])  # panic: likelihood is NAN
// after
text = open("raw_dump.txt").read()
sentences = [s for s in text.split("\n") if s]
sentences = [s[:4000] for s in sentences]  # cap pathological lengths
from tokenizers import Tokenizer, models, trainers
# train on the pre-split sentences (train_from_iterator)

sentences = [s for s in corpus if s.strip()]
MAX_CHARS = 4000
if any(len(s) > MAX_CHARS for s in sentences):
    sentences = [s[:MAX_CHARS] for s in sentences]  # cap pathological lengths
trainer.train_from_iterator(sentences)

def is_trainable_sentence(s: str, max_chars: int = 4000) -> bool:
    return 0 < len(s) <= max_chars

# panic, not exception: run training in a subprocess and detect non-zero exit
proc = subprocess.run([sys.executable, "train.py", corpus_path])
if proc.returncode != 0 and b"likelihood is NAN" in proc.stderr:
    chunk_corpus_and_retry(corpus_path)

When it happens

Trigger: Calling `UnigramTrainer.train()` (via do_train) on a corpus containing sentences long enough that lattice marginal computation overflows to NaN, typically with a large chunk of very long concatenated text (unsplit documents, log lines, or text without sentence boundaries).

Common situations: Training on raw scraped text, minified files, or corpora where newlines/sentence delimiters were stripped, producing multi-hundred-thousand-character 'sentences'; very long sequences interacting with tiny subword probabilities.

Understand the failure class

Background: Tensor shape mismatch errors ("must have shape", "expected shape ... got ..."): when tensor dimensions disagree with what an op or layer was told to expect — this error's family across 6 libraries.

Related errors

• Uninitialized Encoding
• NormalizedString bad split
• You're trying to run a `Unigram` model but you're file was…
• Uninitialized Model
• Helper

AI-assisted analysis of huggingface/tokenizers@6cfd9d385c (2026-09-09). Data as JSON: /api/errors/2010d09a254bafb2. Report an issue: GitHub.

Appendix: source

Thrown at tokenizers/src/models/unigram/trainer.rs:468

    fn run_e_step(&self, model: &Unigram, sentences: &[Sentence]) -> (f64, u32, Vec<f64>) {
        let all_sentence_freq: u32 = sentences.iter().map(|(_a, b)| *b).sum();

        let chunk_size = std::cmp::max(sentences.len() / current_num_threads(), 1);
        let collected: (f64, u32, Vec<f64>) = sentences
            .maybe_par_chunks(chunk_size)
            .map(|sentences_chunk| {
                let mut expected: Vec<f64> = vec![0.0; model.len()];
                let mut objs: f64 = 0.0;
                let mut ntokens: u32 = 0;

                for (string, freq) in sentences_chunk {
                    let mut lattice = Lattice::from(string, model.bos_id, model.eos_id);
                    model.populate_nodes(&mut lattice);

                    let z: f64 = lattice.populate_marginal(*freq as f64, &mut expected);
                    if z.is_nan() {
                        panic!("likelihood is NAN. Input sentence may be too long.");
                    }
                    ntokens += lattice.viterbi().len() as u32;
                    objs -= z / (all_sentence_freq as f64);
                }
                (objs, ntokens, expected)
            })
            .reduce(
                || (0.0, 0, vec![0.0; model.len()]),
                |(objs, ntokens, expected), (lobjs, lntokens, lexpected)| {
                    (
                        objs + lobjs,
                        ntokens + lntokens,
                        expected
                            .iter()
                            .zip(lexpected)
                            .map(|(global_el, local_el)| global_el + local_el)
                            .collect(),
                    )

View on GitHub (pinned to 6cfd9d385c)