{"record":{"id":"2010d09a254bafb2","repo":"huggingface/tokenizers","slug":"likelihood-is-nan-input-sentence-may-be-too-long","errorCode":null,"errorMessage":"likelihood is NAN. Input sentence may be too long.","messagePattern":"likelihood is NAN\\. Input sentence may be too long\\.","errorType":"panic","errorClass":null,"httpStatus":null,"severity":"error","filePath":"tokenizers/src/models/unigram/trainer.rs","lineNumber":468,"sourceCode":"\n    fn run_e_step(&self, model: &Unigram, sentences: &[Sentence]) -> (f64, u32, Vec<f64>) {\n        let all_sentence_freq: u32 = sentences.iter().map(|(_a, b)| *b).sum();\n\n        let chunk_size = std::cmp::max(sentences.len() / current_num_threads(), 1);\n        let collected: (f64, u32, Vec<f64>) = sentences\n            .maybe_par_chunks(chunk_size)\n            .map(|sentences_chunk| {\n                let mut expected: Vec<f64> = vec![0.0; model.len()];\n                let mut objs: f64 = 0.0;\n                let mut ntokens: u32 = 0;\n\n                for (string, freq) in sentences_chunk {\n                    let mut lattice = Lattice::from(string, model.bos_id, model.eos_id);\n                    model.populate_nodes(&mut lattice);\n\n                    let z: f64 = lattice.populate_marginal(*freq as f64, &mut expected);\n                    if z.is_nan() {\n                        panic!(\"likelihood is NAN. Input sentence may be too long.\");\n                    }\n                    ntokens += lattice.viterbi().len() as u32;\n                    objs -= z / (all_sentence_freq as f64);\n                }\n                (objs, ntokens, expected)\n            })\n            .reduce(\n                || (0.0, 0, vec![0.0; model.len()]),\n                |(objs, ntokens, expected), (lobjs, lntokens, lexpected)| {\n                    (\n                        objs + lobjs,\n                        ntokens + lntokens,\n                        expected\n                            .iter()\n                            .zip(lexpected)\n                            .map(|(global_el, local_el)| global_el + local_el)\n                            .collect(),\n                    )","sourceCodeStart":450,"sourceCodeEnd":486,"githubUrl":"https://github.com/huggingface/tokenizers/blob/6cfd9d385ca0ed91c10b49f0ce97d02cfde1b607/tokenizers/src/models/unigram/trainer.rs#L450-L486","documentation":"During Unigram model training, run_e_step computes the marginal likelihood z of each sentence's lattice via populate_marginal. If z is NaN — a numerical overflow/underflow caused by extremely long sentences producing probabilities that exceed f64's usable range — training panics with this message rather than continuing with corrupted statistics.","triggerScenarios":"Calling `UnigramTrainer.train()` (via do_train) on a corpus containing sentences long enough that lattice marginal computation overflows to NaN, typically with a large chunk of very long concatenated text (unsplit documents, log lines, or text without sentence boundaries).","commonSituations":"Training on raw scraped text, minified files, or corpora where newlines/sentence delimiters were stripped, producing multi-hundred-thousand-character 'sentences'; very long sequences interacting with tiny subword probabilities.","solutions":["Pre-split the corpus into shorter sentences (e.g. split on newlines, sentence tokenization, or fixed-length chunks) before training.","Cap input length by truncating or chunking long lines to a few thousand characters before passing sentences to the trainer.","Inspect the corpus for pathological entries (no whitespace/punctuation for megabytes) and clean or filter them.","If it persists, retry with a smaller seed sentence size / limit exhaustive training parameters that increase lattice size."],"exampleFix":"// before\ntrainer = UnigramTrainer(vocab_size=80000, special_tokens=[\"<unk>\"])\ntrainer.train(files=[\"raw_dump.txt\"])  # panic: likelihood is NAN\n// after\ntext = open(\"raw_dump.txt\").read()\nsentences = [s for s in text.split(\"\\n\") if s]\nsentences = [s[:4000] for s in sentences]  # cap pathological lengths\nfrom tokenizers import Tokenizer, models, trainers\n# train on the pre-split sentences (train_from_iterator)","handlingStrategy":"validation","validationCode":"sentences = [s for s in corpus if s.strip()]\nMAX_CHARS = 4000\nif any(len(s) > MAX_CHARS for s in sentences):\n    sentences = [s[:MAX_CHARS] for s in sentences]  # cap pathological lengths\ntrainer.train_from_iterator(sentences)","typeGuard":"def is_trainable_sentence(s: str, max_chars: int = 4000) -> bool:\n    return 0 < len(s) <= max_chars","tryCatchPattern":"# panic, not exception: run training in a subprocess and detect non-zero exit\nproc = subprocess.run([sys.executable, \"train.py\", corpus_path])\nif proc.returncode != 0 and b\"likelihood is NAN\" in proc.stderr:\n    chunk_corpus_and_retry(corpus_path)","preventionTips":["Pre-segment corpora into sentences (split on newlines or use sentence tokenization) before Unigram training.","Reject or truncate extremely long lines during corpus preprocessing.","Inspect corpora for unbroken blobs (no whitespace/punctuation) that inflate lattice size.","Test training on a sample first to catch numerical issues before full runs."],"tags":["rust","training","unigram","numerical-overflow","panic"],"backgroundTag":"tensor-shape-mismatch","analyzedSha":"6cfd9d385ca0ed91c10b49f0ce97d02cfde1b607","analyzedAt":"2026-09-09T11:43:25.027Z","contentChangedAt":"2026-09-09T11:43:25.027Z","schemaVersion":2},"datasetVersion":"2026-09-16T04:17:20.429Z"}