{"record":{"id":"1cb8c37dfc4d8bca","repo":"stanfordnlp/CoreNLP","slug":"unexpected-empty-sentence","errorCode":null,"errorMessage":"unexpected empty sentence: ","messagePattern":"unexpected empty sentence: ","errorType":"exception","errorClass":"IllegalStateException","httpStatus":null,"severity":"error","filePath":"src/edu/stanford/nlp/pipeline/WordsToSentencesAnnotator.java","lineNumber":216,"sourceCode":"      log.info(\"Tokens are: \" + tokens);\n    }\n\n    String docID = annotation.get(CoreAnnotations.DocIDAnnotation.class);\n    // assemble the sentence annotations\n    int lineNumber = 0;\n    // section annotations to mark sentences with\n    CoreMap sectionAnnotations = null;\n    List<CoreMap> sentences = new ArrayList<>();\n    // keep track of current section to assign sentences to sections\n    int currSectionIndex = 0;\n    List<CoreMap> sections = annotation.get(CoreAnnotations.SectionsAnnotation.class);\n    for (List<CoreLabel> sentenceTokens: wts.process(tokens)) {\n      if (countLineNumbers) {\n        ++lineNumber;\n      }\n      if (sentenceTokens.isEmpty()) {\n        if (!countLineNumbers) {\n          throw new IllegalStateException(\"unexpected empty sentence: \" + sentenceTokens);\n        } else {\n          continue;\n        }\n      }\n\n      // get the sentence text from the first and last character offsets\n      int begin = sentenceTokens.get(0).get(CoreAnnotations.CharacterOffsetBeginAnnotation.class);\n      int last = sentenceTokens.size() - 1;\n      int end = sentenceTokens.get(last).get(CoreAnnotations.CharacterOffsetEndAnnotation.class);\n      String sentenceText = text.substring(begin, end);\n\n      // create a sentence annotation with text and token offsets\n      Annotation sentence = new Annotation(sentenceText);\n      sentence.set(CoreAnnotations.CharacterOffsetBeginAnnotation.class, begin);\n      sentence.set(CoreAnnotations.CharacterOffsetEndAnnotation.class, end);\n      sentence.set(CoreAnnotations.TokensAnnotation.class, sentenceTokens);\n      sentence.set(CoreAnnotations.SentenceIndexAnnotation.class, sentences.size());\n","sourceCodeStart":198,"sourceCodeEnd":234,"githubUrl":"https://github.com/stanfordnlp/CoreNLP/blob/1b7edd19c4d0d7b1f13a2591425b9b60a0b1af7a/src/edu/stanford/nlp/pipeline/WordsToSentencesAnnotator.java#L198-L234","documentation":"During annotate(), after the sentence model processes the token list, each produced sentence must be non-empty. If the sentence splitter emits an empty sentence and line-number counting is disabled, this IllegalStateException is thrown because the rest of the code cannot compute offsets for an empty span. This indicates an internal inconsistency in the splitter's output.","triggerScenarios":"Calling WordsToSentencesAnnotator.annotate() when wts.process(tokens) returns a List<CoreLabel> that is empty — typically caused by unusual token inputs (e.g. only whitespace/newline tokens) or a splitter configured without keepEmptySentences handling producing a degenerate sentence.","commonSituations":"Annotating documents containing long runs of newlines or blank regions; tokenizer producing stray whitespace tokens; ssplit.eolonly or boundary-token configurations interacting badly with the input; countLineNumbers mode disabled while such input is processed.","solutions":["Pre-clean the input text to remove excessive blank lines/whitespace before tokenizing","Enable the line-number counting mode (sgmlinecount style option) which skips empty sentences via continue instead of throwing","Upgrade/patch to a CoreNLP version where the splitter filters empty sentences (setDocument normalizes this)","Inspect the offending document region and normalize the token stream feeding ssplit"],"exampleFix":"// before\nString text = docText; // may contain huge runs of \\n\n// after\nString text = docText.replaceAll(\"\\\\n{3,}\", \"\\n\\n\").trim();\nAnnotation ann = new Annotation(text);","handlingStrategy":"validation","validationCode":"String clean = rawText.replaceAll(\"\\\\n{3,}\", \"\\n\\n\").trim();\nif (clean.isEmpty()) throw new IllegalArgumentException(\"Document has no tokenizable content\");","typeGuard":null,"tryCatchPattern":"try {\n  ssplitAnnotator.annotate(annotation);\n} catch (IllegalStateException e) {\n  if (e.getMessage() != null && e.getMessage().startsWith(\"unexpected empty sentence\")) {\n    // normalize the offending document (strip blank regions) and re-run the pipeline\n    annotation = normalizeAndReannotate(rawText);\n  } else throw e;\n}","preventionTips":["Normalize whitespace/newline runs in input documents before tokenization","Skip or log documents that tokenize to zero tokens instead of annotating them","Keep CoreNLP updated — newer versions filter degenerate empty sentences"],"tags":["nlp","sentence-splitting","invariant","illegal-state","empty-input"],"backgroundTag":"internal-invariant-violation","analyzedSha":"1b7edd19c4d0d7b1f13a2591425b9b60a0b1af7a","analyzedAt":"2026-09-10T02:24:07.274Z","contentChangedAt":"2026-09-10T02:24:07.274Z","schemaVersion":2},"datasetVersion":"2026-09-17T15:17:12.973Z"}