stanfordnlp/CoreNLP · error · IllegalStateException

unexpected empty sentence:

Error message

unexpected empty sentence: 

What it means

During annotate(), after the sentence model processes the token list, each produced sentence must be non-empty. If the sentence splitter emits an empty sentence and line-number counting is disabled, this IllegalStateException is thrown because the rest of the code cannot compute offsets for an empty span. This indicates an internal inconsistency in the splitter's output.

Solutions

1. Pre-clean the input text to remove excessive blank lines/whitespace before tokenizing
2. Enable the line-number counting mode (sgmlinecount style option) which skips empty sentences via continue instead of throwing
3. Upgrade/patch to a CoreNLP version where the splitter filters empty sentences (setDocument normalizes this)
4. Inspect the offending document region and normalize the token stream feeding ssplit

Example fix

// before
String text = docText; // may contain huge runs of \n
// after
String text = docText.replaceAll("\\n{3,}", "\n\n").trim();
Annotation ann = new Annotation(text);

String clean = rawText.replaceAll("\\n{3,}", "\n\n").trim();
if (clean.isEmpty()) throw new IllegalArgumentException("Document has no tokenizable content");

try {
  ssplitAnnotator.annotate(annotation);
} catch (IllegalStateException e) {
  if (e.getMessage() != null && e.getMessage().startsWith("unexpected empty sentence")) {
    // normalize the offending document (strip blank regions) and re-run the pipeline
    annotation = normalizeAndReannotate(rawText);
  } else throw e;
}

When it happens

Trigger: Calling WordsToSentencesAnnotator.annotate() when wts.process(tokens) returns a List<CoreLabel> that is empty — typically caused by unusual token inputs (e.g. only whitespace/newline tokens) or a splitter configured without keepEmptySentences handling producing a degenerate sentence.

Common situations: Annotating documents containing long runs of newlines or blank regions; tokenizer producing stray whitespace tokens; ssplit.eolonly or boundary-token configurations interacting badly with the input; countLineNumbers mode disabled while such input is processed.

Understand the failure class

Background: "This is a bug, please report it": internal invariant violations, unreachable panics, and SNH errors explained — this error's family across 47 libraries.

Related errors

• Either logic is broken or Gabor can't code.
• originalWords and sentence of different sizes…
• Suffix pointer moved too far!
• Unknown minimizer
• adjustFinalToken: Unexpected final char: |

AI-assisted analysis of stanfordnlp/CoreNLP@1b7edd19c4 (2026-09-10). Data as JSON: /api/errors/1cb8c37dfc4d8bca. Report an issue: GitHub.

Appendix: source

Thrown at src/edu/stanford/nlp/pipeline/WordsToSentencesAnnotator.java:216

      log.info("Tokens are: " + tokens);
    }

    String docID = annotation.get(CoreAnnotations.DocIDAnnotation.class);
    // assemble the sentence annotations
    int lineNumber = 0;
    // section annotations to mark sentences with
    CoreMap sectionAnnotations = null;
    List<CoreMap> sentences = new ArrayList<>();
    // keep track of current section to assign sentences to sections
    int currSectionIndex = 0;
    List<CoreMap> sections = annotation.get(CoreAnnotations.SectionsAnnotation.class);
    for (List<CoreLabel> sentenceTokens: wts.process(tokens)) {
      if (countLineNumbers) {
        ++lineNumber;
      }
      if (sentenceTokens.isEmpty()) {
        if (!countLineNumbers) {
          throw new IllegalStateException("unexpected empty sentence: " + sentenceTokens);
        } else {
          continue;
        }
      }

      // get the sentence text from the first and last character offsets
      int begin = sentenceTokens.get(0).get(CoreAnnotations.CharacterOffsetBeginAnnotation.class);
      int last = sentenceTokens.size() - 1;
      int end = sentenceTokens.get(last).get(CoreAnnotations.CharacterOffsetEndAnnotation.class);
      String sentenceText = text.substring(begin, end);

      // create a sentence annotation with text and token offsets
      Annotation sentence = new Annotation(sentenceText);
      sentence.set(CoreAnnotations.CharacterOffsetBeginAnnotation.class, begin);
      sentence.set(CoreAnnotations.CharacterOffsetEndAnnotation.class, end);
      sentence.set(CoreAnnotations.TokensAnnotation.class, sentenceTokens);
      sentence.set(CoreAnnotations.SentenceIndexAnnotation.class, sentences.size());

View on GitHub (pinned to 1b7edd19c4)