stanfordnlp/CoreNLP · error · IllegalArgumentException

WordsToSentencesAnnotator: unable to find words/tokens in:

Error message

WordsToSentencesAnnotator: unable to find words/tokens in: 

What it means

WordsToSentencesAnnotator requires tokenized text: sentence splitting operates on the TokensAnnotation key. If the Annotation does not contain tokens, the annotator throws this IllegalArgumentException because it cannot split sentences without a token stream. It almost always means a required upstream annotator (tokenize) was skipped.

Solutions

1. Add "tokenize" before "ssplit" in the annotators list (e.g. annotators=tokenize,ssplit)
2. Run TokenizerAnnotator (or a tokenizer) on the Annotation before sentence splitting
3. If building Annotations manually, populate CoreAnnotations.TokensAnnotation with CoreLabel tokens first
4. Verify the Annotation actually contains text — an empty document yields no tokens

Example fix

// before
props.setProperty("annotators", "ssplit");
// after
props.setProperty("annotators", "tokenize,ssplit");

if (!annotation.containsKey(CoreAnnotations.TokensAnnotation.class)) {
  throw new IllegalStateException("Run the tokenize annotator before ssplit");
}

try {
  ssplitAnnotator.annotate(annotation);
} catch (IllegalArgumentException e) {
  if (e.getMessage() != null && e.getMessage().contains("unable to find words/tokens")) {
    new TokenizerAnnotator().annotate(annotation); // tokenize then retry once
    ssplitAnnotator.annotate(annotation);
  } else throw e;
}

When it happens

Trigger: Calling WordsToSentencesAnnotator.annotate(annotation) on an Annotation that lacks CoreAnnotations.TokensAnnotation — e.g. running only the ssplit annotator without tokenize, or building an Annotation manually and never tokenizing it.

Common situations: Pipeline configured as "annotators=ssplit" (missing tokenize); custom Annotation built by hand without running TokenizerAnnotator; reusing an Annotation that was reset or created from an empty/blank string; wrong annotator ordering in the pipeline.

Understand the failure class

Background: "missing required argument" and "the following required arguments were not provided": what required-argument errors mean and how to fix them — this error's family across 20 libraries.

Related errors

• Can only operate on preterminals
• Cannot read from null object!
• Cannot run Natural Logic forward entailment without…
• Cannot run OpenIE without a parse tree!
• CoreMap must have either a Calendar or DocDate annotation

AI-assisted analysis of stanfordnlp/CoreNLP@1b7edd19c4 (2026-09-10). Data as JSON: /api/errors/8d50ee0aba4e963a. Report an issue: GitHub.

Appendix: source

Thrown at src/edu/stanford/nlp/pipeline/WordsToSentencesAnnotator.java:183

  public static WordsToSentencesAnnotator nonSplitter() {
    WordToSentenceProcessor<CoreLabel> wts = new WordToSentenceProcessor<>(true);
    return new WordsToSentencesAnnotator(false, false, wts);
  }


  /**
   * If setCountLineNumbers is set to true, we count line numbers by
   * telling the underlying splitter to return empty lists of tokens
   * and then treating those empty lists as empty lines.  We don't
   * actually include empty sentences in the annotation, though.
   */
  @Override
  public void annotate(Annotation annotation) {
    if (VERBOSE) {
      log.info("Sentence splitting ... " + annotation);
    }
    if (!annotation.containsKey(CoreAnnotations.TokensAnnotation.class)) {
      throw new IllegalArgumentException("WordsToSentencesAnnotator: unable to find words/tokens in: " + annotation);
    }

    if (annotation.containsKey(CoreAnnotations.SentencesAnnotation.class)) {
      if (!loggedExtraSplit) {
        log.error("Multiple WordsToSentencesAnnotator or other sentence splitters are operating on this document!");
        loggedExtraSplit = true;
      }
      return;
    }

    // get text and tokens from the document
    String text = annotation.get(CoreAnnotations.TextAnnotation.class);
    List<CoreLabel> tokens = annotation.get(CoreAnnotations.TokensAnnotation.class);
    if (VERBOSE) {
      log.info("Tokens are: " + tokens);
    }

    String docID = annotation.get(CoreAnnotations.DocIDAnnotation.class);

View on GitHub (pinned to 1b7edd19c4)