stanfordnlp/CoreNLP · error · RuntimeException

Unable to find words/tokens in

Error message

Unable to find words/tokens in: ${annotation}

What it means

MorphaAnnotator requires the annotation to contain sentence or token annotations before it can lemmatize. If neither SentencesAnnotation nor TokensAnnotation is present, it throws this RuntimeException because there are no words/tokens to run the morphological analyzer on.

Solutions

  1. Add 'tokenize' and 'ssplit' before 'lemma' in the annotators list, e.g. annotators=tokenize,ssplit,pos,lemma
  2. If working at the token level, ensure TokensAnnotation is set on the annotation before calling annotate()
  3. Verify you call annotate() on the whole pipeline output, not a fresh Annotation created from raw text

Example fix

// before
props.setProperty("annotators", "lemma");
// after
props.setProperty("annotators", "tokenize,ssplit,pos,lemma");
Defensive patterns

Strategy: validation

Validate before calling

if (!annotation.containsKey(CoreAnnotations.SentencesAnnotation.class) && !annotation.containsKey(CoreAnnotations.TokensAnnotation.class)) { throw new IllegalStateException("Run tokenize,ssplit before lemma"); }

Try / catch

try { annotator.annotate(annotation); } catch (RuntimeException e) { if (e.getMessage().startsWith("Unable to find words/tokens")) { pipeline.annotate(annotation); annotator.annotate(annotation); } else throw e; }

Prevention

When it happens

Trigger: Calling MorphaAnnotator.annotate() on a CoreMap/Annotation that has not been tokenized (no TokensAnnotation) or segmented into sentences (no SentencesAnnotation), e.g. running the 'lemma' annotator without 'tokenize,ssplit' earlier in the pipeline.

Common situations: Misordered Stanford CoreNLP pipeline properties where lemma is requested before tokenize/ssplit; calling annotators manually on a raw Annotation built only with CoreAnnotations.TextAnnotation.

Understand the failure class

Background: "is required", "must be set", "missing required field": configuration validation errors across open-source libraries — this error's family across 36 libraries.

Related errors

AI-assisted analysis of stanfordnlp/CoreNLP@1b7edd19c4 (2026-09-10). Data as JSON: /api/errors/d157b0074a007d49. Report an issue: GitHub.

Appendix: source

Thrown at src/edu/stanford/nlp/pipeline/MorphaAnnotator.java:63

  @Override
  public void annotate(Annotation annotation) {
    if (VERBOSE) {
      log.info("Finding lemmas ...");
    }
    Morphology morphology = new Morphology();
    if (annotation.containsKey(CoreAnnotations.SentencesAnnotation.class)) {
      for (CoreMap sentence : annotation.get(CoreAnnotations.SentencesAnnotation.class)) {
        List<CoreLabel> tokens = sentence.get(CoreAnnotations.TokensAnnotation.class);
        //log.info("Lemmatizing sentence: " + tokens);
        for (CoreLabel token : tokens) {
          String text = token.get(CoreAnnotations.TextAnnotation.class);
          String posTag = token.get(CoreAnnotations.PartOfSpeechAnnotation.class);
          addLemma(morphology, CoreAnnotations.LemmaAnnotation.class, token, text, posTag);
        }
      }
    } else {
      throw new RuntimeException("Unable to find words/tokens in: " +
                                 annotation);
    }
  }


  private static void addLemma(Morphology morpha,
                        Class<? extends CoreAnnotation<String>> ann,
                        CoreMap map, String word, String tag) {
    if ( ! tag.isEmpty()) {
      String phrasalVerb = phrasalVerb(morpha, word, tag);
      if (phrasalVerb == null) {
        map.set(ann, morpha.lemma(word, tag));
      } else {
        map.set(ann, phrasalVerb);
      }
    } else {
      map.set(ann, morpha.stem(word));
    }

View on GitHub (pinned to 1b7edd19c4)