{"record":{"id":"d157b0074a007d49","repo":"stanfordnlp/CoreNLP","slug":"unable-to-find-words-tokens-in-annotation","errorCode":null,"errorMessage":"Unable to find words/tokens in: ${annotation}","messagePattern":"Unable to find words/tokens in: (.+?)","errorType":"exception","errorClass":"RuntimeException","httpStatus":null,"severity":"error","filePath":"src/edu/stanford/nlp/pipeline/MorphaAnnotator.java","lineNumber":63,"sourceCode":"\n  @Override\n  public void annotate(Annotation annotation) {\n    if (VERBOSE) {\n      log.info(\"Finding lemmas ...\");\n    }\n    Morphology morphology = new Morphology();\n    if (annotation.containsKey(CoreAnnotations.SentencesAnnotation.class)) {\n      for (CoreMap sentence : annotation.get(CoreAnnotations.SentencesAnnotation.class)) {\n        List<CoreLabel> tokens = sentence.get(CoreAnnotations.TokensAnnotation.class);\n        //log.info(\"Lemmatizing sentence: \" + tokens);\n        for (CoreLabel token : tokens) {\n          String text = token.get(CoreAnnotations.TextAnnotation.class);\n          String posTag = token.get(CoreAnnotations.PartOfSpeechAnnotation.class);\n          addLemma(morphology, CoreAnnotations.LemmaAnnotation.class, token, text, posTag);\n        }\n      }\n    } else {\n      throw new RuntimeException(\"Unable to find words/tokens in: \" +\n                                 annotation);\n    }\n  }\n\n\n  private static void addLemma(Morphology morpha,\n                        Class<? extends CoreAnnotation<String>> ann,\n                        CoreMap map, String word, String tag) {\n    if ( ! tag.isEmpty()) {\n      String phrasalVerb = phrasalVerb(morpha, word, tag);\n      if (phrasalVerb == null) {\n        map.set(ann, morpha.lemma(word, tag));\n      } else {\n        map.set(ann, phrasalVerb);\n      }\n    } else {\n      map.set(ann, morpha.stem(word));\n    }","sourceCodeStart":45,"sourceCodeEnd":81,"githubUrl":"https://github.com/stanfordnlp/CoreNLP/blob/1b7edd19c4d0d7b1f13a2591425b9b60a0b1af7a/src/edu/stanford/nlp/pipeline/MorphaAnnotator.java#L45-L81","documentation":"MorphaAnnotator requires the annotation to contain sentence or token annotations before it can lemmatize. If neither SentencesAnnotation nor TokensAnnotation is present, it throws this RuntimeException because there are no words/tokens to run the morphological analyzer on.","triggerScenarios":"Calling MorphaAnnotator.annotate() on a CoreMap/Annotation that has not been tokenized (no TokensAnnotation) or segmented into sentences (no SentencesAnnotation), e.g. running the 'lemma' annotator without 'tokenize,ssplit' earlier in the pipeline.","commonSituations":"Misordered Stanford CoreNLP pipeline properties where lemma is requested before tokenize/ssplit; calling annotators manually on a raw Annotation built only with CoreAnnotations.TextAnnotation.","solutions":["Add 'tokenize' and 'ssplit' before 'lemma' in the annotators list, e.g. annotators=tokenize,ssplit,pos,lemma","If working at the token level, ensure TokensAnnotation is set on the annotation before calling annotate()","Verify you call annotate() on the whole pipeline output, not a fresh Annotation created from raw text"],"exampleFix":"// before\nprops.setProperty(\"annotators\", \"lemma\");\n// after\nprops.setProperty(\"annotators\", \"tokenize,ssplit,pos,lemma\");","handlingStrategy":"validation","validationCode":"if (!annotation.containsKey(CoreAnnotations.SentencesAnnotation.class) && !annotation.containsKey(CoreAnnotations.TokensAnnotation.class)) { throw new IllegalStateException(\"Run tokenize,ssplit before lemma\"); }","typeGuard":null,"tryCatchPattern":"try { annotator.annotate(annotation); } catch (RuntimeException e) { if (e.getMessage().startsWith(\"Unable to find words/tokens\")) { pipeline.annotate(annotation); annotator.annotate(annotation); } else throw e; }","preventionTips":["Always list tokenize and ssplit before lemma/ner/pos in the annotators property","Use StanfordCoreNLP pipeline instead of invoking annotators directly on raw text"],"tags":["java","nlp","pipeline","missing-annotation"],"backgroundTag":"missing-required-config-field","analyzedSha":"1b7edd19c4d0d7b1f13a2591425b9b60a0b1af7a","analyzedAt":"2026-09-10T02:24:07.274Z","contentChangedAt":"2026-09-10T02:24:07.274Z","schemaVersion":2},"datasetVersion":"2026-09-15T23:17:13.987Z"}