What causes "unable to find words/tokens in: ${annotation}" in stanfordnlp/CoreNLP?

Set annotators=tokenize,ssplit,pos so tokens exist before POS tagging

stanfordnlp/CoreNLP · error · RuntimeException

unable to find words/tokens in

Error message

unable to find words/tokens in: ${annotation}

What it means

An IllegalStateException precondition check in POSTaggerAnnotator.annotate: the annotation contains no SentencesAnnotation (no tokenized sentences), so there are no words to tag; run tokenization/sentence splitting first.

Solutions

Set annotators=tokenize,ssplit,pos so tokens exist before POS tagging
Run the tokenizer on the annotation (TokenizerAnnotator) before invoking POSTaggerAnnotator
Check that your multithreaded wrapper passes sentence/token-annotated CoreMaps

Example fix

// before
props.setProperty("annotators", "pos");
// after
props.setProperty("annotators", "tokenize,ssplit,pos");

Defensive patterns

Strategy: validation

Validate before calling

if (!annotation.containsKey(CoreAnnotations.TokensAnnotation.class)) { pipeline.annotate(annotation); }

Try / catch

try { posAnnotator.annotate(annotation); } catch (RuntimeException e) { if (e.getMessage().contains("unable to find words/tokens")) { pipeline.annotate(annotation); posAnnotator.annotate(annotation); } else throw e; }

Prevention

Keep annotator order tokenize,ssplit,pos in properties
In multicore setups, verify each worker receives tokenized CoreMaps

When it happens

Trigger: Calling POSTaggerAnnotator.annotate() on an annotation without TokensAnnotation or SentencesAnnotation — typically when 'pos' is requested without 'tokenize' and 'ssplit'. Also reachable via testMulticoreAnnotation when input lacks tokens.

Common situations: Pipeline misconfiguration omitting tokenize/ssplit; calling the POS tagger processor directly on a document-level annotation built manually; multithreaded usage feeding annotations that skipped earlier stages.

Understand the failure class

Background: "is required", "must be set", "missing required field": configuration validation errors across open-source libraries — this error's family across 36 libraries.

Related errors

AI-assisted analysis of stanfordnlp/CoreNLP@1b7edd19c4 (2026-09-10). Data as JSON: /api/errors/97d2f87ae04c084f. Report an issue: GitHub.

Appendix: source

Thrown at src/edu/stanford/nlp/pipeline/POSTaggerAnnotator.java:114

      if (nThreads == 1) {
        for (CoreMap sentence : annotation.get(CoreAnnotations.SentencesAnnotation.class)) {
          doOneSentence(sentence);
        }
      } else {
        MulticoreWrapper<CoreMap, CoreMap> wrapper = new MulticoreWrapper<>(nThreads, new POSTaggerProcessor());
        for (CoreMap sentence : annotation.get(CoreAnnotations.SentencesAnnotation.class)) {
          wrapper.put(sentence);
          while (wrapper.peek()) {
            wrapper.poll();
          }
        }
        wrapper.join();
        while (wrapper.peek()) {
          wrapper.poll();
        }
      }
    } else {
      throw new RuntimeException("unable to find words/tokens in: " + annotation);
    }
  }

  private class POSTaggerProcessor implements ThreadsafeProcessor<CoreMap, CoreMap> {
    @Override
    public CoreMap process(CoreMap sentence) {
      return doOneSentence(sentence);
    }

    @Override
    public ThreadsafeProcessor<CoreMap, CoreMap> newInstance() {
      return this;
    }
  }

  private CoreMap doOneSentence(CoreMap sentence) {
    List<CoreLabel> tokens = sentence.get(CoreAnnotations.TokensAnnotation.class);
    List<TaggedWord> tagged = null;

View on GitHub (pinned to 1b7edd19c4)