{"record":{"id":"97d2f87ae04c084f","repo":"stanfordnlp/CoreNLP","slug":"unable-to-find-words-tokens-in-annotation-97d2f8","errorCode":null,"errorMessage":"unable to find words/tokens in: ${annotation}","messagePattern":"unable to find words/tokens in: (.+?)","errorType":"exception","errorClass":"RuntimeException","httpStatus":null,"severity":"error","filePath":"src/edu/stanford/nlp/pipeline/POSTaggerAnnotator.java","lineNumber":114,"sourceCode":"      if (nThreads == 1) {\n        for (CoreMap sentence : annotation.get(CoreAnnotations.SentencesAnnotation.class)) {\n          doOneSentence(sentence);\n        }\n      } else {\n        MulticoreWrapper<CoreMap, CoreMap> wrapper = new MulticoreWrapper<>(nThreads, new POSTaggerProcessor());\n        for (CoreMap sentence : annotation.get(CoreAnnotations.SentencesAnnotation.class)) {\n          wrapper.put(sentence);\n          while (wrapper.peek()) {\n            wrapper.poll();\n          }\n        }\n        wrapper.join();\n        while (wrapper.peek()) {\n          wrapper.poll();\n        }\n      }\n    } else {\n      throw new RuntimeException(\"unable to find words/tokens in: \" + annotation);\n    }\n  }\n\n  private class POSTaggerProcessor implements ThreadsafeProcessor<CoreMap, CoreMap> {\n    @Override\n    public CoreMap process(CoreMap sentence) {\n      return doOneSentence(sentence);\n    }\n\n    @Override\n    public ThreadsafeProcessor<CoreMap, CoreMap> newInstance() {\n      return this;\n    }\n  }\n\n  private CoreMap doOneSentence(CoreMap sentence) {\n    List<CoreLabel> tokens = sentence.get(CoreAnnotations.TokensAnnotation.class);\n    List<TaggedWord> tagged = null;","sourceCodeStart":96,"sourceCodeEnd":132,"githubUrl":"https://github.com/stanfordnlp/CoreNLP/blob/1b7edd19c4d0d7b1f13a2591425b9b60a0b1af7a/src/edu/stanford/nlp/pipeline/POSTaggerAnnotator.java#L96-L132","documentation":"An IllegalStateException precondition check in POSTaggerAnnotator.annotate: the annotation contains no SentencesAnnotation (no tokenized sentences), so there are no words to tag; run tokenization/sentence splitting first.","triggerScenarios":"Calling POSTaggerAnnotator.annotate() on an annotation without TokensAnnotation or SentencesAnnotation — typically when 'pos' is requested without 'tokenize' and 'ssplit'. Also reachable via testMulticoreAnnotation when input lacks tokens.","commonSituations":"Pipeline misconfiguration omitting tokenize/ssplit; calling the POS tagger processor directly on a document-level annotation built manually; multithreaded usage feeding annotations that skipped earlier stages.","solutions":["Set annotators=tokenize,ssplit,pos so tokens exist before POS tagging","Run the tokenizer on the annotation (TokenizerAnnotator) before invoking POSTaggerAnnotator","Check that your multithreaded wrapper passes sentence/token-annotated CoreMaps"],"exampleFix":"// before\nprops.setProperty(\"annotators\", \"pos\");\n// after\nprops.setProperty(\"annotators\", \"tokenize,ssplit,pos\");","handlingStrategy":"validation","validationCode":"if (!annotation.containsKey(CoreAnnotations.TokensAnnotation.class)) { pipeline.annotate(annotation); }","typeGuard":null,"tryCatchPattern":"try { posAnnotator.annotate(annotation); } catch (RuntimeException e) { if (e.getMessage().contains(\"unable to find words/tokens\")) { pipeline.annotate(annotation); posAnnotator.annotate(annotation); } else throw e; }","preventionTips":["Keep annotator order tokenize,ssplit,pos in properties","In multicore setups, verify each worker receives tokenized CoreMaps"],"tags":["java","nlp","pipeline","pos-tagging"],"backgroundTag":"missing-required-config-field","analyzedSha":"1b7edd19c4d0d7b1f13a2591425b9b60a0b1af7a","analyzedAt":"2026-09-10T02:24:07.274Z","contentChangedAt":"2026-09-10T02:24:07.274Z","schemaVersion":2},"datasetVersion":"2026-09-15T23:17:13.987Z"}