{"record":{"id":"8d50ee0aba4e963a","repo":"stanfordnlp/CoreNLP","slug":"wordstosentencesannotator-unable-to-find-words-to","errorCode":null,"errorMessage":"WordsToSentencesAnnotator: unable to find words/tokens in: ","messagePattern":"WordsToSentencesAnnotator: unable to find words/tokens in: ","errorType":"validation","errorClass":"IllegalArgumentException","httpStatus":null,"severity":"error","filePath":"src/edu/stanford/nlp/pipeline/WordsToSentencesAnnotator.java","lineNumber":183,"sourceCode":"  public static WordsToSentencesAnnotator nonSplitter() {\n    WordToSentenceProcessor<CoreLabel> wts = new WordToSentenceProcessor<>(true);\n    return new WordsToSentencesAnnotator(false, false, wts);\n  }\n\n\n  /**\n   * If setCountLineNumbers is set to true, we count line numbers by\n   * telling the underlying splitter to return empty lists of tokens\n   * and then treating those empty lists as empty lines.  We don't\n   * actually include empty sentences in the annotation, though.\n   */\n  @Override\n  public void annotate(Annotation annotation) {\n    if (VERBOSE) {\n      log.info(\"Sentence splitting ... \" + annotation);\n    }\n    if (!annotation.containsKey(CoreAnnotations.TokensAnnotation.class)) {\n      throw new IllegalArgumentException(\"WordsToSentencesAnnotator: unable to find words/tokens in: \" + annotation);\n    }\n\n    if (annotation.containsKey(CoreAnnotations.SentencesAnnotation.class)) {\n      if (!loggedExtraSplit) {\n        log.error(\"Multiple WordsToSentencesAnnotator or other sentence splitters are operating on this document!\");\n        loggedExtraSplit = true;\n      }\n      return;\n    }\n\n    // get text and tokens from the document\n    String text = annotation.get(CoreAnnotations.TextAnnotation.class);\n    List<CoreLabel> tokens = annotation.get(CoreAnnotations.TokensAnnotation.class);\n    if (VERBOSE) {\n      log.info(\"Tokens are: \" + tokens);\n    }\n\n    String docID = annotation.get(CoreAnnotations.DocIDAnnotation.class);","sourceCodeStart":165,"sourceCodeEnd":201,"githubUrl":"https://github.com/stanfordnlp/CoreNLP/blob/1b7edd19c4d0d7b1f13a2591425b9b60a0b1af7a/src/edu/stanford/nlp/pipeline/WordsToSentencesAnnotator.java#L165-L201","documentation":"WordsToSentencesAnnotator requires tokenized text: sentence splitting operates on the TokensAnnotation key. If the Annotation does not contain tokens, the annotator throws this IllegalArgumentException because it cannot split sentences without a token stream. It almost always means a required upstream annotator (tokenize) was skipped.","triggerScenarios":"Calling WordsToSentencesAnnotator.annotate(annotation) on an Annotation that lacks CoreAnnotations.TokensAnnotation — e.g. running only the ssplit annotator without tokenize, or building an Annotation manually and never tokenizing it.","commonSituations":"Pipeline configured as \"annotators=ssplit\" (missing tokenize); custom Annotation built by hand without running TokenizerAnnotator; reusing an Annotation that was reset or created from an empty/blank string; wrong annotator ordering in the pipeline.","solutions":["Add \"tokenize\" before \"ssplit\" in the annotators list (e.g. annotators=tokenize,ssplit)","Run TokenizerAnnotator (or a tokenizer) on the Annotation before sentence splitting","If building Annotations manually, populate CoreAnnotations.TokensAnnotation with CoreLabel tokens first","Verify the Annotation actually contains text — an empty document yields no tokens"],"exampleFix":"// before\nprops.setProperty(\"annotators\", \"ssplit\");\n// after\nprops.setProperty(\"annotators\", \"tokenize,ssplit\");","handlingStrategy":"validation","validationCode":"if (!annotation.containsKey(CoreAnnotations.TokensAnnotation.class)) {\n  throw new IllegalStateException(\"Run the tokenize annotator before ssplit\");\n}","typeGuard":null,"tryCatchPattern":"try {\n  ssplitAnnotator.annotate(annotation);\n} catch (IllegalArgumentException e) {\n  if (e.getMessage() != null && e.getMessage().contains(\"unable to find words/tokens\")) {\n    new TokenizerAnnotator().annotate(annotation); // tokenize then retry once\n    ssplitAnnotator.annotate(annotation);\n  } else throw e;\n}","preventionTips":["Always list \"tokenize\" before \"ssplit\" in the annotators property","Never build Annotation objects by hand without populating TokensAnnotation","Write a pipeline smoke test over a sample document to catch missing annotator stages early"],"tags":["nlp","tokenization","pipeline","illegal-argument","precondition"],"backgroundTag":"missing-required-argument","analyzedSha":"1b7edd19c4d0d7b1f13a2591425b9b60a0b1af7a","analyzedAt":"2026-09-10T02:24:07.274Z","contentChangedAt":"2026-09-10T02:24:07.274Z","schemaVersion":2},"datasetVersion":"2026-09-17T15:17:12.973Z"}