{"record":{"id":"e3e36388d7277c8a","repo":"stanfordnlp/CoreNLP","slug":"tokenizer-unable-to-find-text-in-annotation-ann","errorCode":null,"errorMessage":"Tokenizer unable to find text in annotation: ${annotation}","messagePattern":"Tokenizer unable to find text in annotation: (.+?)","errorType":"exception","errorClass":"RuntimeException","httpStatus":null,"severity":"error","filePath":"src/edu/stanford/nlp/pipeline/TokenizerAnnotator.java","lineNumber":453,"sourceCode":"      // label newlines\n      setNewlineStatus(tokens);\n\n      // set indexes into document wide token list\n      setTokenBeginTokenEnd(tokens);\n\n      // run post processing\n      for (CoreLabelProcessor postProcessor : postProcessors) {\n        tokens = postProcessor.process(tokens);\n      }\n\n      // add tokens list to annotation\n      annotation.set(CoreAnnotations.TokensAnnotation.class, tokens);\n\n      if (VERBOSE) {\n        log.info(\"Tokenized: \" + annotation.get(CoreAnnotations.TokensAnnotation.class));\n      }\n    } else {\n      throw new RuntimeException(\"Tokenizer unable to find text in annotation: \" + annotation);\n    }\n\n    // If the annotation was already processed before and already has\n    // a SentenceAnnotation.class, recreating the tokenization\n    // invalidates any existing sentence annotation\n    annotation.remove(CoreAnnotations.SentencesAnnotation.class);\n    if (this.cleanxmlAnnotator != null) {\n      this.cleanxmlAnnotator.annotate(annotation);\n    }\n    if (this.ssplitAnnotator != null) {\n      this.ssplitAnnotator.annotate(annotation);\n    }\n  }\n\n  @Override\n  public Set<Class<? extends CoreAnnotation>> requires() {\n    return Collections.emptySet();\n  }","sourceCodeStart":435,"sourceCodeEnd":471,"githubUrl":"https://github.com/stanfordnlp/CoreNLP/blob/1b7edd19c4d0d7b1f13a2591425b9b60a0b1af7a/src/edu/stanford/nlp/pipeline/TokenizerAnnotator.java#L435-L471","documentation":"TokenizerAnnotator.annotate expects the annotation to carry text via TextAnnotation (or a list of CoreMaps). When no text can be found on the annotation, it throws RuntimeException because there is nothing to tokenize. This is a missing-input error, not a tokenization failure.","triggerScenarios":"Calling annotate() on an Annotation created without text (e.g. new Annotation((String) null)) or one whose TextAnnotation is absent/empty while the code path requires it; also when using annotate(List<CoreMap>) style input incorrectly.","commonSituations":"Building Annotation manually without setting text; clearing annotations before re-annotating; passing an Annotation that only has tokens/sentences from a prior stage but no raw text; deserialized annotations missing the text key.","solutions":["Create the Annotation with the text: new Annotation(\"some text\") or annotation.set(CoreAnnotations.TextAnnotation.class, text).","Verify the text is non-null before calling the tokenizer annotator.","Run the annotator at the start of the pipeline before any stage that might strip text.","If re-tokenizing, rebuild the Annotation from the original string."],"exampleFix":"// before\nAnnotation ann = new Annotation((String) null);\npipeline.annotate(ann);\n// after\nAnnotation ann = new Annotation(\"Hello world\");\npipeline.annotate(ann);","handlingStrategy":"validation","validationCode":"String text = ann.get(CoreAnnotations.TextAnnotation.class);\nif (text == null || text.isEmpty()) throw new IllegalStateException(\"Annotation has no text; set CoreAnnotations.TextAnnotation before tokenizing\");","typeGuard":"boolean hasText(Annotation ann) { String t = ann.get(CoreAnnotations.TextAnnotation.class); return t != null && !t.isEmpty(); }","tryCatchPattern":"try { tokenizer.annotate(ann); } catch (RuntimeException e) { if (e.getMessage().startsWith(\"Tokenizer unable to find text\")) { throw new MissingTextException(e); } throw e; }","preventionTips":["Always construct Annotation with a non-null string","Do not clear text annotations mid-pipeline","Assert hasText(ann) before any annotator that tokenizes"],"tags":["java","annotation","missing-input","corenlp"],"backgroundTag":"empty-required-field","analyzedSha":"1b7edd19c4d0d7b1f13a2591425b9b60a0b1af7a","analyzedAt":"2026-09-10T02:24:07.274Z","contentChangedAt":"2026-09-10T02:24:07.274Z","schemaVersion":2},"datasetVersion":"2026-09-15T23:17:13.987Z"}