stanfordnlp/CoreNLP · error · RuntimeException

Tokenizer unable to find text in annotation

Error message

Tokenizer unable to find text in annotation: ${annotation}

What it means

TokenizerAnnotator.annotate expects the annotation to carry text via TextAnnotation (or a list of CoreMaps). When no text can be found on the annotation, it throws RuntimeException because there is nothing to tokenize. This is a missing-input error, not a tokenization failure.

Solutions

1. Create the Annotation with the text: new Annotation("some text") or annotation.set(CoreAnnotations.TextAnnotation.class, text).
2. Verify the text is non-null before calling the tokenizer annotator.
3. Run the annotator at the start of the pipeline before any stage that might strip text.
4. If re-tokenizing, rebuild the Annotation from the original string.

Example fix

// before
Annotation ann = new Annotation((String) null);
pipeline.annotate(ann);
// after
Annotation ann = new Annotation("Hello world");
pipeline.annotate(ann);

String text = ann.get(CoreAnnotations.TextAnnotation.class);
if (text == null || text.isEmpty()) throw new IllegalStateException("Annotation has no text; set CoreAnnotations.TextAnnotation before tokenizing");

boolean hasText(Annotation ann) { String t = ann.get(CoreAnnotations.TextAnnotation.class); return t != null && !t.isEmpty(); }

try { tokenizer.annotate(ann); } catch (RuntimeException e) { if (e.getMessage().startsWith("Tokenizer unable to find text")) { throw new MissingTextException(e); } throw e; }

When it happens

Trigger: Calling annotate() on an Annotation created without text (e.g. new Annotation((String) null)) or one whose TextAnnotation is absent/empty while the code path requires it; also when using annotate(List<CoreMap>) style input incorrectly.

Common situations: Building Annotation manually without setting text; clearing annotations before re-annotating; passing an Annotation that only has tokens/sentences from a prior stage but no raw text; deserialized annotations missing the text key.

Understand the failure class

Background: "must not be empty", "cannot be empty" — required-field validation errors across open-source libraries — this error's family across 41 libraries.

Related errors

• Shouldn't happen:
• Error reading saved links
• RuntimeIOException wrapping IOException
• Error creating data exporter
• Error setting up training

AI-assisted analysis of stanfordnlp/CoreNLP@1b7edd19c4 (2026-09-10). Data as JSON: /api/errors/e3e36388d7277c8a. Report an issue: GitHub.

Appendix: source

Thrown at src/edu/stanford/nlp/pipeline/TokenizerAnnotator.java:453

      // label newlines
      setNewlineStatus(tokens);

      // set indexes into document wide token list
      setTokenBeginTokenEnd(tokens);

      // run post processing
      for (CoreLabelProcessor postProcessor : postProcessors) {
        tokens = postProcessor.process(tokens);
      }

      // add tokens list to annotation
      annotation.set(CoreAnnotations.TokensAnnotation.class, tokens);

      if (VERBOSE) {
        log.info("Tokenized: " + annotation.get(CoreAnnotations.TokensAnnotation.class));
      }
    } else {
      throw new RuntimeException("Tokenizer unable to find text in annotation: " + annotation);
    }

    // If the annotation was already processed before and already has
    // a SentenceAnnotation.class, recreating the tokenization
    // invalidates any existing sentence annotation
    annotation.remove(CoreAnnotations.SentencesAnnotation.class);
    if (this.cleanxmlAnnotator != null) {
      this.cleanxmlAnnotator.annotate(annotation);
    }
    if (this.ssplitAnnotator != null) {
      this.ssplitAnnotator.annotate(annotation);
    }
  }

  @Override
  public Set<Class<? extends CoreAnnotation>> requires() {
    return Collections.emptySet();
  }

View on GitHub (pinned to 1b7edd19c4)