stanfordnlp/CoreNLP · error · RuntimeException

Unable to find sentences or tokens in

Error message

Unable to find sentences or tokens in ${annotation}

What it means

TokensRegexNERAnnotator.annotate() requires the input Annotation to carry tokens (CoreAnnotations.TokensAnnotation) or sentences; it throws a plain RuntimeException when neither is present. This means the annotation was never tokenized before this annotator ran, so there is nothing for the TokensRegex rules to match against.

Solutions

1. Add 'tokenize' (and usually 'ssplit') to the pipeline properties so tokens exist before tokensregexner runs.
2. If annotating a single sentence/string manually, call the TokenizerAnnotator or annotate with a tokenize-ssplit pipeline first.
3. If constructing the Annotation programmatically, set CoreAnnotations.TokensAnnotation with a non-empty List<CoreLabel> before calling annotate().
4. Wrap the call in try-catch for RuntimeException and re-run with tokenization when tokens are missing.

Example fix

// before
Properties props = new Properties();
props.setProperty("annotators", "tokensregexner");
// after
Properties props = new Properties();
props.setProperty("annotators", "tokenize,ssplit,tokensregexner");

if (annotation.get(CoreAnnotations.TokensAnnotation.class) == null && annotation.get(CoreAnnotations.SentencesAnnotation.class) == null) {
  throw new IllegalStateException("Run tokenize/ssplit before TokensRegexNERAnnotator");
}

try { annotator.annotate(annotation); } catch (RuntimeException e) { if (e.getMessage().startsWith("Unable to find sentences or tokens")) { pipeline.annotate(annotation); annotator.annotate(annotation); } else throw e; }

When it happens

Trigger: Calling annotate(Annotation) directly on a CoreMap/Annotation created programmatically without running tokenize first, or building a pipeline where TokensRegexNER runs before tokenize/ssa so annotation.get(TokensAnnotation.class) returns null.

Common situations: Custom StanfordCoreNLP pipeline property lists that omit 'tokenize,ssplit' before 'tokensregexner'; unit tests constructing a new Annotation("text") and invoking the annotator directly; streaming code that passes partial CoreMaps lacking TokensAnnotation.

Understand the failure class

Background: "missing required argument" and "the following required arguments were not provided": what required-argument errors mean and how to fix them — this error's family across 20 libraries.

Related errors

• Could not parse date string
• Could not parse date string
• TIMEX3 should only contain text
• unexpected content
• unexpected element

AI-assisted analysis of stanfordnlp/CoreNLP@1b7edd19c4 (2026-09-10). Data as JSON: /api/errors/a21611dd379c5a16. Report an issue: GitHub.

Appendix: source

Thrown at src/edu/stanford/nlp/pipeline/TokensRegexNERAnnotator.java:349

  @Override
  public void annotate(Annotation annotation) {
    if (verbose) {
      logger.info("Adding TokensRegexNER annotations ... ");
    }

    List<CoreMap> sentences = annotation.get(CoreAnnotations.SentencesAnnotation.class);
    if (sentences != null) {
      for (CoreMap sentence : sentences) {
        List<CoreLabel> tokens = sentence.get(CoreAnnotations.TokensAnnotation.class);
        annotateMatched(tokens);
      }
    } else {
      List<CoreLabel> tokens = annotation.get(CoreAnnotations.TokensAnnotation.class);
      if (tokens != null){
        annotateMatched(tokens);
      } else {
        throw new RuntimeException("Unable to find sentences or tokens in " + annotation);
      }
    }

    if (verbose)
      logger.info("done.");
  }

  private MultiPatternMatcher<CoreMap> createPatternMatcher(Map<SequencePattern<CoreMap>, Entry> patternToEntry) {
    // Convert to tokensregex pattern

    List<TokenSequencePattern> patterns = new ArrayList<>(entries.size());
    for (Entry entry:entries) {
      TokenSequencePattern pattern;

      Boolean ignoreCaseEntry = ignoreCaseList.get(entryToMappingFileNumber.get(entry));
      int patternFlags = ignoreCaseEntry? Pattern.CASE_INSENSITIVE | Pattern.UNICODE_CASE:0;
      int stringMatchFlags = ignoreCaseEntry? (NodePattern.CASE_INSENSITIVE | NodePattern.UNICODE_CASE):0;
      Env env = TokenSequencePattern.getNewEnv();

View on GitHub (pinned to 1b7edd19c4)