{"record":{"id":"a21611dd379c5a16","repo":"stanfordnlp/CoreNLP","slug":"unable-to-find-sentences-or-tokens-in-annotation","errorCode":null,"errorMessage":"Unable to find sentences or tokens in ${annotation}","messagePattern":"Unable to find sentences or tokens in (.+?)","errorType":"exception","errorClass":"RuntimeException","httpStatus":null,"severity":"error","filePath":"src/edu/stanford/nlp/pipeline/TokensRegexNERAnnotator.java","lineNumber":349,"sourceCode":"\n  @Override\n  public void annotate(Annotation annotation) {\n    if (verbose) {\n      logger.info(\"Adding TokensRegexNER annotations ... \");\n    }\n\n    List<CoreMap> sentences = annotation.get(CoreAnnotations.SentencesAnnotation.class);\n    if (sentences != null) {\n      for (CoreMap sentence : sentences) {\n        List<CoreLabel> tokens = sentence.get(CoreAnnotations.TokensAnnotation.class);\n        annotateMatched(tokens);\n      }\n    } else {\n      List<CoreLabel> tokens = annotation.get(CoreAnnotations.TokensAnnotation.class);\n      if (tokens != null){\n        annotateMatched(tokens);\n      } else {\n        throw new RuntimeException(\"Unable to find sentences or tokens in \" + annotation);\n      }\n    }\n\n    if (verbose)\n      logger.info(\"done.\");\n  }\n\n  private MultiPatternMatcher<CoreMap> createPatternMatcher(Map<SequencePattern<CoreMap>, Entry> patternToEntry) {\n    // Convert to tokensregex pattern\n\n    List<TokenSequencePattern> patterns = new ArrayList<>(entries.size());\n    for (Entry entry:entries) {\n      TokenSequencePattern pattern;\n\n      Boolean ignoreCaseEntry = ignoreCaseList.get(entryToMappingFileNumber.get(entry));\n      int patternFlags = ignoreCaseEntry? Pattern.CASE_INSENSITIVE | Pattern.UNICODE_CASE:0;\n      int stringMatchFlags = ignoreCaseEntry? (NodePattern.CASE_INSENSITIVE | NodePattern.UNICODE_CASE):0;\n      Env env = TokenSequencePattern.getNewEnv();","sourceCodeStart":331,"sourceCodeEnd":367,"githubUrl":"https://github.com/stanfordnlp/CoreNLP/blob/1b7edd19c4d0d7b1f13a2591425b9b60a0b1af7a/src/edu/stanford/nlp/pipeline/TokensRegexNERAnnotator.java#L331-L367","documentation":"TokensRegexNERAnnotator.annotate() requires the input Annotation to carry tokens (CoreAnnotations.TokensAnnotation) or sentences; it throws a plain RuntimeException when neither is present. This means the annotation was never tokenized before this annotator ran, so there is nothing for the TokensRegex rules to match against.","triggerScenarios":"Calling annotate(Annotation) directly on a CoreMap/Annotation created programmatically without running tokenize first, or building a pipeline where TokensRegexNER runs before tokenize/ssa so annotation.get(TokensAnnotation.class) returns null.","commonSituations":"Custom StanfordCoreNLP pipeline property lists that omit 'tokenize,ssplit' before 'tokensregexner'; unit tests constructing a new Annotation(\"text\") and invoking the annotator directly; streaming code that passes partial CoreMaps lacking TokensAnnotation.","solutions":["Add 'tokenize' (and usually 'ssplit') to the pipeline properties so tokens exist before tokensregexner runs.","If annotating a single sentence/string manually, call the TokenizerAnnotator or annotate with a tokenize-ssplit pipeline first.","If constructing the Annotation programmatically, set CoreAnnotations.TokensAnnotation with a non-empty List<CoreLabel> before calling annotate().","Wrap the call in try-catch for RuntimeException and re-run with tokenization when tokens are missing."],"exampleFix":"// before\nProperties props = new Properties();\nprops.setProperty(\"annotators\", \"tokensregexner\");\n// after\nProperties props = new Properties();\nprops.setProperty(\"annotators\", \"tokenize,ssplit,tokensregexner\");","handlingStrategy":"validation","validationCode":"if (annotation.get(CoreAnnotations.TokensAnnotation.class) == null && annotation.get(CoreAnnotations.SentencesAnnotation.class) == null) {\n  throw new IllegalStateException(\"Run tokenize/ssplit before TokensRegexNERAnnotator\");\n}","typeGuard":null,"tryCatchPattern":"try { annotator.annotate(annotation); } catch (RuntimeException e) { if (e.getMessage().startsWith(\"Unable to find sentences or tokens\")) { pipeline.annotate(annotation); annotator.annotate(annotation); } else throw e; }","preventionTips":["Always include 'tokenize,ssplit' before 'tokensregexner' in the annotators list","In unit tests, use a full StanfordCoreNLP pipeline rather than calling the annotator directly on a bare Annotation","Assert TokensAnnotation is present before invoking annotation-stage annotators"],"tags":["nlp","annotation-pipeline","missing-tokens","runtime-exception"],"backgroundTag":"missing-required-argument","analyzedSha":"1b7edd19c4d0d7b1f13a2591425b9b60a0b1af7a","analyzedAt":"2026-09-10T02:24:07.274Z","contentChangedAt":"2026-09-10T02:24:07.274Z","schemaVersion":2},"datasetVersion":"2026-09-17T15:17:12.973Z"}