stanfordnlp/CoreNLP · error · IllegalArgumentException

TokensRegexNERAnnotator ERROR: Header does not contain…

Q: What causes "TokensRegexNERAnnotator ERROR: Header does not contain 'pattern': " in stanfordnlp/CoreNLP?

Add a 'pattern' column to the header row of the mapping file and provide a TokensRegex pattern in each row.

Error message

TokensRegexNERAnnotator ERROR: Header does not contain 'pattern':

What it means

readEntries requires the mapping file header to include a 'pattern' column (PATTERN_FIELD); if getIndex returns < 0 it throws IllegalArgumentException noting the header doesn't contain 'pattern'. The pattern column is mandatory because every entry is driven by a TokensRegex pattern.

Solutions

Add a 'pattern' column to the header row of the mapping file and provide a TokensRegex pattern in each row.
Check casing/spelling — the header field must be exactly 'pattern'.
If the file is a plain RegexNER mapping, either convert it or use the regexner annotator instead.
Compare against the shipped default mapping (edu/stanford/nlp/models/regexner/type_mappings.txt) for the expected format.

Example fix

// header before
word	ner
// header after
pattern	ner
[ { word:/(?i:microsoft)/ } ]	ORGANIZATION

Defensive patterns

Strategy: validation

Validate before calling

String[] header = firstLine.split("\t", -1);
if (!Arrays.asList(header).contains("pattern")) {
  throw new IllegalStateException("Mapping header missing required 'pattern' column: " + String.join(",", header));
}

Try / catch

try { annotator = new TokensRegexNERAnnotator(name, props); } catch (IllegalArgumentException e) { if (e.getMessage().contains("Header does not contain 'pattern'")) { convertToTokensRegexFormat(mappingPath); } else throw e; }

Prevention

Start every mapping file with a header line containing at least 'pattern' and the annotation field (default 'ner')
Don't reuse plain regexner mapping files with tokensregexner without conversion
Validate headers at deploy time with a startup check

When it happens

Trigger: Providing a mapping file whose header row lacks 'pattern' — e.g. headers like 'word ner' copied from the plain RegexNERAnnotator format instead of the TokensRegexNER format.

Common situations: Reusing a regexner mapping file with tokensregexner; header written with different casing or whitespace ('Pattern'); header accidentally deleted when editing the file.

Understand the failure class

Background: "is required", "must be set", "missing required field": configuration validation errors across open-source libraries — this error's family across 36 libraries.

Related errors

AI-assisted analysis of stanfordnlp/CoreNLP@1b7edd19c4 (2026-09-10). Data as JSON: /api/errors/724122b7b661007c. Report an issue: GitHub.

Appendix: source

Thrown at src/edu/stanford/nlp/pipeline/TokensRegexNERAnnotator.java:673

   *  @return the updated list of Entries
   */
  private static List<Entry> readEntries(String annotatorName,
                                         String[] headerFields,
                                         String[] annotationFieldnames,
                                         List<Entry> entries,
                                         TrieMap<String,Entry> seenRegexes,
                                         String mappingFilename,
                                         BufferedReader mapping,
                                         Set<String> noDefaultOverwriteLabels,
                                         boolean ignoreCase, Integer mappingFileIndex,
                                         Map<Entry, Integer> entryToMappingFileNumber, boolean verbose) throws IOException {
    int origEntriesSize = entries.size();
    int isTokensRegex = 0;
    int lineCount = 0;
    Map<String,Integer> headerIndexMap = getHeaderIndexMap(headerFields);
    int iPattern = getIndex(headerIndexMap, PATTERN_FIELD);
    if (iPattern < 0) {
      throw new IllegalArgumentException("TokensRegexNERAnnotator " + annotatorName
        + " ERROR: Header does not contain 'pattern': " + StringUtils.join(headerFields));
    }
    int iOverwrite = getIndex(headerIndexMap, OVERWRITE_FIELD);
    int iPriority = getIndex(headerIndexMap, PRIORITY_FIELD);
    int iWeight = getIndex(headerIndexMap, WEIGHT_FIELD);
    int iGroup = getIndex(headerIndexMap, GROUP_FIELD);
    int[] annotationCols = new int[annotationFieldnames.length];
    int iLastAnnotationField = -1;
    for (int i = 0; i < annotationFieldnames.length; i++) {
      annotationCols[i] = getIndex(headerIndexMap, annotationFieldnames[i]);
      if (annotationCols[i] < 0) {
        throw new IllegalArgumentException("TokensRegexNERAnnotator " + annotatorName
            + " ERROR: Header does not contain annotation field '" + annotationFieldnames[i] + "': " + StringUtils.join(headerFields));
      }
      if (annotationCols[i] > iLastAnnotationField) {
        iLastAnnotationField = annotationCols[i];
      }
    }

View on GitHub (pinned to 1b7edd19c4)