stanfordnlp/CoreNLP · error · IllegalArgumentException

Duplicate header field:

Error message

Duplicate header field:

What it means

getHeaderIndexMap builds a column-name-to-index map from the mapping file's header row and throws IllegalArgumentException when the same field name appears twice. Duplicate column names would make column resolution ambiguous, so it fails fast.

Solutions

  1. Open the mapping file and remove the duplicate column name from the header line.
  2. Ensure each header field appears exactly once; give repeated columns distinct names.
  3. Check the annotator's 'annotationFieldnames' option for duplicated field names if they're concatenated with the header.
  4. If files were merged, keep only one header line and merge the column lists manually.

Example fix

// header line before
pattern	priority	ner	ner
// header line after
pattern	priority	ner	description
Defensive patterns

Strategy: validation

Validate before calling

String[] header = firstLine.split("\t", -1);
Set<String> seen = new HashSet<>();
for (String h : header) { if (!seen.add(h.trim())) throw new IllegalStateException("Duplicate header field: " + h); }

Try / catch

try { annotator = new TokensRegexNERAnnotator(name, props); } catch (IllegalArgumentException e) { if (e.getMessage().startsWith("Duplicate header field")) { reloadDedupedMapping(); } else throw e; }

Prevention

When it happens

Trigger: A TokensRegexNER mapping file whose first line (header) lists the same column name twice, e.g. 'pattern,ner,ner,description'; also occurs when the configured annotationFieldnames contain a duplicate resolved against the header.

Common situations: Hand-edited mapping files where a column was duplicated when adding a new annotation field; concatenating two mapping files including two header rows without deduplicating columns.

Understand the failure class

Background: Schema validation failed / invalid input schema: payload rejected because its shape doesn't match the expected schema — this error's family across 28 libraries.

Related errors

AI-assisted analysis of stanfordnlp/CoreNLP@1b7edd19c4 (2026-09-10). Data as JSON: /api/errors/44b4d80772239564. Report an issue: GitHub.

Appendix: source

Thrown at src/edu/stanford/nlp/pipeline/TokensRegexNERAnnotator.java:637

      try (BufferedReader rd = IOUtils.readerFromString(mapping)){
        readEntries(annotatorName, headerList.get(mappingFileIndex), annotationFieldnames, entries, seenRegexes, mapping, rd, noDefaultOverwriteLabels, ignoreCaseList.get(mappingFileIndex), mappingFileIndex, entryToMappingFileNumber, verbose);
      } catch (IOException e) {
        throw new RuntimeIOException("Couldn't read TokensRegexNER from " + mapping, e);
      }
    }

    if (mappings.length != 1) {
      logger.log(annotatorName + ": Read " + entries.size() + " unique entries from " + mappings.length + " files");
    }
    return entries;
  }

  private static Map<String,Integer> getHeaderIndexMap(String[] headerFields) {
    Map<String,Integer> map = new HashMap<>();
    for (int i = 0; i < headerFields.length; i++) {
      String field = headerFields[i];
      if (map.containsKey(field)) {
        throw new IllegalArgumentException("Duplicate header field: " + field);
      }
      map.put(field,i);
    }
    return map;
  }


  private static int getIndex(Map<String,Integer> map, String name) {
    Integer index = map.get(name);
    if (index == null) return -1;
    else return index;
  }

  /**
   *  Reads a list of Entries from a mapping file and update the given entries.
   *  Line numbers start from 1.
   *
   *  @return the updated list of Entries

View on GitHub (pinned to 1b7edd19c4)