stanfordnlp/CoreNLP · error · IllegalArgumentException

Array lengths don't match

Error message

Array lengths don't match: ${words.size()} vs ${ner.size()} (sentence ${sentenceid})

What it means

TSVUtils.parseSentence requires the NER column to have exactly as many entries as the words list. When sizes differ, tokens could get NER labels misassigned, so the method throws before building tokens. Same family of length checks as the lemma/pos checks immediately above.

Solutions

Pad or trim the NER column so it has one entry per word (use 'O' for no-entity rows)
Normalize all sentences to the same column count before parsing
Pre-validate each sentence's column sizes in your ingestion code
Re-export the dataset with all annotation columns present

Example fix

// before
List<String> ner = readColumn(rows, 3); // some rows missing col 3
// after
while (ner.size() < words.size()) ner.add("O");

Defensive patterns

Strategy: validation

Validate before calling

if (ner.size() != words.size()) {
  while (ner.size() < words.size()) ner.add("O");
}

Try / catch

try {
  TSVUtils.parseSentence(words, lemmas, pos, ner, sentenceid);
} catch (IllegalArgumentException e) {
  log.warn("Dropping sentence with misaligned NER column: " + sentenceid);
}

Prevention

Ensure NER column is exported for every row, using 'O' when absent
Check column counts per sentence in a pre-parse audit step
Avoid concatenating files with differing column schemas

When it happens

Trigger: Calling parseSentence with an ner list whose size differs from words.size() — typically a TSV sentence where some rows lack the NER column or have extra columns.

Common situations: TSV files that only partially include NER annotations; concatenating sentences where some have a NER column and some do not; off-by-one row splitting when reading the file.

Understand the failure class

Background: Schema validation failed / invalid input schema: payload rejected because its shape doesn't match the expected schema — this error's family across 28 libraries.

Related errors

AI-assisted analysis of stanfordnlp/CoreNLP@1b7edd19c4 (2026-09-10). Data as JSON: /api/errors/cc521f970fa714f3. Report an issue: GitHub.

Appendix: source

Thrown at src/edu/stanford/nlp/process/TSVUtils.java:253

    }
    return tree;
  }

  /** Create an Annotation object (with a single sentence) from the given specification. */
  private static Annotation parseSentence(Optional<String> docid, Optional<Integer> sentenceIndex, String gloss,
                                          Function<List<CoreLabel>,SemanticGraph> tree,
                                          Function<List<CoreLabel>,SemanticGraph> maltTree,
                                          List<String> words, List<String> lemmas, List<String> pos, List<String> ner,
                                          Optional<String> sentenceid) {
    // Error checks
    if (lemmas.size() != words.size()) {
      throw new IllegalArgumentException("Array lengths don't match: " + words.size() + " vs " + lemmas.size() + " (sentence " + sentenceid.orElse("???") +")");
    }
    if (pos.size() != words.size()) {
      throw new IllegalArgumentException("Array lengths don't match: " + words.size() + " vs " + pos.size() + " (sentence " + sentenceid.orElse("???") +")");
    }
    if (ner.size() != words.size()) {
      throw new IllegalArgumentException("Array lengths don't match: " + words.size() + " vs " + ner.size() + " (sentence " + sentenceid.orElse("???") +")");
    }

    // Create structure
    List<CoreLabel> tokens = new ArrayList<>(words.size());
    int beginChar = 0;
    for (int i = 0; i < words.size(); ++i) {
      CoreLabel token = new CoreLabel(12);
      token.setWord(words.get(i));
      token.setValue(words.get(i));
      token.setBeginPosition(beginChar);
      token.setEndPosition(beginChar + words.get(i).length());
      beginChar += words.get(i).length() + 1;
      token.setLemma(lemmas.get(i));
      token.setTag(pos.get(i));
      token.setNER(ner.get(i));
      token.set(CoreAnnotations.DocIDAnnotation.class, docid.orElse("???"));
      token.set(CoreAnnotations.SentenceIndexAnnotation.class, sentenceIndex.orElse(-1));
      token.set(CoreAnnotations.IndexAnnotation.class, i + 1);

View on GitHub (pinned to 1b7edd19c4)