stanfordnlp/CoreNLP · error · IllegalArgumentException

Array lengths don't match

Error message

Array lengths don't match: ${words.size()} vs ${pos.size()} (sentence ${sentenceid})

What it means

TSVUtils.parseSentence validates that every per-token column (words, lemmas, pos, ner) has the same length before building CoreLabel tokens. This throw fires when the POS-tag column has fewer or more entries than the words column, which would otherwise silently misalign annotations. It is a data-integrity guard for TSV sentence parsing.

Solutions

Re-encode the TSV so every sentence row has exactly the same number of fields in words, lemma, pos, and ner columns
Check for rows with missing tabs/fields with awk or a preprocessor before parsing
If POS data is unavailable, fill each row with a placeholder such as '_' or 'X' to keep lengths equal
Verify the file's field separator matches what TSVUtils expects (tab, not spaces)

Example fix

// before (uneven columns)
the	dog	NN
barks
// after
the	the	_	_
dog	dog	NN	_
barks	bark	VBZ	_

Defensive patterns

Strategy: validation

Validate before calling

if (words.size() != pos.size() || words.size() != lemmas.size() || words.size() != ner.size()) {
  throw new IllegalArgumentException("TSV sentence has mismatched column lengths: words=" + words.size() + " pos=" + pos.size());
}

Type guard

boolean columnsAligned(List<?>... cols) {
  return cols.length == 0 || Arrays.stream(cols).allMatch(c -> c.size() == cols[0].size());
}

Try / catch

try {
  TSVUtils.parseSentence(words, lemmas, pos, ner, sentenceid);
} catch (IllegalArgumentException e) {
  log.error("Skipping malformed sentence " + sentenceid + ": " + e.getMessage());
}

Prevention

Validate every TSV row has a fixed column count before parsing
Pad missing annotation fields with '_' or 'O' placeholders at export time
Split rows strictly on tabs, never on whitespace

When it happens

Trigger: Calling parseSentence (or TSVUtils.tsvToRedwood) with a TSV row/sentence whose pos column list differs in size from the words list — e.g. blank POS fields skipped, tabs miscounted, or extra fields in a row.

Common situations: Malformed CoNLL-style TSV files where some rows have missing or extra columns; exporting data from another tool that omits POS for punctuation; splitting rows on whitespace when fields contain spaces.

Understand the failure class

Background: Schema validation failed / invalid input schema: payload rejected because its shape doesn't match the expected schema — this error's family across 28 libraries.

Related errors

AI-assisted analysis of stanfordnlp/CoreNLP@1b7edd19c4 (2026-09-10). Data as JSON: /api/errors/74a16ffab794d367. Report an issue: GitHub.

Appendix: source

Thrown at src/edu/stanford/nlp/process/TSVUtils.java:250

          tree.addEdge(governor, dependent, GrammaticalRelation.valueOf(Language.English, relation), Double.NEGATIVE_INFINITY, false);
        }
      }
    }
    return tree;
  }

  /** Create an Annotation object (with a single sentence) from the given specification. */
  private static Annotation parseSentence(Optional<String> docid, Optional<Integer> sentenceIndex, String gloss,
                                          Function<List<CoreLabel>,SemanticGraph> tree,
                                          Function<List<CoreLabel>,SemanticGraph> maltTree,
                                          List<String> words, List<String> lemmas, List<String> pos, List<String> ner,
                                          Optional<String> sentenceid) {
    // Error checks
    if (lemmas.size() != words.size()) {
      throw new IllegalArgumentException("Array lengths don't match: " + words.size() + " vs " + lemmas.size() + " (sentence " + sentenceid.orElse("???") +")");
    }
    if (pos.size() != words.size()) {
      throw new IllegalArgumentException("Array lengths don't match: " + words.size() + " vs " + pos.size() + " (sentence " + sentenceid.orElse("???") +")");
    }
    if (ner.size() != words.size()) {
      throw new IllegalArgumentException("Array lengths don't match: " + words.size() + " vs " + ner.size() + " (sentence " + sentenceid.orElse("???") +")");
    }

    // Create structure
    List<CoreLabel> tokens = new ArrayList<>(words.size());
    int beginChar = 0;
    for (int i = 0; i < words.size(); ++i) {
      CoreLabel token = new CoreLabel(12);
      token.setWord(words.get(i));
      token.setValue(words.get(i));
      token.setBeginPosition(beginChar);
      token.setEndPosition(beginChar + words.get(i).length());
      beginChar += words.get(i).length() + 1;
      token.setLemma(lemmas.get(i));
      token.setTag(pos.get(i));
      token.setNER(ner.get(i));

View on GitHub (pinned to 1b7edd19c4)