stanfordnlp/CoreNLP · error · IllegalArgumentException

Array lengths don't match

Error message

Array lengths don't match: ${words.size()} vs ${lemmas.size()} (sentence ${sentenceid})

What it means

TSVUtils.parseSentence builds a sentence from parallel column lists (words, lemmas, pos, ner). Before constructing, it validates that each list has the same length as words; mismatched lemmas (or pos/ner) trigger this IllegalArgumentException, naming both sizes and the sentence id (or ??? if absent).

Solutions

Find the sentence identified in the message and fix the source row so every column has the same field count.
Pre-validate each line by splitting on the delimiter and asserting equal field counts before calling parseSentence.
Decide on a policy for missing lemmas (fill with the word itself or "_") instead of emitting empty fields.

Example fix

// before
parseSentence(tree, maltTree, words, lemmas, pos, ner, sentenceid);
// after
if (lemmas.size() != words.size()) {
  lemmas = padOrTrim(lemmas, words.size());
}
parseSentence(tree, maltTree, words, lemmas, pos, ner, sentenceid);

Defensive patterns

Strategy: validation

Validate before calling

if (lemmas.size() != words.size() || pos.size() != words.size() || ner.size() != words.size()) {
  throw new IllegalArgumentException("Column sizes differ: words=" + words.size() + " lemmas=" + lemmas.size() + " pos=" + pos.size() + " ner=" + ner.size());
}

Try / catch

try {
  parseSentence(tree, maltTree, words, lemmas, pos, ner, sentenceid);
} catch (IllegalArgumentException e) {
  if (e.getMessage().startsWith("Array lengths don't match")) { logSkip(sentenceid, e); return null; }
  throw e;
}

Prevention

Validate equal field counts on every line before parsing the corpus.
Normalize missing annotation fields to '_' or copy the word instead of leaving empty fields.
Include sentence ids in data so malformed rows are easy to locate from the message.

When it happens

Trigger: Parsing a TSV/CoNLL-style file where a row has fewer or more lemma fields than word fields, e.g. a lemma column containing a literal tab or a missing trailing tab; calling parseSentence directly with lists of unequal size.

Common situations: Malformed or truncated lines in CoNLL/TSV corpora; regenerated files where one annotation column was dropped or a splitter wrote empty fields inconsistently; sentence id available in data helps locate the offending line.

Related errors

AI-assisted analysis of stanfordnlp/CoreNLP@1b7edd19c4 (2026-09-10). Data as JSON: /api/errors/005fadaedf548ac4. Report an issue: GitHub.

Appendix: source

Thrown at src/edu/stanford/nlp/process/TSVUtils.java:247

          tree.addVertex(governor);
        }
        if (!"ref".equals(relation)) {
          tree.addEdge(governor, dependent, GrammaticalRelation.valueOf(Language.English, relation), Double.NEGATIVE_INFINITY, false);
        }
      }
    }
    return tree;
  }

  /** Create an Annotation object (with a single sentence) from the given specification. */
  private static Annotation parseSentence(Optional<String> docid, Optional<Integer> sentenceIndex, String gloss,
                                          Function<List<CoreLabel>,SemanticGraph> tree,
                                          Function<List<CoreLabel>,SemanticGraph> maltTree,
                                          List<String> words, List<String> lemmas, List<String> pos, List<String> ner,
                                          Optional<String> sentenceid) {
    // Error checks
    if (lemmas.size() != words.size()) {
      throw new IllegalArgumentException("Array lengths don't match: " + words.size() + " vs " + lemmas.size() + " (sentence " + sentenceid.orElse("???") +")");
    }
    if (pos.size() != words.size()) {
      throw new IllegalArgumentException("Array lengths don't match: " + words.size() + " vs " + pos.size() + " (sentence " + sentenceid.orElse("???") +")");
    }
    if (ner.size() != words.size()) {
      throw new IllegalArgumentException("Array lengths don't match: " + words.size() + " vs " + ner.size() + " (sentence " + sentenceid.orElse("???") +")");
    }

    // Create structure
    List<CoreLabel> tokens = new ArrayList<>(words.size());
    int beginChar = 0;
    for (int i = 0; i < words.size(); ++i) {
      CoreLabel token = new CoreLabel(12);
      token.setWord(words.get(i));
      token.setValue(words.get(i));
      token.setBeginPosition(beginChar);
      token.setEndPosition(beginChar + words.get(i).length());
      beginChar += words.get(i).length() + 1;

View on GitHub (pinned to 1b7edd19c4)