{"record":{"id":"cc521f970fa714f3","repo":"stanfordnlp/CoreNLP","slug":"array-lengths-don-t-match-words-size-vs-ne","errorCode":null,"errorMessage":"Array lengths don't match: ${words.size()} vs ${ner.size()} (sentence ${sentenceid})","messagePattern":"Array lengths don't match: (.+?) vs (.+?) \\(sentence (.+?)\\)","errorType":"validation","errorClass":"IllegalArgumentException","httpStatus":null,"severity":"error","filePath":"src/edu/stanford/nlp/process/TSVUtils.java","lineNumber":253,"sourceCode":"    }\n    return tree;\n  }\n\n  /** Create an Annotation object (with a single sentence) from the given specification. */\n  private static Annotation parseSentence(Optional<String> docid, Optional<Integer> sentenceIndex, String gloss,\n                                          Function<List<CoreLabel>,SemanticGraph> tree,\n                                          Function<List<CoreLabel>,SemanticGraph> maltTree,\n                                          List<String> words, List<String> lemmas, List<String> pos, List<String> ner,\n                                          Optional<String> sentenceid) {\n    // Error checks\n    if (lemmas.size() != words.size()) {\n      throw new IllegalArgumentException(\"Array lengths don't match: \" + words.size() + \" vs \" + lemmas.size() + \" (sentence \" + sentenceid.orElse(\"???\") +\")\");\n    }\n    if (pos.size() != words.size()) {\n      throw new IllegalArgumentException(\"Array lengths don't match: \" + words.size() + \" vs \" + pos.size() + \" (sentence \" + sentenceid.orElse(\"???\") +\")\");\n    }\n    if (ner.size() != words.size()) {\n      throw new IllegalArgumentException(\"Array lengths don't match: \" + words.size() + \" vs \" + ner.size() + \" (sentence \" + sentenceid.orElse(\"???\") +\")\");\n    }\n\n    // Create structure\n    List<CoreLabel> tokens = new ArrayList<>(words.size());\n    int beginChar = 0;\n    for (int i = 0; i < words.size(); ++i) {\n      CoreLabel token = new CoreLabel(12);\n      token.setWord(words.get(i));\n      token.setValue(words.get(i));\n      token.setBeginPosition(beginChar);\n      token.setEndPosition(beginChar + words.get(i).length());\n      beginChar += words.get(i).length() + 1;\n      token.setLemma(lemmas.get(i));\n      token.setTag(pos.get(i));\n      token.setNER(ner.get(i));\n      token.set(CoreAnnotations.DocIDAnnotation.class, docid.orElse(\"???\"));\n      token.set(CoreAnnotations.SentenceIndexAnnotation.class, sentenceIndex.orElse(-1));\n      token.set(CoreAnnotations.IndexAnnotation.class, i + 1);","sourceCodeStart":235,"sourceCodeEnd":271,"githubUrl":"https://github.com/stanfordnlp/CoreNLP/blob/1b7edd19c4d0d7b1f13a2591425b9b60a0b1af7a/src/edu/stanford/nlp/process/TSVUtils.java#L235-L271","documentation":"TSVUtils.parseSentence requires the NER column to have exactly as many entries as the words list. When sizes differ, tokens could get NER labels misassigned, so the method throws before building tokens. Same family of length checks as the lemma/pos checks immediately above.","triggerScenarios":"Calling parseSentence with an ner list whose size differs from words.size() — typically a TSV sentence where some rows lack the NER column or have extra columns.","commonSituations":"TSV files that only partially include NER annotations; concatenating sentences where some have a NER column and some do not; off-by-one row splitting when reading the file.","solutions":["Pad or trim the NER column so it has one entry per word (use 'O' for no-entity rows)","Normalize all sentences to the same column count before parsing","Pre-validate each sentence's column sizes in your ingestion code","Re-export the dataset with all annotation columns present"],"exampleFix":"// before\nList<String> ner = readColumn(rows, 3); // some rows missing col 3\n// after\nwhile (ner.size() < words.size()) ner.add(\"O\");","handlingStrategy":"validation","validationCode":"if (ner.size() != words.size()) {\n  while (ner.size() < words.size()) ner.add(\"O\");\n}","typeGuard":null,"tryCatchPattern":"try {\n  TSVUtils.parseSentence(words, lemmas, pos, ner, sentenceid);\n} catch (IllegalArgumentException e) {\n  log.warn(\"Dropping sentence with misaligned NER column: \" + sentenceid);\n}","preventionTips":["Ensure NER column is exported for every row, using 'O' when absent","Check column counts per sentence in a pre-parse audit step","Avoid concatenating files with differing column schemas"],"tags":["java","data-integrity","tsv","ner"],"backgroundTag":"schema-validation-failed","analyzedSha":"1b7edd19c4d0d7b1f13a2591425b9b60a0b1af7a","analyzedAt":"2026-09-10T02:24:07.274Z","contentChangedAt":"2026-09-10T02:24:07.274Z","schemaVersion":2},"datasetVersion":"2026-09-17T15:17:12.973Z"}