{"record":{"id":"005fadaedf548ac4","repo":"stanfordnlp/CoreNLP","slug":"array-lengths-don-t-match-words-size-vs-le","errorCode":null,"errorMessage":"Array lengths don't match: ${words.size()} vs ${lemmas.size()} (sentence ${sentenceid})","messagePattern":"Array lengths don't match: (.+?) vs (.+?) \\(sentence (.+?)\\)","errorType":"validation","errorClass":"IllegalArgumentException","httpStatus":null,"severity":"error","filePath":"src/edu/stanford/nlp/process/TSVUtils.java","lineNumber":247,"sourceCode":"          tree.addVertex(governor);\n        }\n        if (!\"ref\".equals(relation)) {\n          tree.addEdge(governor, dependent, GrammaticalRelation.valueOf(Language.English, relation), Double.NEGATIVE_INFINITY, false);\n        }\n      }\n    }\n    return tree;\n  }\n\n  /** Create an Annotation object (with a single sentence) from the given specification. */\n  private static Annotation parseSentence(Optional<String> docid, Optional<Integer> sentenceIndex, String gloss,\n                                          Function<List<CoreLabel>,SemanticGraph> tree,\n                                          Function<List<CoreLabel>,SemanticGraph> maltTree,\n                                          List<String> words, List<String> lemmas, List<String> pos, List<String> ner,\n                                          Optional<String> sentenceid) {\n    // Error checks\n    if (lemmas.size() != words.size()) {\n      throw new IllegalArgumentException(\"Array lengths don't match: \" + words.size() + \" vs \" + lemmas.size() + \" (sentence \" + sentenceid.orElse(\"???\") +\")\");\n    }\n    if (pos.size() != words.size()) {\n      throw new IllegalArgumentException(\"Array lengths don't match: \" + words.size() + \" vs \" + pos.size() + \" (sentence \" + sentenceid.orElse(\"???\") +\")\");\n    }\n    if (ner.size() != words.size()) {\n      throw new IllegalArgumentException(\"Array lengths don't match: \" + words.size() + \" vs \" + ner.size() + \" (sentence \" + sentenceid.orElse(\"???\") +\")\");\n    }\n\n    // Create structure\n    List<CoreLabel> tokens = new ArrayList<>(words.size());\n    int beginChar = 0;\n    for (int i = 0; i < words.size(); ++i) {\n      CoreLabel token = new CoreLabel(12);\n      token.setWord(words.get(i));\n      token.setValue(words.get(i));\n      token.setBeginPosition(beginChar);\n      token.setEndPosition(beginChar + words.get(i).length());\n      beginChar += words.get(i).length() + 1;","sourceCodeStart":229,"sourceCodeEnd":265,"githubUrl":"https://github.com/stanfordnlp/CoreNLP/blob/1b7edd19c4d0d7b1f13a2591425b9b60a0b1af7a/src/edu/stanford/nlp/process/TSVUtils.java#L229-L265","documentation":"TSVUtils.parseSentence builds a sentence from parallel column lists (words, lemmas, pos, ner). Before constructing, it validates that each list has the same length as words; mismatched lemmas (or pos/ner) trigger this IllegalArgumentException, naming both sizes and the sentence id (or ??? if absent).","triggerScenarios":"Parsing a TSV/CoNLL-style file where a row has fewer or more lemma fields than word fields, e.g. a lemma column containing a literal tab or a missing trailing tab; calling parseSentence directly with lists of unequal size.","commonSituations":"Malformed or truncated lines in CoNLL/TSV corpora; regenerated files where one annotation column was dropped or a splitter wrote empty fields inconsistently; sentence id available in data helps locate the offending line.","solutions":["Find the sentence identified in the message and fix the source row so every column has the same field count.","Pre-validate each line by splitting on the delimiter and asserting equal field counts before calling parseSentence.","Decide on a policy for missing lemmas (fill with the word itself or \"_\") instead of emitting empty fields."],"exampleFix":"// before\nparseSentence(tree, maltTree, words, lemmas, pos, ner, sentenceid);\n// after\nif (lemmas.size() != words.size()) {\n  lemmas = padOrTrim(lemmas, words.size());\n}\nparseSentence(tree, maltTree, words, lemmas, pos, ner, sentenceid);","handlingStrategy":"validation","validationCode":"if (lemmas.size() != words.size() || pos.size() != words.size() || ner.size() != words.size()) {\n  throw new IllegalArgumentException(\"Column sizes differ: words=\" + words.size() + \" lemmas=\" + lemmas.size() + \" pos=\" + pos.size() + \" ner=\" + ner.size());\n}","typeGuard":null,"tryCatchPattern":"try {\n  parseSentence(tree, maltTree, words, lemmas, pos, ner, sentenceid);\n} catch (IllegalArgumentException e) {\n  if (e.getMessage().startsWith(\"Array lengths don't match\")) { logSkip(sentenceid, e); return null; }\n  throw e;\n}","preventionTips":["Validate equal field counts on every line before parsing the corpus.","Normalize missing annotation fields to '_' or copy the word instead of leaving empty fields.","Include sentence ids in data so malformed rows are easy to locate from the message."],"tags":["parsing","validation","tsv"],"backgroundTag":"array-length-mismatch","analyzedSha":"1b7edd19c4d0d7b1f13a2591425b9b60a0b1af7a","analyzedAt":"2026-09-10T02:24:07.274Z","contentChangedAt":"2026-09-10T02:24:07.274Z","schemaVersion":2},"datasetVersion":"2026-09-17T15:17:12.973Z"}