{"record":{"id":"74a16ffab794d367","repo":"stanfordnlp/CoreNLP","slug":"array-lengths-don-t-match-words-size-vs-po","errorCode":null,"errorMessage":"Array lengths don't match: ${words.size()} vs ${pos.size()} (sentence ${sentenceid})","messagePattern":"Array lengths don't match: (.+?) vs (.+?) \\(sentence (.+?)\\)","errorType":"validation","errorClass":"IllegalArgumentException","httpStatus":null,"severity":"error","filePath":"src/edu/stanford/nlp/process/TSVUtils.java","lineNumber":250,"sourceCode":"          tree.addEdge(governor, dependent, GrammaticalRelation.valueOf(Language.English, relation), Double.NEGATIVE_INFINITY, false);\n        }\n      }\n    }\n    return tree;\n  }\n\n  /** Create an Annotation object (with a single sentence) from the given specification. */\n  private static Annotation parseSentence(Optional<String> docid, Optional<Integer> sentenceIndex, String gloss,\n                                          Function<List<CoreLabel>,SemanticGraph> tree,\n                                          Function<List<CoreLabel>,SemanticGraph> maltTree,\n                                          List<String> words, List<String> lemmas, List<String> pos, List<String> ner,\n                                          Optional<String> sentenceid) {\n    // Error checks\n    if (lemmas.size() != words.size()) {\n      throw new IllegalArgumentException(\"Array lengths don't match: \" + words.size() + \" vs \" + lemmas.size() + \" (sentence \" + sentenceid.orElse(\"???\") +\")\");\n    }\n    if (pos.size() != words.size()) {\n      throw new IllegalArgumentException(\"Array lengths don't match: \" + words.size() + \" vs \" + pos.size() + \" (sentence \" + sentenceid.orElse(\"???\") +\")\");\n    }\n    if (ner.size() != words.size()) {\n      throw new IllegalArgumentException(\"Array lengths don't match: \" + words.size() + \" vs \" + ner.size() + \" (sentence \" + sentenceid.orElse(\"???\") +\")\");\n    }\n\n    // Create structure\n    List<CoreLabel> tokens = new ArrayList<>(words.size());\n    int beginChar = 0;\n    for (int i = 0; i < words.size(); ++i) {\n      CoreLabel token = new CoreLabel(12);\n      token.setWord(words.get(i));\n      token.setValue(words.get(i));\n      token.setBeginPosition(beginChar);\n      token.setEndPosition(beginChar + words.get(i).length());\n      beginChar += words.get(i).length() + 1;\n      token.setLemma(lemmas.get(i));\n      token.setTag(pos.get(i));\n      token.setNER(ner.get(i));","sourceCodeStart":232,"sourceCodeEnd":268,"githubUrl":"https://github.com/stanfordnlp/CoreNLP/blob/1b7edd19c4d0d7b1f13a2591425b9b60a0b1af7a/src/edu/stanford/nlp/process/TSVUtils.java#L232-L268","documentation":"TSVUtils.parseSentence validates that every per-token column (words, lemmas, pos, ner) has the same length before building CoreLabel tokens. This throw fires when the POS-tag column has fewer or more entries than the words column, which would otherwise silently misalign annotations. It is a data-integrity guard for TSV sentence parsing.","triggerScenarios":"Calling parseSentence (or TSVUtils.tsvToRedwood) with a TSV row/sentence whose pos column list differs in size from the words list — e.g. blank POS fields skipped, tabs miscounted, or extra fields in a row.","commonSituations":"Malformed CoNLL-style TSV files where some rows have missing or extra columns; exporting data from another tool that omits POS for punctuation; splitting rows on whitespace when fields contain spaces.","solutions":["Re-encode the TSV so every sentence row has exactly the same number of fields in words, lemma, pos, and ner columns","Check for rows with missing tabs/fields with awk or a preprocessor before parsing","If POS data is unavailable, fill each row with a placeholder such as '_' or 'X' to keep lengths equal","Verify the file's field separator matches what TSVUtils expects (tab, not spaces)"],"exampleFix":"// before (uneven columns)\nthe\tdog\tNN\nbarks\n// after\nthe\tthe\t_\t_\ndog\tdog\tNN\t_\nbarks\tbark\tVBZ\t_","handlingStrategy":"validation","validationCode":"if (words.size() != pos.size() || words.size() != lemmas.size() || words.size() != ner.size()) {\n  throw new IllegalArgumentException(\"TSV sentence has mismatched column lengths: words=\" + words.size() + \" pos=\" + pos.size());\n}","typeGuard":"boolean columnsAligned(List<?>... cols) {\n  return cols.length == 0 || Arrays.stream(cols).allMatch(c -> c.size() == cols[0].size());\n}","tryCatchPattern":"try {\n  TSVUtils.parseSentence(words, lemmas, pos, ner, sentenceid);\n} catch (IllegalArgumentException e) {\n  log.error(\"Skipping malformed sentence \" + sentenceid + \": \" + e.getMessage());\n}","preventionTips":["Validate every TSV row has a fixed column count before parsing","Pad missing annotation fields with '_' or 'O' placeholders at export time","Split rows strictly on tabs, never on whitespace"],"tags":["java","data-integrity","tsv"],"backgroundTag":"schema-validation-failed","analyzedSha":"1b7edd19c4d0d7b1f13a2591425b9b60a0b1af7a","analyzedAt":"2026-09-10T02:24:07.274Z","contentChangedAt":"2026-09-10T02:24:07.274Z","schemaVersion":2},"datasetVersion":"2026-09-17T15:17:12.973Z"}