{"record":{"id":"a774d91a774972f7","repo":"stanfordnlp/CoreNLP","slug":"tokens-size-d-pos-size-d-n","errorCode":null,"errorMessage":"tokens.size(): %d != pos.size(): %d%n","messagePattern":"tokens\\.size\\(\\): (.+?) != pos\\.size\\(\\): (.+?)%n","errorType":"exception","errorClass":"RuntimeException","httpStatus":null,"severity":"error","filePath":"src/edu/stanford/nlp/trees/GrammaticalStructure.java","lineNumber":338,"sourceCode":"     throw new RuntimeException(String.format(\"Dependencies should be for the format 'type(arg-idx, arg-idx)'. Could not parse '%s'\", dep));\n  }\n\n  /**\n   * Create a grammatical structure from its string representation.\n   *\n   * Like buildCoNLLXGrammaticalStructure,\n   * this method fakes up the parts of the tree structure that are not\n   * used by the grammatical relation transformation operations.\n   *\n   * <i>Note:</i> Added by daniel cer\n   *\n   * @param tokens\n   * @param posTags\n   * @param deps\n   */\n  public static GrammaticalStructure fromStringReps(List<String> tokens, List<String> posTags, List<String> deps) {\n    if (tokens.size() != posTags.size()) {\n      throw new RuntimeException(String.format(\n              \"tokens.size(): %d != pos.size(): %d%n\", tokens.size(), posTags\n                      .size()));\n    }\n\n    List<TreeGraphNode> tgWordNodes = new ArrayList<>(tokens.size());\n    List<TreeGraphNode> tgPOSNodes = new ArrayList<>(tokens.size());\n\n    CoreLabel rootLabel = new CoreLabel();\n    rootLabel.setValue(\"ROOT\");\n    List<IndexedWord> nodeWords = new ArrayList<>(tgPOSNodes.size() + 1);\n    nodeWords.add(new IndexedWord(rootLabel));\n\n    UniversalSemanticHeadFinder headFinder = new UniversalSemanticHeadFinder();\n\n    Iterator<String> posIter = posTags.iterator();\n    for (String wordString : tokens) {\n      String posString = posIter.next();\n      CoreLabel wordLabel = new CoreLabel();","sourceCodeStart":320,"sourceCodeEnd":356,"githubUrl":"https://github.com/stanfordnlp/CoreNLP/blob/1b7edd19c4d0d7b1f13a2591425b9b60a0b1af7a/src/edu/stanford/nlp/trees/GrammaticalStructure.java#L320-L356","documentation":"fromStringReps requires parallel lists: one entry per token in tokens and one POS tag per token in posTags. If the sizes differ, it throws this RuntimeException reporting both sizes, because it would otherwise zip mismatched word/tag pairs into the tree. It is a strict input-consistency check before building the TreeGraphNode word/POS lists.","triggerScenarios":"Calling GrammaticalStructure.fromStringReps(tokens, posTags, deps) where tokens.size() != posTags.size() — e.g. POS tags that include or omit punctuation/indices, tokens split differently than tags, or an off-by-one when building the lists.","commonSituations":"Reading tokens and tags from different preprocessing steps with different tokenizations; including the ROOT pseudo-item in one list but not the other; merging outputs of a tokenizer and tagger that disagree on contractions/hyphens.","solutions":["Ensure both lists come from the same tokenization and have equal size before calling fromStringReps","Add an assertion/log listing token-tag pairs to spot where the lists diverge","Align punctuation and multi-word tokens (e.g. split \"don't\" consistently on both sides)","Use a single Sentence/tokenizer pipeline to produce tokens and posTags so they stay in sync"],"exampleFix":"// before\nfromStringReps(Arrays.asList(\"I\",\"saw\"), Arrays.asList(\"PRP\",\"VBD\",\".\") /* extra '.' */, deps);\n// after\nfromStringReps(Arrays.asList(\"I\",\"saw\",\".\"), Arrays.asList(\"PRP\",\"VBD\",\".\"), deps);","handlingStrategy":"validation","validationCode":"if (tokens.size() != posTags.size()) {\n    throw new IllegalArgumentException(\"tokens/posTags size mismatch: \" + tokens.size() + \" vs \" + posTags.size());\n}","typeGuard":null,"tryCatchPattern":"try {\n    GrammaticalStructure gs = GrammaticalStructure.fromStringReps(tokens, posTags, deps);\n} catch (RuntimeException e) {\n    if (e.getMessage().startsWith(\"tokens.size():\")) {\n        // re-tokenize so tokens and tags align, then retry\n    } else throw e;\n}","preventionTips":["Derive tokens and POS tags from the same tokenizer/tagger run","Check list sizes before every fromStringReps call","Watch for punctuation and contraction splitting differences between tools","Pair each token with its tag in one data structure instead of parallel lists"],"tags":["java","input-validation","mismatch","dependency-parsing"],"backgroundTag":"value-out-of-range","analyzedSha":"1b7edd19c4d0d7b1f13a2591425b9b60a0b1af7a","analyzedAt":"2026-09-10T02:24:07.274Z","contentChangedAt":"2026-09-10T02:24:07.274Z","schemaVersion":2},"datasetVersion":"2026-09-17T15:17:12.973Z"}