{"record":{"id":"c74c2f8e4e5c1ae0","repo":"stanfordnlp/CoreNLP","slug":"originalwords-and-sentence-of-different-sizes","errorCode":null,"errorMessage":"originalWords and sentence of different sizes: ","messagePattern":"originalWords and sentence of different sizes: ","errorType":"exception","errorClass":"IllegalStateException","httpStatus":null,"severity":"error","filePath":"src/edu/stanford/nlp/parser/lexparser/LexicalizedParserQuery.java","lineNumber":315,"sourceCode":"      if ( ! bparser.parse(sentenceB)) {\n        return parseSucceeded;\n      } else {\n        parseSucceeded = true;\n      }\n    }\n    return true;\n  }\n\n\n  @Override\n  public void restoreOriginalWords(Tree tree) {\n    if (originalSentence == null || tree == null) {\n      return;\n    }\n    List<Tree> leaves = tree.getLeaves();\n    int expectedSize = addedPunct ? originalSentence.size() + 1 : originalSentence.size();\n    if (leaves.size() != expectedSize) {\n      throw new IllegalStateException(\"originalWords and sentence of different sizes: \" + expectedSize + \" vs. \" + leaves.size() +\n                                      \"\\n Orig: \" + SentenceUtils.listToString(originalSentence) +\n                                      \"\\n Pars: \" + SentenceUtils.listToString(leaves));\n    }\n    Iterator<Tree> leafIterator = leaves.iterator();\n    for (HasWord word : originalSentence) {\n      Tree leaf = leafIterator.next();\n      if (!(word instanceof Label)) {\n        continue;\n      }\n      leaf.setLabel((Label) word);\n    }\n  }\n\n\n  /**\n   * Parse a (speech) lattice with the PCFG parser.\n   *\n   * @param lr a lattice to parse","sourceCodeStart":297,"sourceCodeEnd":333,"githubUrl":"https://github.com/stanfordnlp/CoreNLP/blob/1b7edd19c4d0d7b1f13a2591425b9b60a0b1af7a/src/edu/stanford/nlp/parser/lexparser/LexicalizedParserQuery.java#L297-L333","documentation":"restoreOriginalWords maps the parse tree's leaves back onto the caller's original word list (undoing punctuation added during preprocessing). If the number of tree leaves does not equal the expected count (original sentence size, plus 1 if final punctuation was added), an IllegalStateException describing the two sizes and both lists is thrown — an internal invariant violation.","triggerScenarios":"Calling getBestParse/getKBestParses/etc. after parse when the tree returned by the parsers has a different leaf count than the (possibly punctuation-augmented) input — e.g. the parser dropped or inserted tokens, or restoreOriginalWords is invoked twice / on a mismatched tree.","commonSituations":"Custom token filters or annotators mutating the sentence between parse and result retrieval; reusing a ParserQuery result after parsing a different sentence; manually modifying the returned tree before calling result accessors; a grammar/tokenizer that normalizes tokens unexpectedly.","solutions":["Don't mutate the input sentence list or the returned tree between parse() and the result accessors (getBestParse, etc.)","Call each result method only once per parse on the same ParserQuery; re-parse for a new sentence","Inspect the Orig/Pars dumps in the message: if punctuation was added (addedPunct), expect leaves = originalSentence.size() + 1","Disable op.testOptions.addMissingFinalPunctuation if your pipeline already guarantees final punctuation, removing the count ambiguity","Report upstream if the parser itself drops tokens (e.g. filter-only grammar); verify with a vanilla model on the same input"],"exampleFix":"// before\nTree t = parserQuery.getBestParse();\nt = myTransform(t);          // mutates leaves\nparserQuery.getKBestParses(); // size mismatch\n// after\nTree t = parserQuery.getBestParse();\nList<List<Tree>> kBest = parserQuery.getKBestParses(); // collect all results first\nTree transformed = myTransform(t);","handlingStrategy":"try-catch","validationCode":"// Don't mutate inputs; verify no code path reuses the query across sentences\nassert parserQuery != null && !treeAlreadyConsumed;","typeGuard":null,"tryCatchPattern":"try {\n  Tree t = parserQuery.getBestParse();\n} catch (IllegalStateException e) {\n  if (e.getMessage().startsWith(\"originalWords and sentence of different sizes\")) {\n    // re-parse fresh or return raw tree without restoring original words\n  }\n  throw e;\n}","preventionTips":["Treat ParserQuery results as one-shot: one parse, then read results","Never modify the input word list or result tree between parse and accessors","If adding final punctuation, account for the +1 leaf in any leaf-count checks"],"tags":["parser","invariant-violation","state-management","java"],"backgroundTag":"internal-invariant-violation","analyzedSha":"1b7edd19c4d0d7b1f13a2591425b9b60a0b1af7a","analyzedAt":"2026-09-10T02:24:07.274Z","contentChangedAt":"2026-09-10T02:24:07.274Z","schemaVersion":2},"datasetVersion":"2026-09-17T15:17:12.973Z"}