What causes "originalWords and sentence of different sizes: " in stanfordnlp/CoreNLP?

Don't mutate the input sentence list or the returned tree between parse() and the result accessors (getBestParse, etc.)

stanfordnlp/CoreNLP · error · IllegalStateException

originalWords and sentence of different sizes:

Error message

originalWords and sentence of different sizes:

What it means

restoreOriginalWords maps the parse tree's leaves back onto the caller's original word list (undoing punctuation added during preprocessing). If the number of tree leaves does not equal the expected count (original sentence size, plus 1 if final punctuation was added), an IllegalStateException describing the two sizes and both lists is thrown — an internal invariant violation.

Solutions

Don't mutate the input sentence list or the returned tree between parse() and the result accessors (getBestParse, etc.)
Call each result method only once per parse on the same ParserQuery; re-parse for a new sentence
Inspect the Orig/Pars dumps in the message: if punctuation was added (addedPunct), expect leaves = originalSentence.size() + 1
Disable op.testOptions.addMissingFinalPunctuation if your pipeline already guarantees final punctuation, removing the count ambiguity
Report upstream if the parser itself drops tokens (e.g. filter-only grammar); verify with a vanilla model on the same input

Example fix

// before
Tree t = parserQuery.getBestParse();
t = myTransform(t);          // mutates leaves
parserQuery.getKBestParses(); // size mismatch
// after
Tree t = parserQuery.getBestParse();
List<List<Tree>> kBest = parserQuery.getKBestParses(); // collect all results first
Tree transformed = myTransform(t);

Defensive patterns

Strategy: try-catch

Validate before calling

// Don't mutate inputs; verify no code path reuses the query across sentences
assert parserQuery != null && !treeAlreadyConsumed;

Try / catch

try {
  Tree t = parserQuery.getBestParse();
} catch (IllegalStateException e) {
  if (e.getMessage().startsWith("originalWords and sentence of different sizes")) {
    // re-parse fresh or return raw tree without restoring original words
  }
  throw e;
}

Prevention

Treat ParserQuery results as one-shot: one parse, then read results
Never modify the input word list or result tree between parse and accessors
If adding final punctuation, account for the +1 leaf in any leaf-count checks

When it happens

Trigger: Calling getBestParse/getKBestParses/etc. after parse when the tree returned by the parsers has a different leaf count than the (possibly punctuation-augmented) input — e.g. the parser dropped or inserted tokens, or restoreOriginalWords is invoked twice / on a mismatched tree.

Common situations: Custom token filters or annotators mutating the sentence between parse and result retrieval; reusing a ParserQuery result after parsing a different sentence; manually modifying the returned tree before calling result accessors; a grammar/tokenizer that normalizes tokens unexpectedly.

Understand the failure class

Background: "This is a bug, please report it": internal invariant violations, unreachable panics, and SNH errors explained — this error's family across 47 libraries.

Related errors

AI-assisted analysis of stanfordnlp/CoreNLP@1b7edd19c4 (2026-09-10). Data as JSON: /api/errors/c74c2f8e4e5c1ae0. Report an issue: GitHub.

Appendix: source

Thrown at src/edu/stanford/nlp/parser/lexparser/LexicalizedParserQuery.java:315

      if ( ! bparser.parse(sentenceB)) {
        return parseSucceeded;
      } else {
        parseSucceeded = true;
      }
    }
    return true;
  }


  @Override
  public void restoreOriginalWords(Tree tree) {
    if (originalSentence == null || tree == null) {
      return;
    }
    List<Tree> leaves = tree.getLeaves();
    int expectedSize = addedPunct ? originalSentence.size() + 1 : originalSentence.size();
    if (leaves.size() != expectedSize) {
      throw new IllegalStateException("originalWords and sentence of different sizes: " + expectedSize + " vs. " + leaves.size() +
                                      "\n Orig: " + SentenceUtils.listToString(originalSentence) +
                                      "\n Pars: " + SentenceUtils.listToString(leaves));
    }
    Iterator<Tree> leafIterator = leaves.iterator();
    for (HasWord word : originalSentence) {
      Tree leaf = leafIterator.next();
      if (!(word instanceof Label)) {
        continue;
      }
      leaf.setLabel((Label) word);
    }
  }


  /**
   * Parse a (speech) lattice with the PCFG parser.
   *
   * @param lr a lattice to parse

View on GitHub (pinned to 1b7edd19c4)