stanfordnlp/CoreNLP · error · UnsupportedOperationException

Sentence too long: length

Error message

Sentence too long: length 

What it means

parseInternal enforces op.testOptions.maxLength: if the tokenized sentence is longer than the configured maximum, the query is marked parseSkipped and UnsupportedOperationException("Sentence too long: length N") is thrown. The maximum exists because exhaustive PCFG parsing is exponential in practice on very long sentences.

Solutions

1. Increase the limit via testOptions: e.g. -maxLength 100 (op.testOptions.maxLength = 100) if memory/time allow
2. Run proper sentence splitting (DocumentPreprocessor / CoreNLP ssplit) so inputs are natural sentence lengths
3. Catch UnsupportedOperationException and treat the sentence as skipped (parseSkipped is set) with a fallback, e.g. dependency parse
4. Truncate or segment very long inputs before parsing

Example fix

// before
Options op = new Options(); // maxLength default 40
LexicalizedParser lp = LexicalizedParser.getParserFromSerialisedFile(op, model);
lp.parse(longSentence); // 120 tokens
// after
Options op = new Options();
op.testOptions.maxLength = 120;
LexicalizedParser lp = LexicalizedParser.getParserFromSerialisedFile(op, model);

// Check length against the parser's configured limit before parsing
if (sentence.size() > op.testOptions.maxLength) {
  // skip, truncate, or segment the sentence
}

try {
  Tree t = parser.parse(sentence);
} catch (UnsupportedOperationException e) {
  if (e.getMessage().startsWith("Sentence too long")) return fallbackParse(sentence);
  throw e;
}

When it happens

Trigger: Calling parse() on a sentence whose token count exceeds maxLength (default often 40), or after raising processing of long documents without adjusting the limit.

Common situations: Parsing whole paragraphs or documents as one "sentence"; running the default max (40) on noisy text that under-tokenizes into giant sentences; speech-transcript or web text with no sentence boundaries.

Understand the failure class

Background: "value must be between 0 and 1" / "out of range" / "must not be negative" errors: fixing range-validation failures across open-source libraries — this error's family across 42 libraries.

Related errors

• Lattice too big:
• Attempt to create ChineseSimWordAvgDepGrammar before…
• Attribute already defined:
• Attribute already defined
• Attribute match already defined:

AI-assisted analysis of stanfordnlp/CoreNLP@1b7edd19c4 (2026-09-10). Data as JSON: /api/errors/2ac634899264b6e4. Report an issue: GitHub.

Appendix: source

Thrown at src/edu/stanford/nlp/parser/lexparser/LexicalizedParserQuery.java:248

          TaggedWord tw = new TaggedWord(word.word(), ((HasTag) word).tag());
          sentenceB.add(tw);
        } else {
          sentenceB.add(new Word(word.word()));
        }
      }
      for (HasWord word : sentenceB) {
        word.setWord(op.wordFunction.apply(word.word()));
      }
    } else {
      sentenceB = new ArrayList<>(sentence);
    }

    if (op.testOptions.addMissingFinalPunctuation) {
      addedPunct = addSentenceFinalPunctIfNeeded(sentenceB, length);
    }
    if (length > op.testOptions.maxLength) {
      parseSkipped = true;
      throw new UnsupportedOperationException("Sentence too long: length " + length);
    }
    TreePrint treePrint = getTreePrint();
    PrintWriter pwOut = op.tlpParams.pw();

    //Insert the boundary symbol
    if(sentence.get(0) instanceof CoreLabel) {
      CoreLabel boundary = new CoreLabel();
      boundary.setWord(Lexicon.BOUNDARY);
      boundary.setValue(Lexicon.BOUNDARY);
      boundary.setTag(Lexicon.BOUNDARY_TAG);
      boundary.setIndex(sentence.size()+1);//1-based indexing used in the parser
      sentenceB.add(boundary);
    } else {
      sentenceB.add(new TaggedWord(Lexicon.BOUNDARY, Lexicon.BOUNDARY_TAG));
    }

    if (Thread.interrupted()) {
      throw new RuntimeInterruptedException();

View on GitHub (pinned to 1b7edd19c4)