{"record":{"id":"aa048e9480e044da","repo":"stanfordnlp/CoreNLP","slug":"parsing-of-sentence-ran-out-of-memory-length","errorCode":null,"errorMessage":"Parsing of sentence ran out of memory (length=","messagePattern":"Parsing of sentence ran out of memory \\(length=","errorType":"console","errorClass":null,"httpStatus":null,"severity":"warning","filePath":"src/edu/stanford/nlp/pipeline/ParserAnnotator.java","lineNumber":367,"sourceCode":"        }\n      } else {\n        List<ScoredObject<Tree>> scoredObjects = pq.getKBestParses(this.kBest);\n        if (scoredObjects == null || scoredObjects.size() < 1) {\n          log.warn(\"Parsing of sentence failed.  \" +\n              \"Will ignore and continue: \" +\n              SentenceUtils.listToString(words));\n        } else {\n          for (ScoredObject<Tree> so : scoredObjects) {\n            // -10000 denotes unknown words\n            Tree tree = so.object();\n            tree.setScore(so.score() % -10000.0);\n            trees.add(tree);\n          }\n        }\n      }\n    } catch (OutOfMemoryError e) {\n      log.error(e); // Beware that we can now get an OOM in logging, too.\n      log.warn(\"Parsing of sentence ran out of memory (length=\" + words.size() + \").  \" +\n              \"Will ignore and try to continue.\");\n    } catch (NoSuchParseException e) {\n      log.warn(\"Parsing of sentence failed, possibly because of out of memory.  \" +\n              \"Will ignore and continue: \" +\n              SentenceUtils.listToString(words));\n    }\n    return trees;\n  }\n\n  @Override\n  public Set<Class<? extends CoreAnnotation>> requires() {\n    if (parser.requiresTags()) {\n      return Collections.unmodifiableSet(new ArraySet<>(Arrays.asList(\n          CoreAnnotations.TextAnnotation.class,\n          CoreAnnotations.TokensAnnotation.class,\n          CoreAnnotations.ValueAnnotation.class,\n          CoreAnnotations.OriginalTextAnnotation.class,\n          CoreAnnotations.CharacterOffsetBeginAnnotation.class,","sourceCodeStart":349,"sourceCodeEnd":385,"githubUrl":"https://github.com/stanfordnlp/CoreNLP/blob/1b7edd19c4d0d7b1f13a2591425b9b60a0b1af7a/src/edu/stanford/nlp/pipeline/ParserAnnotator.java#L349-L385","documentation":"ParserAnnotator catches OutOfMemoryError thrown while parsing a single sentence and logs this warning instead of crashing the pipeline. The Stanford parser builds large chart structures, so very long sentences can exhaust the JVM heap. The failed sentence is skipped and the pipeline continues with the next one.","triggerScenarios":"Calling StanfordCoreNLP with the 'parse' (or 'depparse'/binarized PCFG) annotator on a document containing an extremely long sentence (hundreds of tokens) when the JVM heap is too small; doOneSentence catches OutOfMemoryError from the parse call.","commonSituations":"Running corenlp on web text, legal documents, or text without proper sentence segmentation, so a 'sentence' is thousands of words; running with default -Xmx on big batch jobs; sentence splitter failing on a malformed document.","solutions":["Increase JVM heap, e.g. java -Xmx8g, since the parser needs memory proportional to sentence length","Fix sentence splitting so no absurdly long sentences are produced (check ssplit.eolonly or fix input text encoding/newlines)","Cap sentence length by pre-splitting very long sentences before annotation","Inspect the logged words.length value to find and repair the offending document","Use the lighter depparse annotator instead of the PCFG parse for very long text"],"exampleFix":"// before\nRuntime.getRuntime().exec(new String[]{\"java\", \"-cp\", \"corenlp.jar\", ...});\n// after\nRuntime.getRuntime().exec(new String[]{\"java\", \"-Xmx8g\", \"-cp\", \"corenlp.jar\", ...});","handlingStrategy":"try-catch","validationCode":"int maxLen = 200;\nif (words.size() > maxLen) {\n    words = splitLongSentence(words, maxLen); // pre-split before annotation\n}","typeGuard":null,"tryCatchPattern":"// this error is already caught internally; to detect it downstream, check for empty parse output\n// and monitor logs for \"ran out of memory (length=\" to identify offending documents","preventionTips":["Run CoreNLP with generous heap (-Xmx4g or more for parsing)","Ensure sentence splitting is correct (fix line endings/encoding)","Pre-split pathological long sentences before annotation"],"tags":["jvm","out-of-memory","parsing","pipeline"],"backgroundTag":"out-of-memory","analyzedSha":"1b7edd19c4d0d7b1f13a2591425b9b60a0b1af7a","analyzedAt":"2026-09-10T02:24:07.274Z","contentChangedAt":"2026-09-10T02:24:07.274Z","schemaVersion":2},"datasetVersion":"2026-09-17T15:17:12.973Z"}