{"record":{"id":"983f2911ad748ab9","repo":"stanfordnlp/CoreNLP","slug":"end-of-token-stream-encountered-before-parsing-cou","errorCode":null,"errorMessage":"End of token stream encountered before parsing could complete.","messagePattern":"End of token stream encountered before parsing could complete\\.","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"error","filePath":"src/edu/stanford/nlp/trees/PennTreeReader.java","lineNumber":156,"sourceCode":"   * that a malformed tree will corrupt the token stream. In this case,\n   * an {@code IOException} will eventually be thrown.\n   *\n   * @return A single tree, or {@code null} at end of token stream.\n   */\n  @Override\n  public Tree readTree() throws IOException {\n    Tree t = null;\n\n    while (tokenizer.hasNext() && t == null) {\n\n      //Setup PDA\n      this.currentTree = null;\n      this.stack = new ArrayList<>();\n\n      try {\n        t = getTreeFromInputStream();\n      } catch (NoSuchElementException e) {\n        throw new IOException(\"End of token stream encountered before parsing could complete.\");\n      }\n\n      if (t != null) {\n        // cdm 20100618: Don't do this!  This was never the historical behavior!!!\n        // Escape empty trees e.g. (())\n        // while(t != null && (t.value() == null || t.value().equals(\"\")) && t.numChildren() <= 1)\n        //   t = t.firstChild();\n\n        if (treeNormalizer != null && treeFactory != null) {\n          t = treeNormalizer.normalizeWholeTree(t, treeFactory);\n        }\n        if (t != null) {\n          t.indexLeaves(true);\n        }\n      }\n    }\n\n    return t;","sourceCodeStart":138,"sourceCodeEnd":174,"githubUrl":"https://github.com/stanfordnlp/CoreNLP/blob/1b7edd19c4d0d7b1f13a2591425b9b60a0b1af7a/src/edu/stanford/nlp/trees/PennTreeReader.java#L138-L174","documentation":"PennTreeReader.readTree parses the token stream produced by the tokenizer; if the stream is exhausted while the tree parser still expects more tokens (unbalanced parentheses or an unfinished tree), getTreeFromInputStream throws NoSuchElementException, which is converted to this IOException signaling incomplete input.","triggerScenarios":"Calling PennTreeReader.readTree()/tree() on input where closing parentheses are missing — e.g. a truncated file, a tree cut off by a line/record limit, or passing a stream containing only an opening '(ROOT ...' with no closing ')'.","commonSituations":"Corrupted or partially downloaded PTB files; reading a tree written without a trailing newline terminator in a multi-line stream; splitting files by size and cutting a tree in half; forgetting the final ')' when generating trees programmatically.","solutions":["Check the input for balanced parentheses and repair/complete the truncated tree at the point the reader stopped.","Wrap readTree in try-catch for IOException and skip/reattempt the malformed tree, resynchronizing on the next '(ROOT' marker.","Ensure trees are separated by blank lines and fully written (all closing parens) before closing the stream.","Validate files with a quick paren-balance counter before feeding them to the reader."],"exampleFix":"// before\nTree t = reader.readTree(); // throws if input truncated\n// after\nTree t;\ntry { t = reader.readTree(); }\ncatch (IOException e) {\n  log.warn(\"Incomplete tree input, resynchronizing: \" + e.getMessage());\n  t = null;\n}","handlingStrategy":"try-catch","validationCode":"// Balance check before reading\nstatic boolean balanced(String text) {\n  int d = 0; boolean esc = false;\n  for (char c : text.toCharArray()) {\n    if (c == '(') d++; else if (c == ')') d--;\n    if (d < 0) return false;\n  }\n  return d == 0;\n}","typeGuard":null,"tryCatchPattern":"try {\n  Tree t = reader.readTree();\n} catch (IOException e) {\n  if (e.getMessage() != null && e.getMessage().contains(\"End of token stream\")) {\n    log.warn(\"Truncated/incomplete tree input; resynchronizing to next tree\");\n    t = null;\n  } else throw e;\n}","preventionTips":["Verify files are fully downloaded (checksums) before parsing treebanks.","Write each tree with all closing parentheses and a blank-line separator.","When splitting files, split on tree boundaries (blank lines), not byte counts."],"tags":["parsing","penntreebank","io","truncated-input"],"backgroundTag":"file-read-failed","analyzedSha":"1b7edd19c4d0d7b1f13a2591425b9b60a0b1af7a","analyzedAt":"2026-09-10T02:24:07.274Z","contentChangedAt":"2026-09-10T02:24:07.274Z","schemaVersion":2},"datasetVersion":"2026-09-17T15:17:12.973Z"}