stanfordnlp/CoreNLP · error · IOException

End of token stream encountered before parsing could…

Error message

End of token stream encountered before parsing could complete.

What it means

PennTreeReader.readTree parses the token stream produced by the tokenizer; if the stream is exhausted while the tree parser still expects more tokens (unbalanced parentheses or an unfinished tree), getTreeFromInputStream throws NoSuchElementException, which is converted to this IOException signaling incomplete input.

Solutions

1. Check the input for balanced parentheses and repair/complete the truncated tree at the point the reader stopped.
2. Wrap readTree in try-catch for IOException and skip/reattempt the malformed tree, resynchronizing on the next '(ROOT' marker.
3. Ensure trees are separated by blank lines and fully written (all closing parens) before closing the stream.
4. Validate files with a quick paren-balance counter before feeding them to the reader.

Example fix

// before
Tree t = reader.readTree(); // throws if input truncated
// after
Tree t;
try { t = reader.readTree(); }
catch (IOException e) {
  log.warn("Incomplete tree input, resynchronizing: " + e.getMessage());
  t = null;
}

// Balance check before reading
static boolean balanced(String text) {
  int d = 0; boolean esc = false;
  for (char c : text.toCharArray()) {
    if (c == '(') d++; else if (c == ')') d--;
    if (d < 0) return false;
  }
  return d == 0;
}

try {
  Tree t = reader.readTree();
} catch (IOException e) {
  if (e.getMessage() != null && e.getMessage().contains("End of token stream")) {
    log.warn("Truncated/incomplete tree input; resynchronizing to next tree");
    t = null;
  } else throw e;
}

When it happens

Trigger: Calling PennTreeReader.readTree()/tree() on input where closing parentheses are missing — e.g. a truncated file, a tree cut off by a line/record limit, or passing a stream containing only an opening '(ROOT ...' with no closing ')'.

Common situations: Corrupted or partially downloaded PTB files; reading a tree written without a trailing newline terminator in a multi-line stream; splitting files by size and cutting a tree in half; forgetting the final ')' when generating trees programmatically.

Understand the failure class

Background: "failed to read file", EACCES, ENOENT and "could not read <path>" errors: when a program can't read a file from disk — this error's family across 49 libraries.

Related errors

• Error on line
• Exception reading key file + sentFileName
• argsToProperties could not read properties file: " + file
• Argument array lengths differ
• Array lengths don't match

AI-assisted analysis of stanfordnlp/CoreNLP@1b7edd19c4 (2026-09-10). Data as JSON: /api/errors/983f2911ad748ab9. Report an issue: GitHub.

Appendix: source

Thrown at src/edu/stanford/nlp/trees/PennTreeReader.java:156

   * that a malformed tree will corrupt the token stream. In this case,
   * an {@code IOException} will eventually be thrown.
   *
   * @return A single tree, or {@code null} at end of token stream.
   */
  @Override
  public Tree readTree() throws IOException {
    Tree t = null;

    while (tokenizer.hasNext() && t == null) {

      //Setup PDA
      this.currentTree = null;
      this.stack = new ArrayList<>();

      try {
        t = getTreeFromInputStream();
      } catch (NoSuchElementException e) {
        throw new IOException("End of token stream encountered before parsing could complete.");
      }

      if (t != null) {
        // cdm 20100618: Don't do this!  This was never the historical behavior!!!
        // Escape empty trees e.g. (())
        // while(t != null && (t.value() == null || t.value().equals("")) && t.numChildren() <= 1)
        //   t = t.firstChild();

        if (treeNormalizer != null && treeFactory != null) {
          t = treeNormalizer.normalizeWholeTree(t, treeFactory);
        }
        if (t != null) {
          t.indexLeaves(true);
        }
      }
    }

    return t;

View on GitHub (pinned to 1b7edd19c4)