What causes "java.io.IOException" in stanfordnlp/CoreNLP?

Catch RuntimeIOException around tokenizer iteration and inspect the IOException cause.

stanfordnlp/CoreNLP · error · RuntimeIOException

java.io.IOException

Error message

java.io.IOException

What it means

PTBTokenizer.getNext() delegates to lexer.next(); the underlying PTBLexer reads from the input Reader and can raise IOException. Because Iterator-style APIs cannot throw checked exceptions, the tokenizer wraps it in a RuntimeIOException (unchecked) with this cause.

Solutions

Catch RuntimeIOException around tokenizer iteration and inspect the IOException cause.
Ensure the Reader/underlying file or stream is open, readable, and not closed prematurely for the whole tokenization loop.
Read the input fully into memory (e.g. StringReader over file contents) if you need deterministic behavior.

Example fix

// before
for (CoreLabel tok : tokenizer) { use(tok); }
// after
try {
  for (CoreLabel tok : tokenizer) { use(tok); }
} catch (RuntimeIOException e) {
  throw new IOException("Tokenization input failed", e);
}

Defensive patterns

Strategy: try-catch

Validate before calling

if (!file.canRead()) throw new IOException("Cannot read tokenizer input: " + file);

Try / catch

try {
  while (tokenizer.hasNext()) consume(tokenizer.next());
} catch (RuntimeIOException e) {
  throw new IOException("Tokenization stream failed", e);
}

Prevention

Keep the underlying Reader open for the whole iteration; use try-with-resources around the loop.
Prefer reading files into memory before tokenizing for deterministic failure points.
Check file readability before constructing the tokenizer.

When it happens

Trigger: Iterating a PTBTokenizer backed by a Reader whose underlying stream fails mid-read: closed FileReader, broken socket/pipe input, unreadable file encountered while reading the next token.

Common situations: File deleted or unmounted during tokenization; reading from a process output stream that died; filesystem/permission errors surfacing lazily at next() rather than at construction.

Understand the failure class

Background: "failed to read file", EACCES, ENOENT and "could not read <path>" errors: when a program can't read a file from disk — this error's family across 49 libraries.

Related errors

AI-assisted analysis of stanfordnlp/CoreNLP@1b7edd19c4 (2026-09-10). Data as JSON: /api/errors/8ad7479657fe339c. Report an issue: GitHub.

Appendix: source

Thrown at src/edu/stanford/nlp/process/PTBTokenizer.java:303

    lexer = new PTBLexer(r, tokenFactory, options);
  }


  /**
   * Internally fetches the next token.
   *
   * @return the next token in the token stream, or null if none exists.
   */
  @SuppressWarnings({"unchecked"})
  @Override
  protected T getNext() {
    // if (lexer == null) {
    //   return null;
    // }
    try {
      return (T) lexer.next();
    } catch (IOException e) {
      throw new RuntimeIOException(e);
    }
    // cdm 2007: this shouldn't be necessary: PTBLexer decides for itself whether to return CRs based on the same flag!
    // get rid of CRs if necessary
    // while (!tokenizeNLs && PTBLexer.cr.equals(((HasWord) token).word())) {
    //   token = (T)lexer.next();
    // }

    // horatio: we used to catch exceptions here, which led to broken
    // behavior and made it very difficult to debug whatever the
    // problem was.
  }

  /**
   * Returns the string literal inserted for newlines when the -tokenizeNLs
   * options is set.
   *
   * @return string literal inserted for "\n".
   */

View on GitHub (pinned to 1b7edd19c4)