stanfordnlp/CoreNLP · error · RuntimeIOException

java.io.IOException

Error message

java.io.IOException

What it means

The LexerTokenizer(Lexer, Reader) constructor calls l.yyreset(r) to bind the lexer to the reader; if that throws an IOException it is rethrown as a RuntimeIOException whose message is the IOException's message (here shown as "java.io.IOException"). It means the lexer could not be reset onto the supplied reader — typically the reader itself was null or already broken.

Solutions

1. Ensure the Reader is non-null and open before constructing the tokenizer
2. Create a fresh Reader over the input for each tokenizer rather than reusing a consumed one
3. Do not close the underlying stream before tokenization completes
4. Catch RuntimeIOException around construction and inspect the cause for the underlying stream error

Example fix

// before
Reader r = new InputStreamReader(in); in.close(); // stream closed too early
LexerTokenizer tok = new LexerTokenizer(new PTBLexer(), r);
// after
Reader r = new BufferedReader(new InputStreamReader(in, StandardCharsets.UTF_8));
LexerTokenizer tok = new LexerTokenizer(new PTBLexer(), r); // close r only after tokenizing

if (reader == null) throw new IllegalArgumentException("Reader is null");
try {
  reader.ready(); // throws IOException if the stream is closed/broken
} catch (IOException e) {
  throw new IllegalStateException("Reader already closed or broken", e);
}

try {
  LexerTokenizer tok = new LexerTokenizer(lexer, reader);
} catch (RuntimeIOException e) {
  throw new TokenizerInitException("Could not attach reader to lexer: " + e.getMessage(), e);
}

When it happens

Trigger: Calling new LexerTokenizer(lexer, reader) where yyreset throws — most often reader is null (NPE/IO inside yyreset) or the underlying stream of the reader is already closed or in an error state.

Common situations: Passing a Reader built from a closed InputStream; reusing a reader/stream that a previous tokenizer already consumed and closed; passing null reader because an earlier IO step failed.

Understand the failure class

Background: "failed to read file", EACCES, ENOENT and "could not read <path>" errors: when a program can't read a file from disk — this error's family across 49 libraries.

Related errors

• Couldn't read RegexNER from reader
• slurpReader IO problem
• Stream Closed
• : Could not open path
• java.io.IOException

AI-assisted analysis of stanfordnlp/CoreNLP@1b7edd19c4 (2026-09-10). Data as JSON: /api/errors/79ed0b433f9fa978. Report an issue: GitHub.

Appendix: source

Thrown at src/edu/stanford/nlp/process/LexerTokenizer.java:65

   */
  public LexerTokenizer(Lexer l) {
    if (l == null) {
      throw new IllegalArgumentException("You can't make a Tokenizer out of a null Lexer!");
    } else {
      this.lexer = l;
    }
  }

  /** Constructs a tokenizer from a {@link Lexer} and makes a {@link Reader}
   *  the active input stream for the tokenizer.
   */
  public LexerTokenizer(Lexer l, Reader r) {
    this(l);

    try {
      l.yyreset(r);
    } catch (IOException e) {
      throw new RuntimeIOException(e.getMessage());
    }

    getNext();
  }


  /**
   * For testing only.
   */
  public static void main(String[] args) throws IOException {
    Tokenizer<String> t = new LexerTokenizer(new JFlexDummyLexer((Reader) null), new BufferedReader(new FileReader(args[0])));
    while (t.hasNext()) {
      System.out.println("token " + t.next());
    }
  }

}

View on GitHub (pinned to 1b7edd19c4)