{"record":{"id":"8ad7479657fe339c","repo":"stanfordnlp/CoreNLP","slug":"java-io-ioexception-8ad747","errorCode":null,"errorMessage":"java.io.IOException","messagePattern":"java\\.io\\.IOException","errorType":"exception","errorClass":"RuntimeIOException","httpStatus":null,"severity":"error","filePath":"src/edu/stanford/nlp/process/PTBTokenizer.java","lineNumber":303,"sourceCode":"    lexer = new PTBLexer(r, tokenFactory, options);\n  }\n\n\n  /**\n   * Internally fetches the next token.\n   *\n   * @return the next token in the token stream, or null if none exists.\n   */\n  @SuppressWarnings({\"unchecked\"})\n  @Override\n  protected T getNext() {\n    // if (lexer == null) {\n    //   return null;\n    // }\n    try {\n      return (T) lexer.next();\n    } catch (IOException e) {\n      throw new RuntimeIOException(e);\n    }\n    // cdm 2007: this shouldn't be necessary: PTBLexer decides for itself whether to return CRs based on the same flag!\n    // get rid of CRs if necessary\n    // while (!tokenizeNLs && PTBLexer.cr.equals(((HasWord) token).word())) {\n    //   token = (T)lexer.next();\n    // }\n\n    // horatio: we used to catch exceptions here, which led to broken\n    // behavior and made it very difficult to debug whatever the\n    // problem was.\n  }\n\n  /**\n   * Returns the string literal inserted for newlines when the -tokenizeNLs\n   * options is set.\n   *\n   * @return string literal inserted for \"\\n\".\n   */","sourceCodeStart":285,"sourceCodeEnd":321,"githubUrl":"https://github.com/stanfordnlp/CoreNLP/blob/1b7edd19c4d0d7b1f13a2591425b9b60a0b1af7a/src/edu/stanford/nlp/process/PTBTokenizer.java#L285-L321","documentation":"PTBTokenizer.getNext() delegates to lexer.next(); the underlying PTBLexer reads from the input Reader and can raise IOException. Because Iterator-style APIs cannot throw checked exceptions, the tokenizer wraps it in a RuntimeIOException (unchecked) with this cause.","triggerScenarios":"Iterating a PTBTokenizer backed by a Reader whose underlying stream fails mid-read: closed FileReader, broken socket/pipe input, unreadable file encountered while reading the next token.","commonSituations":"File deleted or unmounted during tokenization; reading from a process output stream that died; filesystem/permission errors surfacing lazily at next() rather than at construction.","solutions":["Catch RuntimeIOException around tokenizer iteration and inspect the IOException cause.","Ensure the Reader/underlying file or stream is open, readable, and not closed prematurely for the whole tokenization loop.","Read the input fully into memory (e.g. StringReader over file contents) if you need deterministic behavior."],"exampleFix":"// before\nfor (CoreLabel tok : tokenizer) { use(tok); }\n// after\ntry {\n  for (CoreLabel tok : tokenizer) { use(tok); }\n} catch (RuntimeIOException e) {\n  throw new IOException(\"Tokenization input failed\", e);\n}","handlingStrategy":"try-catch","validationCode":"if (!file.canRead()) throw new IOException(\"Cannot read tokenizer input: \" + file);","typeGuard":null,"tryCatchPattern":"try {\n  while (tokenizer.hasNext()) consume(tokenizer.next());\n} catch (RuntimeIOException e) {\n  throw new IOException(\"Tokenization stream failed\", e);\n}","preventionTips":["Keep the underlying Reader open for the whole iteration; use try-with-resources around the loop.","Prefer reading files into memory before tokenizing for deterministic failure points.","Check file readability before constructing the tokenizer."],"tags":["io","exception","tokenizer"],"backgroundTag":"file-read-failed","analyzedSha":"1b7edd19c4d0d7b1f13a2591425b9b60a0b1af7a","analyzedAt":"2026-09-10T02:24:07.274Z","contentChangedAt":"2026-09-10T02:24:07.274Z","schemaVersion":2},"datasetVersion":"2026-09-16T04:17:20.429Z"}