apache/hadoop · error · IOException

Stream is closed!

Error message

Stream is closed!

What it means

S3APrefetchingInputStream (fs.s3a.input.stream.type=Prefetch, or legacy fs.s3a.prefetch.enabled=true) throws IOException(FSExceptionMessages.STREAM_IS_CLOSED) from throwIfClosed() when read/seek/available/getPos is invoked after close(). Closing sets the underlying inputStream to null, so any later I/O on the prefetching stream fails with this guard.

Source

Thrown at hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/prefetch/S3APrefetchingInputStream.java:295

   * Gets the internal IO statistics.
   *
   * @return the internal IO statistics.
   */
  @Override
  public IOStatistics getIOStatistics() {
    if (!isClosed()) {
      ioStatistics = inputStream.getIOStatistics();
    }
    return ioStatistics;
  }

  protected boolean isClosed() {
    return inputStream == null;
  }

  protected void throwIfClosed() throws IOException {
    if (isClosed()) {
      throw new IOException(FSExceptionMessages.STREAM_IS_CLOSED);
    }
  }

  // Unsupported functions.

  @Override
  public boolean seekToNewSource(long targetPos) throws IOException {
    return false;
  }

  @Override
  public boolean markSupported() {
    return false;
  }
}

View on GitHub (pinned to 2add963021)

Solutions

  1. Give the stream a single owner; use try-with-resources per consumer.
  2. For shared reads, open a separate FileSystem/FSDataInputStream per reader instead of sharing one handle.
  3. Synchronize close(): do not close while other threads may still read (CountDownLatch/executor shutdown discipline).
  4. Treat this IOException as a lifecycle defect - log the stack with the operation and fix the ownership, do not retry.

Example fix

// before: shared handle, racy close
FSDataInputStream in = fs.open(path);
new Thread(() -> in.close()).start();
int b = in.read(); // may throw: Stream is closed!

// after: one stream per consumer
try (FSDataInputStream in = fs.open(path)) {
  consume(in);
}
Defensive patterns

Strategy: validation

Validate before calling

// one stream per consumer; never share across threads
try (FSDataInputStream in = fs.open(path)) {
  return consume(in);
}

Try / catch

try {
  return in.read();
} catch (IOException e) {
  if ("Stream is closed".equals(e.getMessage())
      || (e.getMessage() != null && e.getMessage().contains("Stream is closed"))) {
    LOG.error("read after close on {}", path); // lifecycle bug: fix ownership
  }
  throw e;
}

Prevention

When it happens

Trigger: Calling read() or seek() after close() on an s3a FSDataInputStream backed by the prefetcher; reading in a background thread while another thread closed the stream; frameworks double-processing files where the second pass reads a closed stream.

Common situations: Concurrent readers sharing one open file handle where one thread finishes and closes; map tasks re-reading input splits after a task-level cleanup closed the stream; caching FileStatus/streams beyond the file's lifetime; race between idle-timeout close (if any) and a slow reader.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/e56a8684abee787c. Report an issue: GitHub.