What causes " : Stream is closed!" in apache/hadoop?

Reopen the stream with fs.open(path) when data is still needed - a closed S3AInputStream cannot be revived

apache/hadoop · error · IOException

<uri>: Stream is closed!

Error message

<uri>: Stream is closed!

What it means

Thrown by S3AInputStream.checkNotClosed(), which guards read/seek/positioned-read operations. The volatile closed flag is set by close()/abort(); because close() is synchronized and later calls are no-ops, any use of the stream after closing fails immediately with '<uri>: Stream is closed!' rather than touching released HTTP resources.

Source

Thrown at hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AInputStream.java:593

    if (bytesRead > 0) {
      pos += bytesRead;
      nextReadPos += bytesRead;
      incrementBytesRead(bytesRead);
    } else {
      streamReadResultNegative();
    }
    getS3AStreamStatistics().readOperationCompleted(len, bytesRead);
    return bytesRead;
  }

  /**
   * Verify that the input stream is open. Non blocking; this gives
   * the last state of the volatile {@link #closed} field.
   * @throws IOException if the connection is closed.
   */
  private void checkNotClosed() throws IOException {
    if (closed) {
      throw new IOException(getUri() + ": " + FSExceptionMessages.STREAM_IS_CLOSED);
    }
  }

  /**
   * Close the stream.
   * This triggers publishing of the stream statistics back to the filesystem
   * statistics.
   * This operation is synchronized, so that only one thread can attempt to
   * close the connection; all later/blocked calls are no-ops.
   * @throws IOException on any problem
   */
  @Override
  public synchronized void close() throws IOException {
    if (!closed) {
      closed = true;
      try {
        stopVectoredIOOperations.set(true);
        // close or abort the stream; blocking

View on GitHub (pinned to 2add963021)

Solutions

Reopen the stream with fs.open(path) when data is still needed - a closed S3AInputStream cannot be revived
Audit close()/abort() call sites and thread ownership so no reader outlives the closer
If the engine calls unbuffer()/close() between batches, re-open the stream on the next read instead of reusing the handle
Wrap stream lifecycle in your own state machine/pool that invalidates entries on close

Example fix

// before
FSDataInputStream in = cachedStream; // may have been closed by the engine
in.read(buf);                 // IOException: <uri>: Stream is closed!

// after - track open state yourself and re-open
if (cachedStream == null || !cachedOpen) {
  cachedStream = fs.open(path);
  cachedOpen = true;
}
cachedStream.read(buf);

Defensive patterns

Strategy: try-catch

Try / catch

try {
  in.read(buf);
} catch (IOException e) {
  if (e.getMessage() != null && e.getMessage().endsWith(FSExceptionMessages.STREAM_IS_CLOSED)) {
    in = fs.open(path); // recover by reopening the stream
  } else {
    throw e;
  }
}

Prevention

One owner per stream: the component that opens it closes it
Never cache FSDataInputStream across engine callbacks that may unbuffer() or close()
Track your own open/closed flag for shared streams and check it before every read

When it happens

Trigger: read(), seek(), readFully(), or readVectored() on an S3AInputStream after close() or abort() was called; a reader thread racing a concurrent close(); reusing a cached FSDataInputStream that an engine has already closed or unbuffered.

Common situations: Reusing a cached file handle across query batches in Hive/Spark after the framework called close() or unbuffer(); closing streams in one method then reading in another; task cancellation closing inputs mid-iteration; custom InputFormats with confused stream ownership.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/758305ed6c514013. Report an issue: GitHub.