apache/hadoop · error · IOException

name + ": Stream is closed!"

Error message

name + ": Stream is closed!"

What it means

S3ARemoteInputStream is the seekable local-facing stream that S3APrefetchingInputStream reads cached/fetched block data through. throwIfClosed() throws IOException("<name>: Stream is closed!") when any read/seek happens after closed=true. Users rarely see it directly; it surfaces when the prefetching stream or test code touches the inner stream post-close.

Source

Thrown at hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/prefetch/S3ARemoteInputStream.java:456

  public boolean markSupported() {
    return false;
  }

  @Override
  public String toString() {
    if (isClosed()) {
      return "closed";
    }

    StringBuilder sb = new StringBuilder();
    sb.append(String.format("nextReadPos = (%d)%n", nextReadPos));
    sb.append(String.format("fpos = (%s)", fpos));
    return sb.toString();
  }

  protected void throwIfClosed() throws IOException {
    if (closed) {
      throw new IOException(
          name + ": " + FSExceptionMessages.STREAM_IS_CLOSED);
    }
  }

  protected void throwIfInvalidSeek(long pos) throws EOFException {
    if (pos < 0) {
      throw new EOFException(FSExceptionMessages.NEGATIVE_SEEK + " " + pos);
    } else if (pos > this.getBlockData().getFileSize()) {
      throw new EOFException(FSExceptionMessages.CANNOT_SEEK_PAST_EOF + " " + pos);
    }
  }

  // Unsupported functions.

  @Override
  public void mark(int readlimit) {
    throw new UnsupportedOperationException("mark not supported");
  }

View on GitHub (pinned to 2add963021)

Solutions

  1. Do not use the S3ARemoteInputStream directly - perform all I/O through S3APrefetchingInputStream.
  2. Ensure clean shutdown of reader threads before closing the prefetching stream (let in-flight block fetches finish or cancel cleanly).
  3. Catch IOException and inspect for 'Stream is closed' to distinguish lifecycle bugs from S3 errors.
  4. Upgrade hadoop-aws - shutdown races around prefetch threads have been the subject of fixes.

Example fix

// before
S3ARemoteInputStream remote = ...;
remote.close();
remote.read(); // IOException: <name>: Stream is closed!

// after
if (!remote.isClosed()) {
  remote.read();
}
Defensive patterns

Strategy: validation

Validate before calling

// test-only style guard if you hold the inner stream directly
if (!remote.isClosed()) {
  remote.read(buf, off, len);
}

Try / catch

try {
  remote.read(buf, off, len);
} catch (IOException e) {
  if (e.getMessage() != null && e.getMessage().contains("Stream is closed")) {
    // stop the reader thread / reopen the enclosing prefetch stream
  } else { throw e; }
}

Prevention

When it happens

Trigger: read()/seek()/getPos() on the S3ARemoteInputStream after close(); prefetch stream teardown racing a queued block-read thread; tests retaining the remote stream after closing the enclosing prefetching input stream.

Common situations: Prefetch shutdown races under heavy concurrency (task close while a data block read is in flight); unit tests of S3ARemoteInputStream; custom code extracting the inner stream via reflection/debugging and using it after the owner closed.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/20d927580115fe2e. Report an issue: GitHub.