apache/hadoop · error · IOException

Stream is closed!

Error message

Stream is closed!

What it means

BufferedFSInputStream is the buffering wrapper used inside FSDataInputStream; once the stream is closed the underlying `in` reference is null, and getPos() checks that and throws IOException("Stream is closed!") from FSExceptionMessages. This is a use-after-close lifecycle bug in the caller, not a filesystem I/O failure. Note that skip() calls getPos() first, so skip() on a closed stream surfaces this same error.

Source

Thrown at hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/BufferedFSInputStream.java:66

   * Creates a <code>BufferedFSInputStream</code>
   * with the specified buffer size,
   * and saves its  argument, the input stream
   * <code>in</code>, for later use.  An internal
   * buffer array of length  <code>size</code>
   * is created and stored in <code>buf</code>.
   *
   * @param   in     the underlying input stream.
   * @param   size   the buffer size.
   * @exception IllegalArgumentException if size {@literal <=} 0.
   */
  public BufferedFSInputStream(FSInputStream in, int size) {
    super(in, size);
  }

  @Override
  public long getPos() throws IOException {
    if (in == null) {
      throw new IOException(FSExceptionMessages.STREAM_IS_CLOSED);
    }
    return ((FSInputStream)in).getPos()-(count-pos);
  }

  @Override
  public long skip(long n) throws IOException {
    if (n <= 0) {
      return 0;
    }

    seek(getPos()+n);
    return n;
  }

  @Override
  public void seek(long pos) throws IOException {
    if (in == null) {
      throw new IOException(FSExceptionMessages.STREAM_IS_CLOSED);

View on GitHub (pinned to 2add963021)

Solutions

1. Fix the lifecycle: capture getPos() before close(), or restructure so nothing touches the stream after close (try-with-resources).
2. Ensure single ownership - only the component that opened the stream closes it, and no background threads outlive it.
3. If threads share the stream, synchronize close() against readers or signal readers to stop before closing.
4. Wrap the stream in a guarded object that tracks a closed flag and fails with a clear, own IllegalStateException on misuse.

Example fix

// before
in.close();
long finalPos = in.getPos(); // throws "Stream is closed!"

// after
long finalPos = in.getPos();
in.close();

final class GuardedStream implements Closeable {
  private final FSDataInputStream in;
  private volatile boolean closed;
  GuardedStream(FileSystem fs, Path p) throws IOException { in = fs.open(p); }
  long pos() throws IOException {
    if (closed) throw new IllegalStateException("getPos after close");
    return in.getPos();
  }
  @Override public void close() throws IOException { closed = true; in.close(); }
}

try {
  return in.getPos();
} catch (IOException e) {
  if (FSExceptionMessages.STREAM_IS_CLOSED.equals(e.getMessage())) {
    throw new IllegalStateException("stream used after close", e); // lifecycle bug: fail loudly
  }
  throw e;
}

When it happens

Trigger: FSDataInputStream.getPos() invoked after close(): progress reporting or final-offset logging in a finally block, a RecordReader computing input split progress after the input stream was closed, or a background thread (prefetch, metrics sampler) racing the close on another thread.

Common situations: MapReduce/Spark input-format cleanup orders close() before a last getPos() for counters; a stream held in a cache gets evicted/closed while a reader still holds it; exception handlers close the stream and generic retry logic then calls getPos on it.

Related errors

• Cannot seek to a negative offset
• Cannot seek after EOF
• File check failed
• Stream is closed!
• {} already exists

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/f0318646296f584e. Report an issue: GitHub.