What causes "Stream is closed!" in apache/hadoop?

Reorder the lifecycle: perform all seeks and reads before close(), or reopen with fs.open(path) and seek on the new instance.

apache/hadoop · error · IOException

Stream is closed!

Error message

Stream is closed!

What it means

DFSInputStream.seek(long) checks the stream's closed flag (an AtomicBoolean set by close()) and throws IOException('Stream is closed!') if the stream was already closed. A DFSInputStream is single-use: once closed (explicitly, via DFSClient shutdown, or by an internal fatal error), any seek on it fails. The check deliberately runs after the EOF/negative-offset checks, so a valid position on a closed stream is what produces this exact message.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSInputStream.java:1648

      seek(curPos+n);
      return n;
    }
    return n < 0 ? -1 : 0;
  }

  /**
   * Seek to a new arbitrary location
   */
  @Override
  public synchronized void seek(long targetPos) throws IOException {
    if (targetPos > getFileLength()) {
      throw new EOFException("Cannot seek after EOF");
    }
    if (targetPos < 0) {
      throw new EOFException("Cannot seek to negative offset");
    }
    if (closed.get()) {
      throw new IOException("Stream is closed!");
    }
    boolean done = false;
    if (pos <= targetPos && targetPos <= blockEnd) {
      //
      // If this seek is to a positive position in the current
      // block, and this piece of data might already be lying in
      // the TCP buffer, then just eat up the intervening data.
      //
      int diff = (int)(targetPos - pos);
      if (diff <= blockReader.available()) {
        try {
          pos += blockReader.skip(diff);
          if (pos == targetPos) {
            done = true;
          } else {
            // The range was already checked. If the block reader returns
            // something unexpected instead of throwing an exception, it is
            // most likely a bug.

View on GitHub (pinned to 2add963021)

Solutions

Reorder the lifecycle: perform all seeks and reads before close(), or reopen with fs.open(path) and seek on the new instance.
If streams are cached, invalidate the cache entry at close time so a stale closed stream is never handed out again.
Track the open/closed state alongside the stream handle in your wrapper class and fail with your own descriptive error before delegating to seek().
In retry logic, always reopen the stream (fs.open) rather than reusing the instance that was closed after a failure.

Example fix

// before
try (FSDataInputStream in = fs.open(path)) {
  parseHeader(in);
} // try-with-resources closes here
cache.setLastOffset(in.getPos()); // later code calls in.seek(offset) on closed stream

// after
long lastOffset;
try (FSDataInputStream in = fs.open(path)) {
  parseHeader(in);
  lastOffset = in.getPos();
}
// when needed again:
try (FSDataInputStream in = fs.open(path)) {
  in.seek(lastOffset);
  parseBody(in);
}

Defensive patterns

Strategy: try-catch

Validate before calling

try {
  in.available(); // cheap probe: throws IOException only when the DFS stream is closed
} catch (IOException closed) {
  in = fs.open(path); // reopen and reposition
  in.seek(offset);
}

Try / catch

try {
  in.seek(offset);
} catch (IOException e) {
  if (String.valueOf(e.getMessage()).contains("Stream is closed")) {
    in = fs.open(path); // single recovery: reopen, reposition, continue
    in.seek(offset);
  } else {
    throw e;
  }
}

Prevention

Scope each FSDataInputStream with try-with-resources around ALL of its use, including final seeks.
Never cache FSDataInputStream across requests without an eviction-on-close policy.
In retry logic, always fs.open(path) a fresh stream instead of reusing a closed one.
Keep the stream owner single-threaded or make close() the sole responsibility of one owner.

When it happens

Trigger: Calling seek() after close() returned, e.g. a finally block or cleanup task that repositions a stream already closed by try-with-resources. Also: seeking a stream obtained from FileSystem.open() whose DFSClient was shut down (clientRunning=false closes cached streams), or a cached/pooled FSDataInputStream that a previous request closed.

Common situations: Stream caching layers (Spark executors, Hive/Impala session caches, homegrown connection pools) that close one stream but keep handing it out; retry loops that close the stream on error then seek before reopening; lambdas/Runnables capturing a stream whose owner already exited its try-with-resources block.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/bcdb977dbcbd6cb0. Report an issue: GitHub.