apache/hadoop · error · IOException

Stream closed

Error message

Stream closed

What it means

DFSStripedInputStream.readWithStrategy(ReaderStrategy) is the engine behind every read on a striped stream (read(), read(byte[]), readFully, positioned reads). It first calls dfsClient.checkOpen() — which throws 'Filesystem closed' if the whole DFSClient is shut down — and then rejects reads on a closed stream with IOException('Stream closed'). Once close() has run, all stripe state is gone and reads cannot be served.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSStripedInputStream.java:395

  private int getStripedBufOffset(long offsetInBlockGroup) {
    final long stripeLen = cellSize * dataBlkNum;
    // compute the position in the curStripeBuf based on "pos"
    return (int) (offsetInBlockGroup % stripeLen);
  }

  @Override
  public synchronized boolean seekToNewSource(long targetPos)
      throws IOException {
    return false;
  }

  @Override
  protected synchronized int readWithStrategy(ReaderStrategy strategy)
      throws IOException {
    dfsClient.checkOpen();
    if (closed.get()) {
      throw new IOException("Stream closed");
    }

    // Number of bytes already read into buffer.
    int result = 0;
    int len = strategy.getTargetLength();
    CorruptedBlocks corruptedBlocks = new CorruptedBlocks();
    if (pos < getFileLength()) {
      int retries = 2;
      boolean isRetryRead = false;
      while (retries > 0) {
        try {
          if (pos > blockEnd || isRetryRead) {
            blockSeekTo(pos);
          }
          int realLen = (int) Math.min(len, (blockEnd - pos + 1L));
          synchronized (infoLock) {
            if (locatedBlocks.isLastBlockComplete()) {
              realLen = (int) Math.min(realLen,

View on GitHub (pinned to 2add963021)

Solutions

1. Reopen the file and re-seek; a closed stream cannot be resurrected.
2. Centralize lifecycle: close exactly once, at the outermost owner, after all readers finish (use try-with-resources or reference counting for shared readers).
3. Match the message to the layer: 'Filesystem closed' means the FileSystem/DFSClient was closed (fix cache shutdown ordering); 'Stream closed' means only this stream is dead (reopen the path).
4. Synchronize close() with in-flight reads so a reader never races the closer.

Example fix

// before
FSDataInputStream in = cache.get(path);
int n = in.read(buf); // another component already called in.close()

// after
FSDataInputStream in = cache.get(path);
if (in == null || consumed) {
  in = fs.open(path); // reopen instead of reading a closed stream
  in.seek(lastGoodPos);
}
int n = in.read(buf);

if (streamClosedFlag.get()) {
  in = fs.open(path);
  in.seek(resumePos);
}
int n = in.read(buf);

try {
  return in.read(buf, off, len);
} catch (IOException e) {
  if (e.getMessage() != null && e.getMessage().contains("closed")) {
    try (FSDataInputStream fresh = fs.open(path)) {
      fresh.seek(resumePos);
      return fresh.read(buf, off, len);
    }
  }
  throw e;
}

When it happens

Trigger: Any read-family call on a closed striped input stream. Two distinct precursors: the stream itself was closed (this message), or the shared DFSClient/FileSystem was closed first (checkOpen throws 'Filesystem closed' before this line is reached).

Common situations: Reading after the try-with-resources scope ended; one thread closing the stream on error while a reader thread is mid-read; cached FileSystem instances closed during application shutdown; test fixtures reused across test methods.

Related errors

• Stream is closed!
• key + ": Stream is closed!"
• Stream closed
• Attempted to read past end of file
• Cannot seek to negative offset

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/c64421b076d536b2. Report an issue: GitHub.