apache/hadoop · error · IOException

Stream closed

Error message

Stream closed

What it means

WebHdfsInputStream.read(byte[],int,int) (inner runner of WebHdfsFileSystem) throws IOException('Stream closed') when the underlying FsPathResponseRunner state is RunnerState.CLOSED. It means read() was invoked after close() (or abort) already shut the runner down. The check is the stream's lifecycle guard, equivalent to the standard 'read after close' error in java.io streams.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java:2522

            feInfoStr.getBytes(StandardCharsets.UTF_8));
        feInfo = PBHelperClient
            .convert(FileEncryptionInfoProto.parseFrom(decodedBytes));
      }
      String location = conn.getHeaderField("Location");
      if (location != null) {
        // This saves the location for datanode where redirect was issued.
        // Need to remove offset because seek can be called after open.
        resolvedUrl = removeOffsetParam(new URL(location));
      } else {
        // This is cached for proxies like httpfsfilesystem.
        cachedConnection = conn;
      }
      originalUrl = super.getUrl();
    }

    int read(byte[] b, int off, int len) throws IOException {
      if (runnerState == RunnerState.CLOSED) {
        throw new IOException("Stream closed");
      }
      if (len == 0) {
        return 0;
      }

      // Before the first read, pos and fileLength will be 0 and readBuffer
      // will all be null. They will be initialized once the first connection
      // is made. Only after that it makes sense to compare pos and fileLength.
      if (pos >= fileLength && readBuffer != null) {
        return -1;
      }

      // If a seek is occurring, the input stream will have been closed, so it
      // needs to be reopened. Use the URLRunner to call AbstractRunner#connect
      // with the previously-cached resolved URL and with the 'redirected' flag
      // set to 'true'. The resolved URL contains the URL of the previously
      // opened DN as opposed to the NN. It is preferable to use the resolved
      // URL when creating a connection because it does not hit the NN or every

View on GitHub (pinned to 2add963021)

Solutions

  1. Ensure every read happens inside the stream's lifecycle scope (try-with-resources) and the reference is not reused afterwards
  2. Null out or clearly scope references after close so later use fails as NPE at your own code, not deep in Hadoop
  3. For framework-driven cancels, check the cancel/interrupt flag in your read loop before each read
  4. Wrap reads in a guard that verifies the stream is still open when multiple threads share it (single ownership is better)

Example fix

// before
FSDataInputStream in;
try (FSDataInputStream s = fs.open(p)) { in = s; }
in.read(buf); // runner already CLOSED -> IOException
// after
try (FSDataInputStream in = fs.open(p)) {
  while (in.read(buf) != -1) { /* process */ }
} // all reads stay inside the scope
Defensive patterns

Strategy: validation

Validate before calling

boolean open = true;
try (FSDataInputStream in = fs.open(p)) {
  open = true;
  while (open && in.read(buf) != -1) { process(buf); }
} finally {
  open = false; // guard flag consulted by any shared reader before read()
}

Try / catch

try {
  return in.read(buf, off, len);
} catch (IOException e) {
  if ("Stream closed".equals(e.getMessage())) {
    return -1; // or raise a domain-specific 'already closed' signal
  }
  throw e;
}

Prevention

When it happens

Trigger: Calling read() on an FSDataInputStream obtained from fs.open() after close() returned; concurrent close from another thread (cancel/timeout path) racing a read; keeping a reference to the stream outside its try-with-resources scope and using it later.

Common situations: Stream reference leaks past its try block (e.g., returned from a helper that closed it); reader threads not joined before the owner closes the stream; input-format/record-reader implementations that read one more record after a cancel() from the framework (task kill, speculative preemption).

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/1a34c7f62b53a614. Report an issue: GitHub.