apache/hadoop · error · IOException

Stream closed

Error message

Stream closed

What it means

ByteRangeInputStream (the base of WebHdfsFileSystem.OffsetUrlInputStream, i.e., every WebHDFS/SWebHDFS FSDataInputStream over HTTP) is a state machine with StreamStatus NORMAL/SEEK/CLOSED. close() sets status=CLOSED; any later read()/available() calls getInputStream(), which throws this IOException. It is the HTTP equivalent of reading a stream after close() — the object cannot be reopened, you must call FileSystem.open() again.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/web/ByteRangeInputStream.java:124

  protected abstract URL getResolvedUrl(final HttpURLConnection connection
  ) throws IOException;

  @VisibleForTesting
  protected InputStream getInputStream() throws IOException {
    switch (status) {
    case NORMAL:
      break;
    case SEEK:
      if (in != null) {
        in.close();
      }
      InputStreamAndFileLength fin = openInputStream(startPos);
      in = fin.in;
      fileLength = fin.length;
      status = StreamStatus.NORMAL;
      break;
    case CLOSED:
      throw new IOException("Stream closed");
    }
    return in;
  }

  @VisibleForTesting
  protected InputStreamAndFileLength openInputStream(long startOffset)
      throws IOException {
    if (startOffset < 0) {
      throw new EOFException("Negative Position");
    }
    // Use the original url if no resolved url exists, eg. if
    // it's the first time a request is made.
    final boolean resolved = resolvedURL.getURL() != null;
    final URLOpener opener = resolved? resolvedURL: originalURL;

    final HttpURLConnection connection = opener.connect(startOffset, resolved);
    resolvedURL.setURL(getResolvedUrl(connection));

View on GitHub (pinned to 2add963021)

Solutions

  1. Move all reads inside the stream's owning scope (try-with-resources block) so nothing reads after close
  2. If the stream is shared, give it exactly one owner responsible for close, or wrap it in an uncloseable filter (CloseShieldInputStream) for secondary readers
  3. Reopen with fs.open(path) when data is still needed after close

Example fix

// before
try (FSDataInputStream in = fs.open(path)) {
  in.read(buf);
}
long skipped = in.read(); // IOException: Stream closed

// after
try (FSDataInputStream in = fs.open(path)) {
  in.read(buf);
  long extra = in.read(); // stays inside the owning scope
}
Defensive patterns

Strategy: validation

Validate before calling

// wrap shared streams so secondary readers cannot close the primary
InputStream shared = org.apache.commons.io.input.CloseShieldInputStream.wrap(fs.open(path));

Try / catch

try {
  int b = in.read();
} catch (IOException e) {
  if ("Stream closed".equals(e.getMessage())) {
    // lifecycle bug in caller code: reopen, do not retry the dead handle
    try (FSDataInputStream fresh = fs.open(path)) { /* reread */ }
  } else { throw e; }
}

Prevention

When it happens

Trigger: Call read(), read(byte[],int,int), or available() on a WebHDFS FSDataInputStream after close() — e.g., reading in a finally block after a try-with-resources block already closed it, or keeping a cached stream handle past its consumer's lifetime.

Common situations: try-with-resources that closes the stream while a helper (checksum computation, logging, metrics) still reads from it; sharing one open stream across components where one closes it; double-ownership of streams in wrappers. close() itself is safe to call twice — only subsequent reads fail.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/75755b41079ad057. Report an issue: GitHub.