What causes "getKey() + ": Stream is closed!"" in apache/hadoop?

Use try-with-resources so exactly one owner closes the stream and nothing reads after.

apache/hadoop · error · IOException

getKey() + ": Stream is closed!"

Error message

getKey() + ": Stream is closed!"

What it means

AnalyticsStream.throwIfClosed() throws IOException("<key>: Stream is closed!") when read(), seek(), available(), getPos(), readVectored() etc. are called after close(). The stream over the S3 Select query payload is single-use; once closed, every I/O method rejects use.

Source

Thrown at hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/impl/streams/AnalyticsStream.java:328

   * from parquet optimisations.
   * Else, AAL will make a decision on which optimisations based on the file extension,
   * if the file ends in .par or .parquet, then parquet specific optimisations are used.
   *
   * @param inputPolicy S3A's input file policy passed down when opening the file
   * @return the AAL read policy
   */
  private InputPolicy mapS3AInputPolicyToAAL(S3AInputPolicy inputPolicy) {
    switch (inputPolicy) {
    case Sequential:
      return InputPolicy.Sequential;
    default:
      return InputPolicy.None;
    }
  }

  protected void throwIfClosed() throws IOException {
    if (closed) {
      throw new IOException(getKey() + ": " + FSExceptionMessages.STREAM_IS_CLOSED);
    }
  }

  /**
   * Increment the bytes read counter if there is a stats instance
   * and the number of bytes read is more than zero.
   * @param bytesRead number of bytes read
   */
  private void incrementBytesRead(long bytesRead) {
    getS3AStreamStatistics().bytesRead(bytesRead);
    if (getContext().getStats() != null && bytesRead > 0) {
      getContext().getStats().incrementBytesRead(bytesRead);
    }
  }
}

View on GitHub (pinned to 2add963021)

Solutions

Use try-with-resources so exactly one owner closes the stream and nothing reads after.
If wrapping the stream, document/track close ownership (who closes: wrapper or caller - never both).
Null out or guard references after close; add a closed flag in your wrapper and check it before delegating.
Catch IOException and treat 'Stream is closed' as a lifecycle bug to fix, not a transient failure to retry.

Example fix

// before
FSDataInputStream in = fs.selectQuery(...).getInputStream();
process(in);
in.close();
process(in); // IOException: Stream is closed!

// after
try (FSDataInputStream in = fs.selectQuery(...).getInputStream()) {
  process(in);
}

Defensive patterns

Strategy: validation

Validate before calling

// single-owner lifecycle: nothing reads after the try block closes
try (FSDataInputStream in = fs.selectQuery(select).getInputStream()) {
  return consume(in);
}

Try / catch

try {
  return in.read();
} catch (IOException e) {
  if (e.getMessage() != null && e.getMessage().endsWith("Stream is closed")) {
    // lifecycle bug: reopen the stream instead of retrying blindly
    try (FSDataInputStream retry = fs.selectQuery(select).getInputStream()) {
      return consume(retry);
    }
  }
  throw e;
}

Prevention

Always open S3 Select result streams with try-with-resources.
Document which wrapper owns close() when decorating streams.
Never cache the stream beyond the scope that opened it.

When it happens

Trigger: Any read/seek on the FSDataInputStream from selectQuery() after close(); two owners closing/reading the same stream (e.g. wrapper closes, then caller reads); finally-block close followed by retry logic reading again.

Common situations: Manual stream management without try-with-resources; libraries wrapping the stream and closing it early (decompressors, JSON parsers); double-processing of a stream in error handlers; keeping a cached stream reference across request lifecycles.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/4f2f87d8dec1e522. Report an issue: GitHub.