What causes "Stream closed" in apache/hadoop?

Close exactly once from the owning component, after all writers finish (drain-then-close ordering)

apache/hadoop · error · IOException

Stream closed

Error message

Stream closed

What it means

CryptoOutputStream.checkStream() guards write and flush paths; after close() set closed = true (buffers and codec already returned/freed), any write()/flush() throws IOException("Stream closed") immediately. This prevents writes into freed direct buffers or a closed cipher.

Source

Thrown at hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/crypto/CryptoOutputStream.java:279

   */
  @Override
  public synchronized void flush() throws IOException {
    if (closed) {
      return;
    }
    encrypt();
    super.flush();
  }
  
  @Override
  public void write(int b) throws IOException {
    oneByteBuf[0] = (byte)(b & 0xff);
    write(oneByteBuf, 0, oneByteBuf.length);
  }
  
  private void checkStream() throws IOException {
    if (closed) {
      throw new IOException("Stream closed");
    }
  }
  
  @Override
  public void setDropBehind(Boolean dropCache) throws IOException,
      UnsupportedOperationException {
    try {
      ((CanSetDropBehind) out).setDropBehind(dropCache);
    } catch (ClassCastException e) {
      throw new UnsupportedOperationException("This stream does not " +
          "support setting the drop-behind caching.");
    }
  }

  @Override
  public void hflush() throws IOException {
    flush();
    if (out instanceof Syncable) {

View on GitHub (pinned to 2add963021)

Solutions

Close exactly once from the owning component, after all writers finish (drain-then-close ordering)
Wrap the whole write session in try-with-resources
Signal writers (flag/countdown latch) to stop before closing on error paths
On error recovery, open a new stream instead of reusing the closed one

Example fix

// before
CryptoOutputStream out = ...;
try {
  out.write(data);
} finally {
  out.close();
}
out.flush(); // IOException: Stream closed

// after
try (CryptoOutputStream out = ...) {
  out.write(data);
  out.flush();
} // single close, no post-close use

Defensive patterns

Strategy: validation

Validate before calling

// No public isClosed() on CryptoOutputStream; gate writers with an owned flag.
private final AtomicBoolean closed = new AtomicBoolean(false);

void write(byte[] data) throws IOException {
  if (closed.get()) {
    throw new IllegalStateException("writer already closed");
  }
  out.write(data);
}

void close() throws IOException {
  if (closed.compareAndSet(false, true)) { out.close(); }
}

Try / catch

try {
  out.write(data);
} catch (IOException e) {
  if ("Stream closed".equals(e.getMessage())) {
    // stream was closed under us (timeout/error handler): reopen or fail cleanly
    throw new IllegalStateException("write after close", e);
  }
  throw e;
}

Prevention

Establish drain-then-close ordering: stop producer threads before closing the sink
Use try-with-resources around the full write session
On error paths, signal writers via a flag/latch before closing the stream
After any close, open a new stream for retries instead of reusing the handle

When it happens

Trigger: write()/flush() after close() on a CryptoOutputStream; a producer thread writing while an error/timeout handler closes the stream concurrently; frameworks closing a sink on failure and then retrying writes to the same handle.

Common situations: Producer-consumer pipelines where a bad record closes the output while another thread is mid-write; sinks closed on timeout with queued writers unaware; double-close followed by reuse in wrapper streams.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/c4c56c4af362e64b. Report an issue: GitHub.