apache/hadoop · error · IOException

stream closed

Error message

stream closed

What it means

After passing argument validation, CBZip2OutputStream.write(byte[], offs, len) checks the lifecycle: once close() has run, this.out is null and the method throws IOException("stream closed") instead of writing. The compressor is single-use; all data must be written before close. (Note the sibling single-byte write() throws the shorter message "closed" — same condition.)

Source

Thrown at hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/bzip2/CBZip2OutputStream.java:873

  public final int getBlockSize() {
    return this.blockSize100k;
  }

  @Override
  public void write(final byte[] buf, int offs, final int len)
      throws IOException {
    if (offs < 0) {
      throw new IndexOutOfBoundsException("offs(" + offs + ") < 0.");
    }
    if (len < 0) {
      throw new IndexOutOfBoundsException("len(" + len + ") < 0.");
    }
    if (offs + len > buf.length) {
      throw new IndexOutOfBoundsException("offs(" + offs + ") + len("
          + len + ") > buf.length(" + buf.length + ").");
    }
    if (this.out == null) {
      throw new IOException("stream closed");
    }

    for (int hi = offs + len; offs < hi;) {
      write0(buf[offs++]);
    }
  }

  private void write0(int b) throws IOException {
    if (this.currentChar != -1) {
      b &= 0xff;
      if (this.currentChar == b) {
        if (++this.runLength > 254) {
          writeRun();
          this.currentChar = -1;
          this.runLength = 0;
        }
        // else nothing to do
      } else {

View on GitHub (pinned to 2add963021)

Solutions

  1. Order closes outside-in: write everything, close the outer wrapper (which flushes), let it close the CBZip2OutputStream — never close the inner stream manually when it is wrapped.
  2. Guard with an application-level closed flag checked before each write batch.
  3. Cancel or join any async flusher/scheduler threads before closing the stream.
  4. Make error paths idempotent: close exactly once (e.g. via try-with-resources) and stop producing after an exception.

Example fix

// before
DataOutputStream dos = new DataOutputStream(cbz);
cbz.close();      // inner stream closed first
dos.writeInt(x);  // flush later hits IOException: stream closed

// after
try (DataOutputStream dos = new DataOutputStream(
        new CBZip2OutputStream(out))) {
  dos.writeInt(x);
} // wrapper flushes, then closes cbz exactly once
Defensive patterns

Strategy: try-catch

Validate before calling

private final AtomicBoolean closed = new AtomicBoolean(false);

void writeAll(byte[] b, int off, int len) throws IOException {
  if (closed.get()) throw new IllegalStateException("stream already closed");
  cbz.write(b, off, len);
}

Try / catch

try {
  cbz.write(buf, off, len);
} catch (IOException e) {
  if ("stream closed".equals(e.getMessage())) {
    // lifecycle bug: a wrapper flushed after close; fix close ordering
    throw new IllegalStateException("write after close (wrapper flush order)", e);
  }
  throw e;
}

Prevention

When it happens

Trigger: Calling write(byte[], offs, len) on a CBZip2OutputStream after close(): output moved/rotated and a late buffer flushed by a wrapper, error paths that closed the stream then continued, or two components both writing to a stream one of them already closed.

Common situations: Wrapped streams (DataOutputStream/CountingOutputStream over CBZip2OutputStream) flushing buffered data during their own close() after the inner bzip2 stream was closed first; scheduled/async flush tasks surviving past file close; MapReduce/Hive output committers closing streams before a final empty-buf write; retry logic re-entering a write loop after cleanup.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/a8b91d17af6542c1. Report an issue: GitHub.