apache/hadoop · error · IOException

Stream closed.

Error message

Stream closed.

What it means

Thrown by AliyunOSSBlockOutputStream.checkOpen(), which every write, flush, and sync entry point calls before touching the stream. The AtomicBoolean 'closed' is set in the close()/abort path, so this IOException means the block output stream was already closed (or aborted after a failed multipart upload) and the application kept writing or flushing.

Source

Thrown at hadoop-tools/hadoop-aliyun/src/main/java/org/apache/hadoop/fs/aliyun/oss/AliyunOSSBlockOutputStream.java:101

   * @return the active block; null if there isn't one.
   * @throws IOException on any failure to create
   */
  private synchronized OSSDataBlocks.DataBlock createBlockIfNeeded()
      throws IOException {
    if (activeBlock == null) {
      blockId++;
      activeBlock = blockFactory.create(blockId, blockSize, statistics);
    }
    return activeBlock;
  }

  /**
   * Check for the filesystem being open.
   * @throws IOException if the filesystem is closed.
   */
  void checkOpen() throws IOException {
    if (closed.get()) {
      throw new IOException("Stream closed.");
    }
  }

  /**
   * The flush operation does not trigger an upload; that awaits
   * the next block being full. What it does do is call {@code flush() }
   * on the current block, leaving it to choose how to react.
   * @throws IOException Any IO problem.
   */
  @Override
  public synchronized void flush() throws IOException {
    checkOpen();

    OSSDataBlocks.DataBlock dataBlock = getActiveBlock();
    if (dataBlock != null) {
      dataBlock.flush();
    }
  }

View on GitHub (pinned to 2add963021)

Solutions

1. Audit the job for code paths that write or flush after close(); ensure exactly one component owns closing the OSS output stream (usually the output committer)
2. Remove redundant close() calls, or guard subsequent writes with a closed flag on your writer
3. Use try-with-resources so the stream closes once, deterministically, at scope exit

Example fix

// before
out.write(data);
out.close();
out.flush(); // Stream closed.

// after
try (FSDataOutputStream out = fs.create(path, overwrite)) {
  out.write(data);
}

// track ownership; only the owner closes
private boolean closed = false;
void writeSafe(FSDataOutputStream out, byte[] b) throws IOException {
  if (closed) return;
  out.write(b);
}

catch (IOException e) { if ("Stream closed.".equals(e.getMessage())) { /* downstream already closed: log and stop writing */ } else throw e; }

When it happens

Trigger: Calling write()/flush()/sync() on an FSDataOutputStream wrapping AliyunOSSBlockOutputStream after close() returned; two threads sharing one output stream where one closes it; double-closing from a finally block combined with a subsequent flush in another finally; writing after a failed multipart upload already set closed=true in the finally clause of close().

Common situations: MapReduce/Spark task cleanup writing trailer bytes after the committer closed the file; inconsistent close ordering in try-with-resources plus manual close; a task attempt that was aborted mid-write but whose record writer continues emitting.

Related errors

• Credentials should not be null.
• Invalid credentials
• Failed to multipart upload to oss, abort it.
• Multi-part upload with id '{uploadId}' to {key}
• Append is not supported!

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/4f99fb9d7dc11ec2. Report an issue: GitHub.