apache/hadoop · error · IOException

Stream is closed!

Error message

Stream is closed!

What it means

ByteBufferInputStream (used by the bytebuffer block buffer for in-flight fast-upload data) tracks liveness by nulling its byteBuffer on close(); verifyOpen() throws IOException(FSExceptionMessages.STREAM_IS_CLOSED) whenever a read/skip/available call happens after that. It means the block's input stream was already closed — either explicitly, by the framework after a successful multipart commit, or after an abort — and code kept reading from it. The javadoc on the read path explicitly declares '@throws IOException if the stream is closed'.

Source

Thrown at hadoop-cloud-storage-project/hadoop-huaweicloud/src/main/java/org/apache/hadoop/fs/obs/OBSDataBlocks.java:695

       * After the stream is closed, set the local reference to the byte buffer
       * to null; this guarantees that future attempts to use stream methods
       * will fail.
       */
      @Override
      public synchronized void close() {
        LOG.debug("ByteBufferInputStream.close() for {}",
            ByteBufferBlock.super.toString());
        byteBuffer = null;
      }

      /**
       * Verify that the stream is open.
       *
       * @throws IOException if the stream is closed
       */
      private void verifyOpen() throws IOException {
        if (byteBuffer == null) {
          throw new IOException(FSExceptionMessages.STREAM_IS_CLOSED);
        }
      }

      public synchronized int read() {
        if (available() > 0) {
          return byteBuffer.get() & OBSCommonUtils.BYTE_TO_INT_MASK;
        } else {
          return -1;
        }
      }

      @Override
      public synchronized long skip(final long offset)
          throws IOException {
        verifyOpen();
        long newPos = position() + offset;
        if (newPos < 0) {
          throw new EOFException(FSExceptionMessages.NEGATIVE_SEEK);

View on GitHub (pinned to 2add963021)

Solutions

1. Audit call sites to ensure no read/skip happens after close() — structure code so close() is the terminal operation in a finally block owned by one place only
2. Wrap usage in try-with-resources so the stream cannot outlive the reading scope
3. If a wrapper stream is involved, make its close() idempotent and prevent read forwarding once closed
4. Check for a preceding exception in logs: the connector may have aborted blocks (closing their streams) before your code touched them again

Example fix

// before
InputStream in = block.getBlockData();
int b = in.read(); // after commit -> IOException: Stream is closed!
in.close();

// after
try (InputStream in = block.getBlockData()) {
  int b = in.read(); // always inside the open window
}

if (in instanceof java.io.Closeable && !isClosed(in)) { /* no portable open-state probe; rely on ownership instead */ }

try {
  return blockStream.read();
} catch (IOException e) {
  if (FSExceptionMessages.STREAM_IS_CLOSED.equals(e.getMessage())) {
    // stream already committed/closed: stop reading, do not retry with the same handle
    return -1;
  }
  throw e;
}

When it happens

Trigger: Calling read()/skip()/available() on a ByteBufferBlock-backed InputStream after close() was invoked; holding a reference to the block stream across the OBSBlockOutputStream close/commit lifecycle and reading from it afterwards; a finally block that closes the stream while another thread is still inside a read loop.

Common situations: Job code that caches FSDataInputStream objects past the owning stream's lifetime; double-close followed by reuse; hand-rolled input-stream wrappers that buffer reads ahead after the inner stream is closed by a timeout/abort path.

Related errors

• {}: Stream is closed!
• Filesystem %s closed
• Cannot seek to a negative offset
• Attempted to seek or read past the end of the file
• Requested more bytes than destination buffer size: request l

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/508f86b29bdb1e50. Report an issue: GitHub.