apache/hadoop · warning · InterruptedIOException

Stream closed or unbuffer is called

Error message

Stream closed or unbuffer is called

What it means

InterruptedIOException thrown by S3AInputStream.checkIfVectoredIOStopped(), which is polled inside vectored-read loops. Setting the stopVectoredIOOperations flag - done by close() and unbuffer() - makes all in-flight vectored reads terminate promptly so HTTP streams and buffers are released instead of leaking.

Source

Thrown at hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AInputStream.java:1197

      throw ex;
    } finally {
      tracker.close();
    }
    changeTracker.processResponse(objectRange.response(), operationName,
            position);
    return objectRange;
  }

  /**
   * Check if vectored io operation has been stooped. This happens
   * when the stream is closed or unbuffer is called.
   * @throws InterruptedIOException throw InterruptedIOException such
   *                                that all running vectored io is
   *                                terminated thus releasing resources.
   */
  private void checkIfVectoredIOStopped() throws InterruptedIOException {
    if (stopVectoredIOOperations.get()) {
      throw new InterruptedIOException("Stream closed or unbuffer is called");
    }
  }

  @Override
  public synchronized void setReadahead(Long readahead) {
    this.readahead = validateReadahead(readahead);
  }

  /**
   * Get the current readahead value.
   * @return a non-negative readahead value
   */
  public synchronized long getReadahead() {
    return readahead;
  }

  /**
   * Calculate the limit for a get request, based on input policy

View on GitHub (pinned to 2add963021)

Solutions

1. Treat it as an interruption, not data corruption: reopen the stream (fs.open) and reissue the readVectored when the data is still needed
2. Fix stream ownership so the reader finishes or cancels its vectored-read futures before anyone calls unbuffer()/close()
3. If you call unbuffer() deliberately, consume or cancel the outstanding futures first
4. Propagate the interrupt properly in worker threads so thread pools are not left in a bad state

Example fix

// before
List<CompletableFuture<ByteBuffer>> futs = in.readVectored(ranges, ByteBuffer::allocate);
engine.unbuffer(in);            // or close() from another thread
futs.get(0).join();            // InterruptedIOException: Stream closed or unbuffer is called

// after - consume futures before releasing the stream
List<CompletableFuture<ByteBuffer>> futs = in.readVectored(ranges, ByteBuffer::allocate);
ByteBuffer b = futs.get(0).join();
in.unbuffer();                  // release resources after consumption

try {
  in.readVectored(ranges, ByteBuffer::allocate).forEach(f -> f.join());
} catch (CompletionException e) {
  if (e.getCause() instanceof InterruptedIOException) {
    in = fs.open(path); // stream was closed/unbuffered: reopen if data is still needed
  } else {
    throw e;
  }
}

When it happens

Trigger: Another thread calls close() or unbuffer() on the stream while readVectored() ranges are still being read; the engine releases resources between query stages while a vectored read is in flight; task cancellation closing inputs mid-read.

Common situations: Hive/Spark calling unbuffer() after a query phase or on task switch; shared streams where one consumer closes while another still reads; preemption cancelling tasks that own the stream.

Related errors

• <uri>: Stream is closed!
• End of stream reached draining data between ranges; expected
• HTTP stream closed before all bytes were read. Expected %,d
• Cannot seek to a negative offset <targetPos>
• End of file reached before reading fully.

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/1c8877294bc07b5f. Report an issue: GitHub.