apache/hadoop · error · UnsupportedOperationException

zero-copy reads were not available, and the ByteBufferPool d

Error message

zero-copy reads were not available, and the ByteBufferPool did not provide us with {} buffer.

What it means

After fallbackRead obtains a pool, it calls bufferPool.getBuffer(useDirect, maxLength); the pool's contract permits returning null, but the fallback read cannot proceed without a buffer, so it throws UnsupportedOperationException stating that the pool did not provide a direct or indirect buffer. `useDirect` is true when the stream implements ByteBufferReadable - the pool must be able to supply that flavor.

Source

Thrown at hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/ByteBufferUtil.java:68

   *
   * @param stream input stream.
   * @param bufferPool bufferPool.
   * @param maxLength maxLength.
   * @throws IOException raised on errors performing I/O.
   * @return byte buffer.
   */
  public static ByteBuffer fallbackRead(
      InputStream stream, ByteBufferPool bufferPool, int maxLength)
          throws IOException {
    if (bufferPool == null) {
      throw new UnsupportedOperationException("zero-copy reads " +
          "were not available, and you did not provide a fallback " +
          "ByteBufferPool.");
    }
    boolean useDirect = streamHasByteBufferRead(stream);
    ByteBuffer buffer = bufferPool.getBuffer(useDirect, maxLength);
    if (buffer == null) {
      throw new UnsupportedOperationException("zero-copy reads " +
          "were not available, and the ByteBufferPool did not provide " +
          "us with " + (useDirect ? "a direct" : "an indirect") +
          "buffer.");
    }
    Preconditions.checkState(buffer.capacity() > 0);
    Preconditions.checkState(buffer.isDirect() == useDirect);
    maxLength = Math.min(maxLength, buffer.capacity());
    boolean success = false;
    try {
      if (useDirect) {
        buffer.clear();
        buffer.limit(maxLength);
        ByteBufferReadable readable = (ByteBufferReadable)stream;
        int totalRead = 0;
        while (true) {
          if (totalRead >= maxLength) {
            success = true;
            break;

View on GitHub (pinned to 2add963021)

Solutions

Use ElasticByteBufferPool, which never returns null and allocates buffers as needed.
Fix the custom pool to allocate (or block) instead of returning null in the fallback read path, and honor the useDirect flavor request.
Ensure every buffer handed out is returned with releaseBuffer once consumed (including on exception paths).
Fall back to byte[] reads when the pool cannot serve the request.

Example fix

// before
ByteBuffer b = in.read(boundedPool, 65536); // pool returns null -> UOE

// after
ByteBuffer b = in.read(new ElasticByteBufferPool(), 65536);

Defensive patterns

Strategy: fallback

Validate before calling

static ByteBuffer getBufferOrFail(ByteBufferPool pool, boolean direct, int len) {
  ByteBuffer b = pool.getBuffer(direct, len);
  if (b == null) {
    b = direct ? ByteBuffer.allocateDirect(len) : ByteBuffer.allocate(len);
  }
  return b;
}

Try / catch

try {
  buf = in.read(pool, maxLength);
} catch (UnsupportedOperationException e) {
  buf = in.read(new org.apache.hadoop.io.ElasticByteBufferPool(), maxLength);
}

Prevention

Prefer ElasticByteBufferPool, which never returns null
Return every borrowed buffer via releaseBuffer, including on exception paths
If you write a custom pool, honor the useDirect flavor request and allocate instead of returning null on the read path

When it happens

Trigger: read(pool, maxLength) on a non-zero-copy stream with a bounded pool that returns null when exhausted or refuses direct buffers; a custom pool whose getBuffer returns null for the requested flavor; pools starved because buffers are never released back via releaseBuffer.

Common situations: Hand-rolled pools tuned for indirect buffers meet a ByteBufferReadable stream that demands direct ones; long-lived readers leak pooled buffers (missing releaseBuffer) until the pool runs dry; tests with zero-capacity pools.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/de463a77c717e8fa. Report an issue: GitHub.