apache/hadoop · error · IndexOutOfBoundsException

Requested more bytes than destination buffer size: request l

Q: What causes "Requested more bytes than destination buffer size: request length ={length}, with offset ={offset}; buffer capacity ={b.length - offset}" in apache/hadoop?

Allocate the destination with at least offset + length bytes

Error message

Requested more bytes than destination buffer size: request length ={length}, with offset ={offset}; buffer capacity ={b.length - offset}

What it means

The positional read read(b, offset, length) first rejects a negative length and a null buffer (Preconditions IllegalArgumentException), then checks destination capacity: if b.length - offset < length it throws IndexOutOfBoundsException with 'Requested more bytes than destination buffer size' plus the request length, offset and capacity. verifyOpen() runs after these checks, so a bounds error can surface even on a closed stream.

Source

Thrown at hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/store/ByteBufferInputStream.java:175

  }

  /**
   * Read in data.
   * @param b destination buffer.
   * @param offset offset within the buffer.
   * @param length length of bytes to read.
   * @throws EOFException if the position is negative
   * @throws IndexOutOfBoundsException if there isn't space for the
   * amount of data requested.
   * @throws IllegalArgumentException other arguments are invalid.
   */
  @SuppressWarnings("NullableProblems")
  public synchronized int read(byte[] b, int offset, int length)
      throws IOException {
    Preconditions.checkArgument(length >= 0, "length is negative");
    Preconditions.checkArgument(b != null, "Null buffer");
    if (b.length - offset < length) {
      throw new IndexOutOfBoundsException(
          FSExceptionMessages.TOO_MANY_BYTES_FOR_DEST_BUFFER
              + ": request length =" + length
              + ", with offset =" + offset
              + "; buffer capacity =" + (b.length - offset));
    }
    verifyOpen();
    if (!hasRemaining()) {
      return -1;
    }

    int toRead = Math.min(length, available());
    byteBuffer.get(b, offset, toRead);
    return toRead;
  }

  @Override
  public String toString() {
    return "ByteBufferInputStream{" +

View on GitHub (pinned to 2add963021)

Solutions

Allocate the destination with at least offset + length bytes
Clamp the request: len = Math.min(len, b.length - offset)
Assert b.length - offset >= length in debug paths before the call

Example fix

// before
byte[] small = new byte[16];
in.read(small, 8, 32); // capacity 8 < requested 32

// after
byte[] buf = new byte[8 + 32];
in.read(buf, 8, 32);

Defensive patterns

Strategy: validation

Validate before calling

int maxLen = b.length - offset;
if (length > maxLen) {
  length = maxLen; // or throw your own clearer error
}
int n = in.read(b, offset, length);

Try / catch

Catch IndexOutOfBoundsException from read(byte[], int, int) and report buffer capacity versus request length; the message already contains all three numbers.

Prevention

Size buffers as offset + maxRead
Clamp lengths to b.length - offset
Keep offset at 0 unless the buffer was sized for the offset

When it happens

Trigger: read(buf, off, len) with len greater than buf.length - off; reusing a non-zero offset with a buffer smaller than offset + length.

Common situations: Buffer math copied from code where offset was 0; reads sized by a record length into a header-sized buffer; off-by-one in loop bounds.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/6bd07f7bd91b56e6. Report an issue: GitHub.