apache/hadoop · error · IndexOutOfBoundsException

Requested more bytes than destination buffer size: request l

Error message

Requested more bytes than destination buffer size: request length ={}, with offset ={}; buffer capacity ={}

What it means

In the block buffer's read(byte[] b, int offset, int length), a precondition asserts the destination array has room: if b.length - offset < length it throws IndexOutOfBoundsException with FSExceptionMessages.TOO_MANY_BYTES_FOR_DEST_BUFFER plus the request length, offset, and capacity. This mirrors java.io.DataInputStream semantics: the caller asked to read more bytes than the destination slice can hold. Preconditions on length>=0 and non-null b are checked just before, so those produce IllegalArgumentException instead.

Source

Thrown at hadoop-cloud-storage-project/hadoop-huaweicloud/src/main/java/org/apache/hadoop/fs/obs/OBSDataBlocks.java:783

      /**
       * Read in data.
       *
       * @param b      destination buffer
       * @param offset offset within the buffer
       * @param length length of bytes to read
       * @return read size
       * @throws EOFException              if the position is negative
       * @throws IndexOutOfBoundsException if there isn't space for the amount
       *                                   of data requested.
       * @throws IllegalArgumentException  other arguments are invalid.
       */
      public synchronized int read(final byte[] b, final int offset,
          final int length)
          throws IOException {
        Preconditions.checkArgument(length >= 0, "length is negative");
        Preconditions.checkArgument(b != null, "Null buffer");
        if (b.length - offset < length) {
          throw new IndexOutOfBoundsException(
              FSExceptionMessages.TOO_MANY_BYTES_FOR_DEST_BUFFER
                  + ": request length ="
                  + length
                  + ", with offset ="
                  + offset
                  + "; buffer capacity ="
                  + (b.length - offset));
        }
        verifyOpen();
        if (!hasRemaining()) {
          return -1;
        }

        int toRead = Math.min(length, available());
        byteBuffer.get(b, offset, toRead);
        return toRead;
      }

View on GitHub (pinned to 2add963021)

Solutions

1. Validate arguments before the call: require offset >= 0, length >= 0, and offset + length <= b.length
2. Compute the requested length as min(length, b.length - offset) so it is always clipped to the destination slice
3. Add unit tests for boundary offsets (offset = b.length, length = 0 is legal; offset+length = b.length is the max)

Example fix

// before
int n = blockStream.read(buf, offset, length); // throws IndexOutOfBoundsException

// after
int maxLen = Math.min(length, buf.length - offset);
int n = blockStream.read(buf, offset, maxLen);

void checkReadArgs(byte[] b, int off, int len) {
  if (b == null) throw new IllegalArgumentException("Null buffer");
  if (len < 0) throw new IllegalArgumentException("length is negative");
  if (off < 0 || b.length - off < len) {
    throw new IllegalArgumentException(
        "offset+length exceeds buffer: off=" + off + ", len=" + len + ", cap=" + b.length);
  }
}
checkReadArgs(buf, offset, length);
int n = blockStream.read(buf, offset, length);

try {
  n = blockStream.read(buf, offset, length);
} catch (IndexOutOfBoundsException e) {
  // caller bug: fix argument arithmetic; do not retry
  throw new IllegalArgumentException("bad read args: off=" + offset + ", len=" + length + ", cap=" + buf.length, e);
}

When it happens

Trigger: Calling read(buf, offset, length) where offset+length > buf.length (e.g. buf=new byte[1024], offset=1000, length=100); reusing a shared buffer with a stale length from a previous, larger buffer; passing offset equal to buf.length with length>0.

Common situations: Off-by-one arithmetic in custom InputStream consumers; buffer-pool code that swaps in smaller buffers but keeps lengths computed against the old size; interop code that assumes read() clips the request instead of throwing.

Related errors

• Cannot seek to a negative offset
• Unsupported block buffer "{}"
• Stream is closed!
• Attempted to seek or read past the end of the file
• Can't open {} because it is a directory

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/935267eead71d997. Report an issue: GitHub.