apache/hadoop · error · IndexOutOfBoundsException

Requested more bytes than destination buffer size: request l

Error message

Requested more bytes than destination buffer size: request length = %s, with offset = %s, buffer capacity = %s

What it means

The second guard in FSUtils.checkReadParameters: after offset passes, it rejects reads where buffer.length < offset + length — i.e. the requested bytes cannot fit in the remaining buffer space. The message (TOO_MANY_BYTES_FOR_DEST_BUFFER from FSExceptionMessages) is the Hadoop-standard wording for 'destination buffer too small for this read'. Integer overflow is also implicitly caught because a wrapped offset+length goes negative and compares less than buffer.length only in pathological cases; treat any hit as a sizing bug at the caller.

Source

Thrown at hadoop-cloud-storage-project/hadoop-tos/src/main/java/org/apache/hadoop/fs/tosfs/util/FSUtils.java:44

import java.net.URI;

public final class FSUtils {
  private static final String OVERFLOW_ERROR_HINT =
      FSExceptionMessages.TOO_MANY_BYTES_FOR_DEST_BUFFER
          + ": request length = %s, with offset = %s, buffer capacity = %s";

  private FSUtils() {
  }

  public static void checkReadParameters(byte[] buffer, int offset, int length) {
    Preconditions.checkArgument(buffer != null, "Null buffer");
    if (offset < 0 || offset > buffer.length) {
      throw new IndexOutOfBoundsException(
          String.format("offset: %s is out of range [%s, %s]", offset, 0, buffer.length));
    }
    Preconditions.checkArgument(length >= 0, "length: %s is negative", length);
    if (buffer.length < offset + length) {
      throw new IndexOutOfBoundsException(
          String.format(OVERFLOW_ERROR_HINT, length, offset, (buffer.length - offset)));
    }
  }

  public static URI normalizeURI(URI fsUri, Configuration hadoopConfig) {
    final String scheme = fsUri.getScheme();
    final String authority = fsUri.getAuthority();

    if (scheme == null && authority == null) {
      fsUri = FileSystem.getDefaultUri(hadoopConfig);
    } else if (scheme != null && authority == null) {
      URI defaultUri = FileSystem.getDefaultUri(hadoopConfig);
      if (scheme.equals(defaultUri.getScheme()) && defaultUri.getAuthority() != null) {
        fsUri = defaultUri;
      }
    }
    return fsUri;
  }

View on GitHub (pinned to 2add963021)

Solutions

  1. Clamp the length: int effectiveLen = Math.min(requestedLen, buffer.length - offset);
  2. Allocate the destination buffer to at least offset + length before the read.
  3. Run FSUtils.checkReadParameters(buffer, offset, length) first to fail with a precise, local stack trace.
  4. In copy loops, recompute both offset and length from the same remaining-bytes value each iteration.

Example fix

// before
byte[] buf = new byte[4096];
in.readFully(buf, 4000, remaining); // 4000 + remaining > 4096

// after
byte[] buf = new byte[4096];
in.readFully(buf, 4000, Math.min(remaining, buf.length - 4000));
Defensive patterns

Strategy: validation

Validate before calling

int effectiveLen = Math.min(length, buffer.length - offset);
if (effectiveLen < 0) { throw new IllegalArgumentException("offset beyond buffer"); }
in.read(buffer, offset, effectiveLen);

Prevention

When it happens

Trigger: read(buffer, offset, length) where offset + length overruns the buffer, e.g. buffer of 4096 with offset 4000 and length 256; readFully(buf, off, fileRemaining) where buf was sized to the previous chunk; passing a full-file length while using a chunk-sized buffer.

Common situations: Reusing one buffer for chunks but passing the total length; computing length as file size instead of min(fileSize - pos, buffer.length - offset); buffer pools handing back smaller buffers than earlier in the run; copy loops that advance offset but forget to shrink length by the same amount.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/3520467e056eb7d7. Report an issue: GitHub.