apache/hadoop · error · IndexOutOfBoundsException

offset: %s is out of range [%s, %s]

Error message

offset: %s is out of range [%s, %s]

What it means

FSUtils.checkReadParameters validates the (buffer, offset, length) triple before every read on the TOS FileSystem input streams. This IndexOutOfBoundsException fires when offset is negative or greater than buffer.length — the Java contract requires 0 <= offset <= buffer.length. It mirrors org.apache.hadoop.fs.FileSystem#verifyReadParameters and means the caller passed a bad starting position, before length is even considered.

Source

Thrown at hadoop-cloud-storage-project/hadoop-tos/src/main/java/org/apache/hadoop/fs/tosfs/util/FSUtils.java:39

import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.FSExceptionMessages;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.util.Preconditions;

import java.net.URI;

public final class FSUtils {
  private static final String OVERFLOW_ERROR_HINT =
      FSExceptionMessages.TOO_MANY_BYTES_FOR_DEST_BUFFER
          + ": request length = %s, with offset = %s, buffer capacity = %s";

  private FSUtils() {
  }

  public static void checkReadParameters(byte[] buffer, int offset, int length) {
    Preconditions.checkArgument(buffer != null, "Null buffer");
    if (offset < 0 || offset > buffer.length) {
      throw new IndexOutOfBoundsException(
          String.format("offset: %s is out of range [%s, %s]", offset, 0, buffer.length));
    }
    Preconditions.checkArgument(length >= 0, "length: %s is negative", length);
    if (buffer.length < offset + length) {
      throw new IndexOutOfBoundsException(
          String.format(OVERFLOW_ERROR_HINT, length, offset, (buffer.length - offset)));
    }
  }

  public static URI normalizeURI(URI fsUri, Configuration hadoopConfig) {
    final String scheme = fsUri.getScheme();
    final String authority = fsUri.getAuthority();

    if (scheme == null && authority == null) {
      fsUri = FileSystem.getDefaultUri(hadoopConfig);
    } else if (scheme != null && authority == null) {
      URI defaultUri = FileSystem.getDefaultUri(hadoopConfig);
      if (scheme.equals(defaultUri.getScheme()) && defaultUri.getAuthority() != null) {

View on GitHub (pinned to 2add963021)

Solutions

1. Fix the call site so offset is within [0, buffer.length]; check the argument order — the classic bug is swapping offset and length.
2. Validate upfront with FSUtils.checkReadParameters(buffer, offset, length) (or equivalent range check) so failures point at your code, not the stack inside the connector.
3. If the offset came from a loop accumulator, audit the loop bounds and how the buffer is re-sliced between iterations.
4. Add a unit test asserting reads with edge offsets 0, buffer.length, and negative values behave as intended.

Example fix

// before
int off = pos; // pos can exceed buf.length
in.read(buf, off, len);

// after
FSUtils.checkReadParameters(buf, off, len);
in.read(buf, Math.max(0, Math.min(off, buf.length)), len);

if (offset < 0 || offset > buffer.length) {
  throw new IllegalArgumentException("bad read offset " + offset + " for buffer of " + buffer.length);
}

When it happens

Trigger: Calling InputStream.read(buffer, offset, length) / FSDataInputStream.readFully variants on a TOS file with a negative offset, or an offset beyond the buffer's length (e.g., offset == buffer.length + 1), including offset > 0 on a zero-length or freshly allocated empty buffer.

Common situations: Caller-side bookkeeping bugs: a progress/position accumulator incremented past the buffer size; reusing a length variable as an offset; readFully into a sub-range computed as (buf, remaining, chunk) with arguments transposed; empty buffers from list-driven code where a 0-byte read buffer still gets a nonzero offset.

Related errors

• Requested more bytes than destination buffer size: request l
• Invalid read parameters: buf.length=%d, off=%d, len=%d
• Requested more bytes than destination buffer size: request l
• write (b[{b.length}], {off}, {len})
• Unsupported block buffer "{name}"

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/cea598d36f9ec5af. Report an issue: GitHub.