apache/hadoop · error · IndexOutOfBoundsException

Limit exceeds buffer size

Error message

Limit exceeds buffer size

What it means

reset(int newlim) re-arms the bounded stream to accept newlim more bytes starting from startOffset. It throws IndexOutOfBoundsException when newlim exceeds the usable capacity (buffer.length - startOffset), because the underlying array cannot physically hold that many bytes.

Source

Thrown at hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/BoundedByteArrayOutputStream.java:105

    } else if (len == 0) {
      return;
    }

    if (currentPointer + len > limit) {
      throw new EOFException("Reach the limit of the buffer");
    }

    System.arraycopy(b, off, buffer, currentPointer, len);
    currentPointer += len;
  }

  /**
   * Reset the limit 
   * @param newlim New Limit
   */
  public void reset(int newlim) {
    if (newlim > (buffer.length - startOffset)) {
      throw new IndexOutOfBoundsException("Limit exceeds buffer size");
    }
    this.limit = newlim;
    this.currentPointer = startOffset;
  }

  /** Reset the buffer */
  public void reset() {
    this.limit = buffer.length - startOffset;
    this.currentPointer = startOffset;
  }

  /**
   * Return the current limit.
   * @return limit.
   */
  public int getLimit() {
    return limit;
  }

View on GitHub (pinned to 2add963021)

Solutions

1. Clamp the new limit to buffer.length - startOffset before calling reset.
2. Keep limit and capacity in one place (single constant/config) so they cannot diverge.
3. If you truly need more room, allocate a larger byte[] and construct a new stream (or use resetBuffer on a subclass) instead of resetting beyond capacity.

Example fix

// before
bbos.reset(neededSize); // throws if neededSize > buffer.length - startOffset

// after
bbos.reset(Math.min(neededSize, buffer.length - startOffset));

int usable = buffer.length - startOffset;
if (newlim > usable) newlim = usable;
bbos.reset(newlim);

try {
  bbos.reset(newlim);
} catch (IndexOutOfBoundsException e) {
  throw new IllegalStateException("Requested limit " + newlim + " exceeds usable capacity " + (buffer.length - startOffset), e);
}

When it happens

Trigger: bbos.reset(buffer.length + extra) or reset(softLimit) after the buffer was constructed with an offset leaving less usable room; passing a size in different units (KB vs bytes) than the buffer was allocated with.

Common situations: Retrying a too-large record with 'reset to needed size' logic that forgets the start offset; configuration units mismatch (softLimit configured in KB, buffer allocated in bytes).

Related errors

• Invalid capacity/limit
• Cannot seek to a negative offset
• Requested more bytes than destination buffer size: request l
• write (b[{b.length}], {off}, {len})
• Reaching the limit of the buffer.

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/2b2e639ae457ef95. Report an issue: GitHub.