apache/beam · error · IllegalArgumentException

offset + length exceeds the buffer limit. offset: , length:

Error message

offset + length exceeds the buffer limit. offset: , length: , limit:

What it means

SubstringByteArrayOutputStream.toString validates that offset + length does not exceed the total bytes written (count). This ensures the requested slice [offset, offset+length) is entirely within the buffer, preventing partial/garbage reads or ArrayIndexOutOfBoundsException from the underlying copy.

Source

Thrown at sdks/java/core/src/main/java/org/apache/beam/sdk/io/TextSource.java:484

   * String s = new String(buffer, offset, length); // 2nd-copy
   * }</pre>
   */
  static class SubstringByteArrayOutputStream extends ByteArrayOutputStream {
    public String toString(int offset, int length, Charset charset) {
      if (offset < 0) {
        throw new IllegalArgumentException("offset is negative: " + offset);
      }
      if (offset > count) {
        throw new IllegalArgumentException(
            "offset exceeds the buffer limit. offset: " + offset + ", limit: " + count);
      }

      if (length < 0) {
        throw new IllegalArgumentException("length is negative: " + length);
      }

      if (offset + length > count) {
        throw new IllegalArgumentException(
            "offset + length exceeds the buffer limit. offset: "
                + offset
                + ", length: "
                + length
                + ", limit: "
                + count);
      }

      return new String(buf, offset, length, charset);
    }
  }

  /**
   * See <a
   * href="https://en.wikipedia.org/wiki/Knuth%E2%80%93Morris%E2%80%93Pratt_algorithm">Knuth–Morris–Pratt
   * algorithm</a>.
   */
  static class KMPDelimiterFinder {

View on GitHub (pinned to 12126d8942)

Solutions

Before slicing, assert offset + length <= buffer.count and handle the truncated-record case (wait for more data or error out).
Clamp length to Math.min(length, buffer.count - offset) if truncation is acceptable in your context.
Fix upstream EOF handling so records are never sliced before the full record has been buffered.

Example fix

// before
String record = buffer.toString(offset, expectedLen, UTF_8);

// after
checkState(offset + expectedLen <= buffer.count, "incomplete record buffered");
String record = buffer.toString(offset, expectedLen, UTF_8);

Defensive patterns

Strategy: validation

Validate before calling

checkState(offset + len <= buffer.count, "requested record [%s, %s) exceeds buffered %s bytes", offset, offset + len, buffer.count);

Try / catch

try {
  String record = buffer.toString(offset, len, UTF_8);
} catch (IllegalArgumentException e) {
  // handle incomplete record: wait for more data or fail the read
}

Prevention

Only slice after confirming the full record has been buffered (EOF handling).
Clamp length to buffer.count - offset when truncation is acceptable.
Track buffered bytes vs expected record sizes explicitly.

When it happens

Trigger: Calling toString(offset, length, charset) where offset + length > count — e.g. length computed from an expected record size larger than what was actually buffered, or offsets from stale buffer state after more/less data was appended than assumed.

Common situations: TextSource custom record-finding logic that assumes a full record was read when the stream ended mid-record; buffer reuse across reads where offsets were computed against an old buffer length; reading past EOF because end-of-stream checks were skipped.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/a6c464e81d2887ac. Report an issue: GitHub.