apache/beam · error · IllegalArgumentException

offset exceeds the buffer limit. offset: , limit:

Error message

offset exceeds the buffer limit. offset: , limit: 

What it means

SubstringByteArrayOutputStream.toString validates that the requested offset does not exceed the number of bytes actually written to the buffer (count). Throwing early with an informative message prevents the confusing StringIndexOutOfBoundsException that would otherwise come from the underlying buffer copy.

Source

Thrown at sdks/java/core/src/main/java/org/apache/beam/sdk/io/TextSource.java:475

  }

  /**
   * This class is created to avoid multiple bytes-copy when making a substring of the output.
   * Without this class, it requires two bytes copies.
   *
   * <pre>{@code
   * ByteArrayOutputStream out = ...;
   * byte[] buffer = out.toByteArray(); // 1st-copy
   * String s = new String(buffer, offset, length); // 2nd-copy
   * }</pre>
   */
  static class SubstringByteArrayOutputStream extends ByteArrayOutputStream {
    public String toString(int offset, int length, Charset charset) {
      if (offset < 0) {
        throw new IllegalArgumentException("offset is negative: " + offset);
      }
      if (offset > count) {
        throw new IllegalArgumentException(
            "offset exceeds the buffer limit. offset: " + offset + ", limit: " + count);
      }

      if (length < 0) {
        throw new IllegalArgumentException("length is negative: " + length);
      }

      if (offset + length > count) {
        throw new IllegalArgumentException(
            "offset + length exceeds the buffer limit. offset: "
                + offset
                + ", length: "
                + length
                + ", limit: "
                + count);
      }

      return new String(buf, offset, length, charset);

View on GitHub (pinned to 12126d8942)

Solutions

1. Check offset <= outputStream.count (or size()) before slicing; clamp or reject earlier in caller code.
2. Ensure the buffer is not reset/copyFrom'd between computing the offset and calling toString.
3. Recompute offsets from the current buffer state each read rather than caching indices from earlier reads.

Example fix

// before
String record = buffer.toString(offset, len, UTF_8);

// after
if (offset <= buffer.count) {
  String record = buffer.toString(offset, len, UTF_8);
} else {
  throw new IllegalStateException("stale offset " + offset + " vs buffer size " + buffer.count);
}

checkArgument(offset <= buffer.count, "offset %s exceeds buffer limit %s", offset, buffer.count);

try {
  String s = buffer.toString(offset, len, UTF_8);
} catch (IllegalArgumentException e) {
  // recompute offsets from current buffer state and retry once
}

When it happens

Trigger: Calling toString(offset, length, charset) where offset > count, i.e. the requested start position is past the end of bytes written so far — e.g. slicing a record using offsets from a previous buffer state after the stream was reset or truncated.

Common situations: TextSource record-extraction code that reuses one buffer across reads and keeps stale offsets; callers slicing at an offset computed against the original data size after buffer reuse (reset/copyFrom); tests probing boundary conditions at exactly count vs count+1.

Related errors

• offset is negative:
• length is negative:
• offset + length exceeds the buffer limit. offset: , length:
• Failed closing channel to %s
• Error determining if %s allows dynamic splitting

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/aa0c4e9756834522. Report an issue: GitHub.