apache/hadoop · error · EOFException

Reaching the limit of the buffer.

Error message

Reaching the limit of the buffer.

What it means

write(int) appends a single byte at currentPointer and throws EOFException once currentPointer has reached the limit — the stream models a fixed-size destination, so writing past its bound is treated as end-of-buffer rather than growth. This is the byte-at-a-time path.

Source

Thrown at hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/BoundedByteArrayOutputStream.java:77

  protected BoundedByteArrayOutputStream(byte[] buf, int offset, int limit) {
    resetBuffer(buf, offset, limit);
  }
  
  protected void resetBuffer(byte[] buf, int offset, int limit) {
    int capacity = buf.length - offset;
    if ((capacity < limit) || (capacity | limit) < 0) {
      throw new IllegalArgumentException("Invalid capacity/limit");
    }
    this.buffer = buf;
    this.startOffset = offset;
    this.currentPointer = offset;
    this.limit = offset + limit;
  }
  
  @Override
  public void write(int b) throws IOException {
    if (currentPointer >= limit) {
      throw new EOFException("Reaching the limit of the buffer.");
    }
    buffer[currentPointer++] = (byte) b;
  }

  @Override
  public void write(byte b[], int off, int len) throws IOException {
    if ((off < 0) || (off > b.length) || (len < 0) || ((off + len) > b.length)
        || ((off + len) < 0)) {
      throw new IndexOutOfBoundsException();
    } else if (len == 0) {
      return;
    }

    if (currentPointer + len > limit) {
      throw new EOFException("Reach the limit of the buffer");
    }

    System.arraycopy(b, off, buffer, currentPointer, len);

View on GitHub (pinned to 2add963021)

Solutions

1. Size the buffer for the maximum expected payload, or track written bytes and compare with getLimit() before each write.
2. Switch to a growable stream (ByteArrayOutputStream / DataOutputBuffer) if the payload size is not bounded.
3. Catch EOFException at the record level and retry with a doubled buffer size (bounded growth policy).

Example fix

// before
for (int b : bytes) out.write(b); // EOFException when full

// after
if (out.size() + bytes.length > out.getLimit()) {
  throw new BufferTooSmallException("need " + bytes.length + ", have " + (out.getLimit() - out.size()));
}
for (int b : bytes) out.write(b);

if (out.size() >= out.getLimit()) {
  throw new IllegalStateException("No space left; limit=" + out.getLimit());
}
out.write(b);

try {
  out.write(b);
} catch (EOFException e) {
  // record-level handling: grow destination and retry, or fail this record
}

When it happens

Trigger: Writing more single bytes than the configured limit, e.g. serializing into a buffer sized from a too-small estimate; calling write(b) in a loop over unbounded input without checking remaining space.

Common situations: Sizing RPC/shuffle scratch buffers from a heuristic record-size estimate while actual records are larger; upgrading data volume without resizing bounded buffers.

Related errors

• Reach the limit of the buffer
• Cannot seek after EOF
• Attempted to seek or read past the end of the file
• Requested more bytes than destination buffer size: request l
• write (b[{b.length}], {off}, {len})

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/108fe4db3a290ec3. Report an issue: GitHub.