apache/hadoop · error · EOFException

Reach the limit of the buffer

Error message

Reach the limit of the buffer

What it means

The bulk write write(byte[], off, len) throws EOFException when currentPointer + len would exceed the limit — the byte range cannot fit in the remaining bounded space. Index checks on the source array pass first; the failure is purely about insufficient destination capacity.

Source

Thrown at hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/BoundedByteArrayOutputStream.java:92

  @Override
  public void write(int b) throws IOException {
    if (currentPointer >= limit) {
      throw new EOFException("Reaching the limit of the buffer.");
    }
    buffer[currentPointer++] = (byte) b;
  }

  @Override
  public void write(byte b[], int off, int len) throws IOException {
    if ((off < 0) || (off > b.length) || (len < 0) || ((off + len) > b.length)
        || ((off + len) < 0)) {
      throw new IndexOutOfBoundsException();
    } else if (len == 0) {
      return;
    }

    if (currentPointer + len > limit) {
      throw new EOFException("Reach the limit of the buffer");
    }

    System.arraycopy(b, off, buffer, currentPointer, len);
    currentPointer += len;
  }

  /**
   * Reset the limit 
   * @param newlim New Limit
   */
  public void reset(int newlim) {
    if (newlim > (buffer.length - startOffset)) {
      throw new IndexOutOfBoundsException("Limit exceeds buffer size");
    }
    this.limit = newlim;
    this.currentPointer = startOffset;
  }

View on GitHub (pinned to 2add963021)

Solutions

1. Check available space first: if (out.getLimit() - out.size() < len) grow/reset the destination, then retry.
2. Reset (reset() or reset(newlim)) between records when reusing one buffer per record.
3. Size the buffer from the real maximum record size plus headroom rather than an average.

Example fix

// before
bbos.write(recordBytes, 0, recordBytes.length); // throws if it doesn't fit

// after
if (bbos.getLimit() - bbos.size() < recordBytes.length) {
  bbos.reset(Math.min(buffer.length, recordBytes.length * 2)); // or allocate larger
}
bbos.write(recordBytes, 0, recordBytes.length);

int remaining = out.getLimit() - out.size();
if (len > remaining) {
  out.reset(Math.min(buffer.length - startOffset, Math.max(len, remaining * 2)));
}
out.write(b, off, len);

try {
  out.write(b, off, len);
} catch (EOFException e) {
  // does not fit: spill/flush current buffer, then retry the write
}

When it happens

Trigger: Calling write(chunk, 0, chunk.length) where chunk.length > limit - currentPointer; repeated accumulation into the bounded stream without resetting; a serializer that estimates its output smaller than it actually writes.

Common situations: Shuffle/spill code and MapOutputBuffer paths where a record must fit into a soft buffer; compressors writing blocks into fixed byte[] destinations; unit tests with hand-sized buffers smaller than test payloads.

Related errors

• Reaching the limit of the buffer.
• Cannot seek after EOF
• Attempted to seek or read past the end of the file
• Requested more bytes than destination buffer size: request l
• write (b[{b.length}], {off}, {len})

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/9455e190af440bb6. Report an issue: GitHub.