{"record":{"id":"108fe4db3a290ec3","repo":"apache/hadoop","slug":"reaching-the-limit-of-the-buffer","errorCode":null,"errorMessage":"Reaching the limit of the buffer.","messagePattern":"Reaching the limit of the buffer\\.","errorType":"exception","errorClass":"EOFException","httpStatus":null,"severity":"error","filePath":"hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/BoundedByteArrayOutputStream.java","lineNumber":77,"sourceCode":"  protected BoundedByteArrayOutputStream(byte[] buf, int offset, int limit) {\n    resetBuffer(buf, offset, limit);\n  }\n  \n  protected void resetBuffer(byte[] buf, int offset, int limit) {\n    int capacity = buf.length - offset;\n    if ((capacity < limit) || (capacity | limit) < 0) {\n      throw new IllegalArgumentException(\"Invalid capacity/limit\");\n    }\n    this.buffer = buf;\n    this.startOffset = offset;\n    this.currentPointer = offset;\n    this.limit = offset + limit;\n  }\n  \n  @Override\n  public void write(int b) throws IOException {\n    if (currentPointer >= limit) {\n      throw new EOFException(\"Reaching the limit of the buffer.\");\n    }\n    buffer[currentPointer++] = (byte) b;\n  }\n\n  @Override\n  public void write(byte b[], int off, int len) throws IOException {\n    if ((off < 0) || (off > b.length) || (len < 0) || ((off + len) > b.length)\n        || ((off + len) < 0)) {\n      throw new IndexOutOfBoundsException();\n    } else if (len == 0) {\n      return;\n    }\n\n    if (currentPointer + len > limit) {\n      throw new EOFException(\"Reach the limit of the buffer\");\n    }\n\n    System.arraycopy(b, off, buffer, currentPointer, len);","sourceCodeStart":59,"sourceCodeEnd":95,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/BoundedByteArrayOutputStream.java#L59-L95","documentation":"write(int) appends a single byte at currentPointer and throws EOFException once currentPointer has reached the limit — the stream models a fixed-size destination, so writing past its bound is treated as end-of-buffer rather than growth. This is the byte-at-a-time path.","triggerScenarios":"Writing more single bytes than the configured limit, e.g. serializing into a buffer sized from a too-small estimate; calling write(b) in a loop over unbounded input without checking remaining space.","commonSituations":"Sizing RPC/shuffle scratch buffers from a heuristic record-size estimate while actual records are larger; upgrading data volume without resizing bounded buffers.","solutions":["Size the buffer for the maximum expected payload, or track written bytes and compare with getLimit() before each write.","Switch to a growable stream (ByteArrayOutputStream / DataOutputBuffer) if the payload size is not bounded.","Catch EOFException at the record level and retry with a doubled buffer size (bounded growth policy)."],"exampleFix":"// before\nfor (int b : bytes) out.write(b); // EOFException when full\n\n// after\nif (out.size() + bytes.length > out.getLimit()) {\n  throw new BufferTooSmallException(\"need \" + bytes.length + \", have \" + (out.getLimit() - out.size()));\n}\nfor (int b : bytes) out.write(b);","handlingStrategy":"validation","validationCode":"if (out.size() >= out.getLimit()) {\n  throw new IllegalStateException(\"No space left; limit=\" + out.getLimit());\n}\nout.write(b);","typeGuard":null,"tryCatchPattern":"try {\n  out.write(b);\n} catch (EOFException e) {\n  // record-level handling: grow destination and retry, or fail this record\n}","preventionTips":["Track bytes written versus getLimit() when sizes are unpredictable.","Use DataOutputBuffer for unbounded payloads; bounded streams only for fixed-size destinations."],"tags":["hadoop","io","buffer","buffer-overflow","eof"],"backgroundTag":"buffer-limit-exceeded","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-23T01:17:44.959Z"}