{"record":{"id":"9455e190af440bb6","repo":"apache/hadoop","slug":"reach-the-limit-of-the-buffer","errorCode":null,"errorMessage":"Reach the limit of the buffer","messagePattern":"Reach the limit of the buffer","errorType":"exception","errorClass":"EOFException","httpStatus":null,"severity":"error","filePath":"hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/BoundedByteArrayOutputStream.java","lineNumber":92,"sourceCode":"  @Override\n  public void write(int b) throws IOException {\n    if (currentPointer >= limit) {\n      throw new EOFException(\"Reaching the limit of the buffer.\");\n    }\n    buffer[currentPointer++] = (byte) b;\n  }\n\n  @Override\n  public void write(byte b[], int off, int len) throws IOException {\n    if ((off < 0) || (off > b.length) || (len < 0) || ((off + len) > b.length)\n        || ((off + len) < 0)) {\n      throw new IndexOutOfBoundsException();\n    } else if (len == 0) {\n      return;\n    }\n\n    if (currentPointer + len > limit) {\n      throw new EOFException(\"Reach the limit of the buffer\");\n    }\n\n    System.arraycopy(b, off, buffer, currentPointer, len);\n    currentPointer += len;\n  }\n\n  /**\n   * Reset the limit \n   * @param newlim New Limit\n   */\n  public void reset(int newlim) {\n    if (newlim > (buffer.length - startOffset)) {\n      throw new IndexOutOfBoundsException(\"Limit exceeds buffer size\");\n    }\n    this.limit = newlim;\n    this.currentPointer = startOffset;\n  }\n","sourceCodeStart":74,"sourceCodeEnd":110,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/BoundedByteArrayOutputStream.java#L74-L110","documentation":"The bulk write write(byte[], off, len) throws EOFException when currentPointer + len would exceed the limit — the byte range cannot fit in the remaining bounded space. Index checks on the source array pass first; the failure is purely about insufficient destination capacity.","triggerScenarios":"Calling write(chunk, 0, chunk.length) where chunk.length > limit - currentPointer; repeated accumulation into the bounded stream without resetting; a serializer that estimates its output smaller than it actually writes.","commonSituations":"Shuffle/spill code and MapOutputBuffer paths where a record must fit into a soft buffer; compressors writing blocks into fixed byte[] destinations; unit tests with hand-sized buffers smaller than test payloads.","solutions":["Check available space first: if (out.getLimit() - out.size() < len) grow/reset the destination, then retry.","Reset (reset() or reset(newlim)) between records when reusing one buffer per record.","Size the buffer from the real maximum record size plus headroom rather than an average."],"exampleFix":"// before\nbbos.write(recordBytes, 0, recordBytes.length); // throws if it doesn't fit\n\n// after\nif (bbos.getLimit() - bbos.size() < recordBytes.length) {\n  bbos.reset(Math.min(buffer.length, recordBytes.length * 2)); // or allocate larger\n}\nbbos.write(recordBytes, 0, recordBytes.length);","handlingStrategy":"validation","validationCode":"int remaining = out.getLimit() - out.size();\nif (len > remaining) {\n  out.reset(Math.min(buffer.length - startOffset, Math.max(len, remaining * 2)));\n}\nout.write(b, off, len);","typeGuard":null,"tryCatchPattern":"try {\n  out.write(b, off, len);\n} catch (EOFException e) {\n  // does not fit: spill/flush current buffer, then retry the write\n}","preventionTips":["Compare payload size to getLimit() - size() before bulk writes.","Reset bounded buffers between records when reusing them."],"tags":["hadoop","io","buffer","buffer-overflow","eof"],"backgroundTag":"buffer-limit-exceeded","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-23T01:17:44.959Z"}