{"record":{"id":"3520467e056eb7d7","repo":"apache/hadoop","slug":"requested-more-bytes-than-destination-buffer-size-352046","errorCode":null,"errorMessage":"Requested more bytes than destination buffer size: request length = %s, with offset = %s, buffer capacity = %s","messagePattern":"Requested more bytes than destination buffer size: request length = (.+?), with offset = (.+?), buffer capacity = (.+?)","errorType":"exception","errorClass":"IndexOutOfBoundsException","httpStatus":null,"severity":"error","filePath":"hadoop-cloud-storage-project/hadoop-tos/src/main/java/org/apache/hadoop/fs/tosfs/util/FSUtils.java","lineNumber":44,"sourceCode":"import java.net.URI;\n\npublic final class FSUtils {\n  private static final String OVERFLOW_ERROR_HINT =\n      FSExceptionMessages.TOO_MANY_BYTES_FOR_DEST_BUFFER\n          + \": request length = %s, with offset = %s, buffer capacity = %s\";\n\n  private FSUtils() {\n  }\n\n  public static void checkReadParameters(byte[] buffer, int offset, int length) {\n    Preconditions.checkArgument(buffer != null, \"Null buffer\");\n    if (offset < 0 || offset > buffer.length) {\n      throw new IndexOutOfBoundsException(\n          String.format(\"offset: %s is out of range [%s, %s]\", offset, 0, buffer.length));\n    }\n    Preconditions.checkArgument(length >= 0, \"length: %s is negative\", length);\n    if (buffer.length < offset + length) {\n      throw new IndexOutOfBoundsException(\n          String.format(OVERFLOW_ERROR_HINT, length, offset, (buffer.length - offset)));\n    }\n  }\n\n  public static URI normalizeURI(URI fsUri, Configuration hadoopConfig) {\n    final String scheme = fsUri.getScheme();\n    final String authority = fsUri.getAuthority();\n\n    if (scheme == null && authority == null) {\n      fsUri = FileSystem.getDefaultUri(hadoopConfig);\n    } else if (scheme != null && authority == null) {\n      URI defaultUri = FileSystem.getDefaultUri(hadoopConfig);\n      if (scheme.equals(defaultUri.getScheme()) && defaultUri.getAuthority() != null) {\n        fsUri = defaultUri;\n      }\n    }\n    return fsUri;\n  }","sourceCodeStart":26,"sourceCodeEnd":62,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-cloud-storage-project/hadoop-tos/src/main/java/org/apache/hadoop/fs/tosfs/util/FSUtils.java#L26-L62","documentation":"The second guard in FSUtils.checkReadParameters: after offset passes, it rejects reads where buffer.length < offset + length — i.e. the requested bytes cannot fit in the remaining buffer space. The message (TOO_MANY_BYTES_FOR_DEST_BUFFER from FSExceptionMessages) is the Hadoop-standard wording for 'destination buffer too small for this read'. Integer overflow is also implicitly caught because a wrapped offset+length goes negative and compares less than buffer.length only in pathological cases; treat any hit as a sizing bug at the caller.","triggerScenarios":"read(buffer, offset, length) where offset + length overruns the buffer, e.g. buffer of 4096 with offset 4000 and length 256; readFully(buf, off, fileRemaining) where buf was sized to the previous chunk; passing a full-file length while using a chunk-sized buffer.","commonSituations":"Reusing one buffer for chunks but passing the total length; computing length as file size instead of min(fileSize - pos, buffer.length - offset); buffer pools handing back smaller buffers than earlier in the run; copy loops that advance offset but forget to shrink length by the same amount.","solutions":["Clamp the length: int effectiveLen = Math.min(requestedLen, buffer.length - offset);","Allocate the destination buffer to at least offset + length before the read.","Run FSUtils.checkReadParameters(buffer, offset, length) first to fail with a precise, local stack trace.","In copy loops, recompute both offset and length from the same remaining-bytes value each iteration."],"exampleFix":"// before\nbyte[] buf = new byte[4096];\nin.readFully(buf, 4000, remaining); // 4000 + remaining > 4096\n\n// after\nbyte[] buf = new byte[4096];\nin.readFully(buf, 4000, Math.min(remaining, buf.length - 4000));","handlingStrategy":"validation","validationCode":"int effectiveLen = Math.min(length, buffer.length - offset);\nif (effectiveLen < 0) { throw new IllegalArgumentException(\"offset beyond buffer\"); }\nin.read(buffer, offset, effectiveLen);","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Always compute length as min(requested, buffer.length - offset).","Never reuse a buffer sized for a previous chunk with a new larger length.","Run FSUtils.checkReadParameters first for an early, precise failure."],"tags":["read","buffer","index-out-of-bounds","buffer-capacity","input-stream"],"backgroundTag":"buffer-overflow-read","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}