What causes "Buffer size of %d is larger than requested size of %d" in apache/iceberg?

Compute the target length from the column's actual precision (ceil((precision+1)/... bits/8)) rather than a fixed size.

apache/iceberg · error · IllegalArgumentException

Buffer size of %d is larger than requested size of %d

Error message

Buffer size of %d is larger than requested size of %d

What it means

padBigEndianBytes pads a decimal's big-endian byte buffer to a requested length; if the source buffer is already longer than the requested new length, padding is impossible and it throws IllegalArgumentException. This guards against calling with a target size smaller than the input.

Source

Thrown at arrow/src/main/java/org/apache/iceberg/arrow/vectorized/parquet/DecimalVectorUtil.java:64

  @VisibleForTesting
  static byte[] padBigEndianBytes(byte[] bigEndianBytes, int newLength) {
    if (bigEndianBytes.length == newLength) {
      return bigEndianBytes;
    } else if (bigEndianBytes.length < newLength) {
      byte[] result = new byte[newLength];
      if (bigEndianBytes.length == 0) {
        return result;
      }

      int start = newLength - bigEndianBytes.length;
      if (bigEndianBytes[0] < 0) {
        Arrays.fill(result, 0, start, (byte) 0xFF);
      }
      System.arraycopy(bigEndianBytes, 0, result, start, bigEndianBytes.length);

      return result;
    }
    throw new IllegalArgumentException(
        String.format(
            Locale.ROOT,
            "Buffer size of %d is larger than requested size of %d",
            bigEndianBytes.length,
            newLength));
  }
}

View on GitHub (pinned to 86d9c8fc54)

Solutions

Compute the target length from the column's actual precision (ceil((precision+1)/... bits/8)) rather than a fixed size.
Pass newLength >= bigEndianBytes.length.
Verify the Parquet decimal precision against the expected Arrow type; fix the schema mapping.

Example fix

// before
byte[] padded = DecimalVectorUtil.padBigEndianBytes(bytes, 8); // bytes.length == 16
// after
int target = Math.max(8, bytes.length);
byte[] padded = DecimalVectorUtil.padBigEndianBytes(bytes, target);

Defensive patterns

Strategy: validation

Validate before calling

if (bytes.length > targetLength) {
  throw new IllegalArgumentException("buffer " + bytes.length + " exceeds target " + targetLength);
}
byte[] padded = DecimalVectorUtil.padBigEndianBytes(bytes, targetLength);

Try / catch

// catch (IllegalArgumentException e) {
//   if (e.getMessage().contains("larger than requested size")) { recomputeTargetWidth(); }
//   else throw e;
// }

Prevention

Derive buffer width from decimal precision, never hardcode.
Assert newLength >= bytes.length before calling.
Test decimal mapping for high-precision columns.

When it happens

Trigger: Calling DecimalVectorUtil.padBigEndianBytes(bytes, newLength) where bytes.length > newLength.

Common situations: Reading DECIMAL columns whose precision implies more bytes than the target Arrow/Parquet buffer width, typically a bug in width computation for high-precision decimals.

Understand the failure class

Background: "value must be between 0 and 1" / "out of range" / "must not be negative" errors: fixing range-validation failures across open-source libraries — this error's family across 42 libraries.

Related errors

AI-assisted analysis of apache/iceberg@86d9c8fc54 (2026-09-12). Data as JSON: /api/errors/767895286b1c7926. Report an issue: GitHub.