What causes "Invalid buffer, not of length {}" in apache/hadoop?

Pad every input to the same stripe length (zero-pad the short final chunk, as HDFS does for the last stripe)

apache/hadoop · error · HadoopIllegalArgumentException

Invalid buffer, not of length {}

Error message

Invalid buffer, not of length {}

What it means

When a byte-array raw decode starts, ByteArrayDecodingState sets decodeLength from the first non-null input's array length and then requires every non-null input byte[] to have exactly that length. Any input longer or shorter than the first valid input throws HadoopIllegalArgumentException("Invalid buffer, not of length <decodeLength>"). Inputs represent same-sized stripes of different units, so mixed lengths are a caller bug.

Source

Thrown at hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/rawcoder/ByteArrayDecodingState.java:104

    ByteBufferDecodingState bbdState = new ByteBufferDecodingState(decoder,
        decodeLength, erasedIndexes, newInputs, newOutputs);
    return bbdState;
  }

  /**
   * Check and ensure the buffers are of the desired length.
   * @param buffers the buffers to check
   */
  void checkInputBuffers(byte[][] buffers) {
    int validInputs = 0;

    for (byte[] buffer : buffers) {
      if (buffer == null) {
        continue;
      }

      if (buffer.length != decodeLength) {
        throw new HadoopIllegalArgumentException(
            "Invalid buffer, not of length " + decodeLength);
      }

      validInputs++;
    }

    if (validInputs < decoder.getNumDataUnits()) {
      throw new HadoopIllegalArgumentException(
          "No enough valid inputs are provided, not recoverable");
    }
  }

  /**
   * Check and ensure the buffers are of the desired length.
   * @param buffers the buffers to check
   */
  void checkOutputBuffers(byte[][] buffers) {
    for (byte[] buffer : buffers) {

View on GitHub (pinned to 2add963021)

Solutions

Pad every input to the same stripe length (zero-pad the short final chunk, as HDFS does for the last stripe)
Allocate all input buffers with identical length before filling them
Pre-check that all non-null inputs share one length before invoking decode

Example fix

// before
byte[][] inputs = { new byte[cellSize], Arrays.copyOf(chunk, chunkLen) };
decoder.decode(inputs, erased, outputs); // short second input -> throws

// after
byte[][] inputs = new byte[numTotalUnits][cellSize];
System.arraycopy(chunk, 0, inputs[1], 0, chunkLen); // zero-padded remainder

Defensive patterns

Strategy: validation

Validate before calling

byte[] first = CoderUtil.findFirstValidInput(inputs);
for (byte[] b : inputs) {
  if (b != null && b.length != first.length) {
    throw new IllegalArgumentException("Input length " + b.length
        + " != " + first.length + "; pad all inputs to one stripe length");
  }
}
decoder.decode(inputs, erased, outputs);

Try / catch

try {
  decoder.decode(inputs, erased, outputs);
} catch (HadoopIllegalArgumentException e) {
  if (e.getMessage().startsWith("Invalid buffer, not of length")) {
    padAndRetry(); // normalize all inputs to one length, then decode again
  }
}

Prevention

Always zero-pad the final short chunk of a stripe to the full stripe length
Allocate all input arrays with one common size constant
Assert uniform input lengths in tests for striped-read code

When it happens

Trigger: Calling RawErasureDecoder.decode(byte[][], int[], byte[][]) with input arrays of differing lengths — e.g., one unit's read returned a short final chunk while others were padded, or buffers were allocated from different sizes.

Common situations: Striped reads where the last block-group chunk is not zero-padded to the cell/stripe length; heterogeneous buffer allocation; manually assembled inputs from cached chunks of different vintages.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/982b817aded674be. Report an issue: GitHub.