apache/hadoop · error · HadoopIllegalArgumentException

Invalid buffer not of length {}

Error message

Invalid buffer not of length {}

What it means

ByteArrayEncodingState derives encodeLength from the first valid input's array length and requires every input and output byte[] in the encode call to be exactly that length (buffer.length != encodeLength -> HadoopIllegalArgumentException). All units of a stripe must be identically sized for the raw encoder to process them.

Source

Thrown at hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/rawcoder/ByteArrayEncodingState.java:99

    ByteBufferEncodingState bbeState = new ByteBufferEncodingState(encoder,
        encodeLength, newInputs, newOutputs);
    return bbeState;
  }

  /**
   * Check and ensure the buffers are of the desired length.
   * @param buffers the buffers to check
   */
  void checkBuffers(byte[][] buffers) {
    for (byte[] buffer : buffers) {
      if (buffer == null) {
        throw new HadoopIllegalArgumentException(
            "Invalid buffer found, not allowing null");
      }

      if (buffer.length != encodeLength) {
        throw new HadoopIllegalArgumentException(
            "Invalid buffer not of length " + encodeLength);
      }
    }
  }
}

View on GitHub (pinned to 2add963021)

Solutions

1. Zero-pad the final chunk so all inputs share one length
2. Allocate all outputs with the same length as the inputs (first input's length)
3. Validate lengths of both arrays in one pre-encode check

Example fix

// before
byte[][] inputs = { new byte[cellSize], Arrays.copyOf(data, n) }; // n < cellSize
encoder.encode(inputs, outputs); // throws

// after
byte[][] inputs = new byte[numDataUnits][cellSize];
System.arraycopy(data, 0, inputs[1], 0, n); // padded to cellSize

int len = CoderUtil.findFirstValidInput(inputs).length;
for (byte[][] arr : new byte[][][]{inputs, outputs}) {
  for (byte[] b : arr) {
    if (b != null && b.length != len) {
      throw new IllegalStateException("All encode buffers must be byte[" + len + "]");
    }
  }
}
encoder.encode(inputs, outputs);

try {
  encoder.encode(inputs, outputs);
} catch (HadoopIllegalArgumentException e) {
  if (e.getMessage().startsWith("Invalid buffer not of length")) padAndRetry();
}

When it happens

Trigger: Calling encode(byte[][], byte[][]) where any input or output array differs in length from the first input — short unpadded final chunks, or outputs allocated with a stale size.

Common situations: Last-stripe data not zero-padded to the stripe length; cell/stripe size changed by policy but buffer allocation code not updated; outputs allocated from a different constant than inputs.

Related errors

• Invalid buffer, not of length {}
• Invalid buffer not of length {}
• Invalid buffer, not of length {}
• Codec not configured for custom codec {}
• No schema options are provided

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/3977226b9308ff19. Report an issue: GitHub.