What causes "Invalid buffer not of length {}" in apache/hadoop?

Allocate each output as new byte[inputLen] where inputLen is the common non-null input length

apache/hadoop · error · HadoopIllegalArgumentException

Invalid buffer not of length {}

Error message

Invalid buffer not of length {}

What it means

During byte-array decode setup, every output buffer must match the decode length derived from the first valid input (buffer.length != decodeLength -> HadoopIllegalArgumentException). Output arrays are where recovered data is written, so each must be exactly the stripe length the decoder works with.

Source

Thrown at hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/rawcoder/ByteArrayDecodingState.java:129

    if (validInputs < decoder.getNumDataUnits()) {
      throw new HadoopIllegalArgumentException(
          "No enough valid inputs are provided, not recoverable");
    }
  }

  /**
   * Check and ensure the buffers are of the desired length.
   * @param buffers the buffers to check
   */
  void checkOutputBuffers(byte[][] buffers) {
    for (byte[] buffer : buffers) {
      if (buffer == null) {
        throw new HadoopIllegalArgumentException(
            "Invalid buffer found, not allowing null");
      }

      if (buffer.length != decodeLength) {
        throw new HadoopIllegalArgumentException(
            "Invalid buffer not of length " + decodeLength);
      }
    }
  }
}

View on GitHub (pinned to 2add963021)

Solutions

Allocate each output as new byte[inputLen] where inputLen is the common non-null input length
Recompute output allocation whenever the schema or cell size changes instead of caching sizes
Pre-validate outputs against the first valid input's length before calling decode

Example fix

// before
byte[][] outputs = { new byte[oldCellSize] }; // != decodeLength
decoder.decode(inputs, erased, outputs); // throws

// after
int len = CoderUtil.findFirstValidInput(inputs).length;
byte[][] outputs = new byte[erasedIndexes.length][len];

Defensive patterns

Strategy: validation

Validate before calling

int len = CoderUtil.findFirstValidInput(inputs).length;
for (byte[] out : outputs) {
  if (out == null || out.length != len) {
    throw new IllegalStateException("Each output must be byte[" + len + "]");
  }
}
decoder.decode(inputs, erased, outputs);

Try / catch

try {
  decoder.decode(inputs, erased, outputs);
} catch (HadoopIllegalArgumentException e) {
  if (e.getMessage().startsWith("Invalid buffer not of length")) {
    outputs = allocateOutputs(len); decoder.decode(inputs, erased, outputs);
  }
}

Prevention

Derive output size from the actual inputs at call time, never from cached config constants
Reallocate decode buffers whenever EC policy cell/stripe size changes
Pre-validate output lengths together with null checks in one guard

When it happens

Trigger: Calling decode with output byte[] arrays allocated at a different size than the inputs — e.g., outputs sized from an older cellSize after a policy change, or allocated before the input length was known.

Common situations: Policy changes (cell size / stripe size) applied to inputs but not output allocation; outputs allocated with capacity instead of exact length; mixed buffer caches keyed by wrong size.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/b5cd7fbd368a79be. Report an issue: GitHub.