What causes "Invalid buffer, not of length {}" in apache/hadoop?

Prepare every input buffer to the same remaining() length: rewind/clear, set position, and pad short data to the stripe length

apache/hadoop · error · HadoopIllegalArgumentException

Invalid buffer, not of length {}

Error message

Invalid buffer, not of length {}

What it means

For ByteBuffer decode, ByteBufferDecodingState sets decodeLength from the first non-null input's remaining() and requires every non-null input buffer to have exactly that many remaining bytes (mismatch -> HadoopIllegalArgumentException("Invalid buffer, not of length <decodeLength>")). A following check also enforces uniform direct/heap type, so all decode inputs must be same-length, same-type buffers with consistent positions.

Source

Thrown at hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/rawcoder/ByteBufferDecodingState.java:107

        inputOffsets, newOutputs, outputOffsets);
    return baeState;
  }

  /**
   * Check and ensure the buffers are of the desired length and type, direct
   * buffers or not.
   * @param buffers the buffers to check
   */
  void checkInputBuffers(ByteBuffer[] buffers) {
    int validInputs = 0;

    for (ByteBuffer buffer : buffers) {
      if (buffer == null) {
        continue;
      }

      if (buffer.remaining() != decodeLength) {
        throw new HadoopIllegalArgumentException(
            "Invalid buffer, not of length " + decodeLength);
      }
      if (buffer.isDirect() != usingDirectBuffer) {
        throw new HadoopIllegalArgumentException(
            "Invalid buffer, isDirect should be " + usingDirectBuffer);
      }

      validInputs++;
    }

    if (validInputs < decoder.getNumDataUnits()) {
      throw new HadoopIllegalArgumentException(
          "No enough valid inputs are provided, not recoverable");
    }
  }

  /**
   * Check and ensure the buffers are of the desired length and type, direct

View on GitHub (pinned to 2add963021)

Solutions

Prepare every input buffer to the same remaining() length: rewind/clear, set position, and pad short data to the stripe length
Use one allocation style (all ByteBuffer.allocateDirect or all allocate) per decode call
Pre-check remaining() and isDirect() of all non-null inputs before invoking decode

Example fix

// before
ByteBuffer[] inputs = { buf0, buf1 }; // buf1.remaining() != buf0.remaining()
decoder.decode(inputs, erased, outputs); // throws

// after
int len = inputs[0].remaining();
for (ByteBuffer b : inputs) {
  if (b != null) b.limit(b.position() + len); // normalize remaining
}

Defensive patterns

Strategy: validation

Validate before calling

ByteBuffer first = CoderUtil.findFirstValidInput(inputs);
int len = first.remaining();
boolean direct = first.isDirect();
for (ByteBuffer b : inputs) {
  if (b != null && (b.remaining() != len || b.isDirect() != direct)) {
    throw new IllegalStateException(
        "All decode inputs need remaining()==" + len + " and isDirect()==" + direct);
  }
}
decoder.decode(inputs, erased, outputs);

Type guard

boolean uniformInputs(ByteBuffer[] inputs) {
  ByteBuffer first = null;
  for (ByteBuffer b : inputs) if (b != null) { first = b; break; }
  if (first == null) return false;
  for (ByteBuffer b : inputs) {
    if (b != null && (b.remaining() != first.remaining()
        || b.isDirect() != first.isDirect())) return false;
  }
  return true;
}

Try / catch

try {
  decoder.decode(inputs, erased, outputs);
} catch (HadoopIllegalArgumentException e) {
  if (e.getMessage().contains("not of length")
      || e.getMessage().contains("isDirect")) {
    normalizeBuffers(); // set consistent limit/position, one allocation type, retry
  }
}

Prevention

Reset buffer positions/limits before each decode; don't reuse buffers with stale state
Pick one buffer style (direct vs heap) per code path and stick to it
Pad short final stripes so every buffer's remaining() matches the stripe length

When it happens

Trigger: Calling RawErasureDecoder.decode(ByteBuffer[], int[], ByteBuffer[]) with buffers whose remaining() differs — e.g., one buffer sliced or read into further than others, a short final chunk not padded, or mixed allocations from direct and heap pools (which additionally trips the isDirect check).

Common situations: Reusing buffers that retain positions/limits from previous operations; striped reads with unpadded final chunks; half-migrated code switching from heap to direct buffers so some inputs differ in type and prepared length.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/1cf0d76308f65391. Report an issue: GitHub.