apache/hadoop · critical · HadoopIllegalArgumentException

No enough valid inputs are provided, not recoverable

Error message

No enough valid inputs are provided, not recoverable

What it means

After validating lengths, ByteArrayDecodingState counts non-null inputs and requires at least decoder.getNumDataUnits() of them. Fewer valid inputs than data units means more erasures than the MDS erasure code can repair, so Hadoop throws HadoopIllegalArgumentException("No enough valid inputs are provided, not recoverable") instead of producing garbage.

Source

Thrown at hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/rawcoder/ByteArrayDecodingState.java:112

   */
  void checkInputBuffers(byte[][] buffers) {
    int validInputs = 0;

    for (byte[] buffer : buffers) {
      if (buffer == null) {
        continue;
      }

      if (buffer.length != decodeLength) {
        throw new HadoopIllegalArgumentException(
            "Invalid buffer, not of length " + decodeLength);
      }

      validInputs++;
    }

    if (validInputs < decoder.getNumDataUnits()) {
      throw new HadoopIllegalArgumentException(
          "No enough valid inputs are provided, not recoverable");
    }
  }

  /**
   * Check and ensure the buffers are of the desired length.
   * @param buffers the buffers to check
   */
  void checkOutputBuffers(byte[][] buffers) {
    for (byte[] buffer : buffers) {
      if (buffer == null) {
        throw new HadoopIllegalArgumentException(
            "Invalid buffer found, not allowing null");
      }

      if (buffer.length != decodeLength) {
        throw new HadoopIllegalArgumentException(
            "Invalid buffer not of length " + decodeLength);

View on GitHub (pinned to 2add963021)

Solutions

Supply at least numDataUnits non-null, correct-length input buffers
Check that the ErasureCoderOptions (schema) unit counts match the actual block group layout before decoding
If failures genuinely exceed parity, treat the group as unrecoverable and fall back to replica/backup recovery — no API call can repair it

Example fix

// before
byte[][] inputs = new byte[numTotalUnits][]; // 4 nulls with 6+3 policy
decoder.decode(inputs, erased, outputs); // throws: not recoverable

// before decode: count readable units
long valid = Arrays.stream(inputs).filter(Objects::nonNull).count();
if (valid < decoder.getNumDataUnits()) {
  throw new IOException("Unrecoverable: only " + valid + " valid units");
}

Defensive patterns

Strategy: validation

Validate before calling

int valid = 0;
for (byte[] b : inputs) if (b != null) valid++;
if (valid < decoder.getNumDataUnits()) {
  throw new IOException("Cannot recover: " + valid + " valid units < "
      + decoder.getNumDataUnits() + " data units; parity exhausted");
}
decoder.decode(inputs, erased, outputs);

Type guard

boolean isRecoverable(Object[] inputs, int numDataUnits) {
  int n = 0;
  for (Object in : inputs) if (in != null && ++n >= numDataUnits) return true;
  return false;
}

Try / catch

try {
  decoder.decode(inputs, erased, outputs);
} catch (HadoopIllegalArgumentException e) {
  if (e.getMessage().contains("not recoverable")) {
    // permanently failed group: alert, stop retrying, use backup/replica path
  }
}

Prevention

Before decoding, compare live units against numDataUnits of the policy schema
Don't retry unrecoverable decodes — more erasures than parity is mathematically hopeless
Monitor datanode failure counts against each EC policy's parity limit

When it happens

Trigger: Calling decode with more than numParityUnits null entries (all others valid), or with an ErasureCoderOptions whose numDataUnits is larger than the number of readable units actually supplied.

Common situations: Multiple concurrent datanode failures beyond policy parity; nulling out failed reads; building decoder options from the wrong schema so unit counts don't match the available inputs.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/6f33d636f9a4f5f3. Report an issue: GitHub.