What causes "Invalid inputs are found, all being null" in apache/hadoop?

Before decoding, verify at least numDataUnits inputs are non-null and readable; if not, the group genuinely cannot be recovered by EC alone

apache/hadoop · error · HadoopIllegalArgumentException

Invalid inputs are found, all being null

Error message

Invalid inputs are found, all being null

What it means

HHUtil.findFirstValidInput scans an input array for the first non-null element and is used by Hitchhiker coding steps to infer the working buffer size; if every element is null it throws HadoopIllegalArgumentException("Invalid inputs are found, all being null"). Decoding needs at least one readable input to proceed at all.

Source

Thrown at hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/coder/util/HHUtil.java:217

    return piggybacks;
  }

  /**
   * Find the valid input from all the inputs.
   *
   * @param <T> Generics Type T.
   * @param inputs input buffers to look for valid input
   * @return the first valid input
   */
  public static <T> T findFirstValidInput(T[] inputs) {
    for (T input : inputs) {
      if (input != null) {
        return input;
      }
    }

    throw new HadoopIllegalArgumentException(
            "Invalid inputs are found, all being null");
  }
}

View on GitHub (pinned to 2add963021)

Solutions

Before decoding, verify at least numDataUnits inputs are non-null and readable; if not, the group genuinely cannot be recovered by EC alone
Propagate read failures as errors rather than silently replacing chunks with null
Pass erasure information via erasedIndexes/erasedMasks while keeping surviving units' buffers non-null

Example fix

// before
for (int i = 0; i < inputs.length; i++) {
  if (readFailed[i]) inputs[i] = null; // may null out everything
}
step.performCoding(inputChunks, outputChunks); // throws: all being null

// after
if (countNonNull(inputs) < numDataUnits) {
  throw new IOException("Block group unrecoverable: fewer than "
      + numDataUnits + " readable units");
}

Defensive patterns

Strategy: validation

Validate before calling

int nonNull = 0;
for (ByteBuffer b : inputs) if (b != null) nonNull++;
if (nonNull == 0) {
  throw new IOException("Cannot decode: no readable input units");
}
if (nonNull < decoder.getNumDataUnits()) {
  throw new IOException("Unrecoverable: only " + nonNull + " of "
      + decoder.getNumDataUnits() + " needed units readable");
}

Type guard

boolean hasEnoughValidInputs(Object[] inputs, int required) {
  int n = 0;
  for (Object in : inputs) if (in != null && ++n >= required) return true;
  return false;
}

Try / catch

try {
  step.performCoding(in, out);
} catch (HadoopIllegalArgumentException e) {
  if (e.getMessage().contains("all being null")) {
    // treat as unrecoverable group: report, do not retry decode
  }
}

Prevention

Never silently substitute null for failed chunk reads; fail or retry the read first
Check readable-unit count against numDataUnits before every decode
Surface per-unit read errors so operators see which datanodes failed

When it happens

Trigger: Calling HH decode/encode paths with an inputs array whose elements are all null — typically every unit is missing/erased, or the caller represented upstream read failures as nulls and lost all of them. Also hit when an empty-capability buffer array (all slots null) is passed to CoderUtil/HHUtil.findFirstValidInput via raw coder state construction.

Common situations: More concurrent failures than the EC policy's parity can tolerate; datanode read errors swallowed and turned into null inputs; logic errors that null out inputs before decoding instead of passing erased indexes.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/8f1e8946f1b6b04f. Report an issue: GitHub.