apache/hadoop · error · HadoopIllegalArgumentException

Invalid numDataUnits and numParityUnits

Error message

Invalid numDataUnits and numParityUnits

What it means

Error "Invalid numDataUnits and numParityUnits" thrown in apache/hadoop.

Source

Thrown at hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/rawcoder/RSRawEncoder.java:46

 * isn't available in some environment. Please always use native implementations
 * when possible. This new Java coder is about 5X faster than the one originated
 * from HDFS-RAID, and also compatible with the native/ISA-L coder.
 */
@InterfaceAudience.Private
public class RSRawEncoder extends RawErasureEncoder {
  // relevant to schema and won't change during encode calls.
  private byte[] encodeMatrix;
  /**
   * Array of input tables generated from coding coefficients previously.
   * Must be of size 32*k*rows
   */
  private byte[] gfTables;

  public RSRawEncoder(ErasureCoderOptions coderOptions) {
    super(coderOptions);

    if (getNumAllUnits() >= RSUtil.GF.getFieldSize()) {
      throw new HadoopIllegalArgumentException(
          "Invalid numDataUnits and numParityUnits");
    }

    encodeMatrix = new byte[getNumAllUnits() * getNumDataUnits()];
    RSUtil.genCauchyMatrix(encodeMatrix, getNumAllUnits(), getNumDataUnits());
    if (allowVerboseDump()) {
      DumpUtil.dumpMatrix(encodeMatrix, getNumDataUnits(), getNumAllUnits());
    }
    gfTables = new byte[getNumAllUnits() * getNumDataUnits() * 32];
    RSUtil.initTables(getNumDataUnits(), getNumParityUnits(), encodeMatrix,
        getNumDataUnits() * getNumDataUnits(), gfTables);
    if (allowVerboseDump()) {
      System.out.println(DumpUtil.bytesToHex(gfTables, -1));
    }
  }

  @Override
  protected void doEncode(ByteBufferEncodingState encodingState) {

View on GitHub (pinned to 2add963021)

Solutions

1. Correct the numDataUnits/numParityUnits in the EC schema to values supported by the RS encoder.

When it happens

Trigger: Thrown at hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/rawcoder/RSRawEncoder.java:46 when the library encounters an invalid state.

Common situations: Occurs when RSRawEncoder is created with invalid numDataUnits/numParityUnits values. Configure valid RS codec unit counts before creating the encoder.

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/130c6d6867d26e02. Report an issue: GitHub.