apache/hadoop · error · HadoopIllegalArgumentException

Invalid buffer found, not allowing null

Error message

Invalid buffer found, not allowing null

What it means

For byte-array encoding, ByteArrayEncodingState.checkBuffers is applied to both the input and output arrays and rejects any null element with HadoopIllegalArgumentException("Invalid buffer found, not allowing null"). Encoding has no erased-unit concept: every data input and every parity output must be a real allocated array.

Source

Thrown at hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/rawcoder/ByteArrayEncodingState.java:94

    }

    for (int i = 0; i < outputs.length; i++) {
      newOutputs[i] = ByteBuffer.allocateDirect(encodeLength);
    }

    ByteBufferEncodingState bbeState = new ByteBufferEncodingState(encoder,
        encodeLength, newInputs, newOutputs);
    return bbeState;
  }

  /**
   * Check and ensure the buffers are of the desired length.
   * @param buffers the buffers to check
   */
  void checkBuffers(byte[][] buffers) {
    for (byte[] buffer : buffers) {
      if (buffer == null) {
        throw new HadoopIllegalArgumentException(
            "Invalid buffer found, not allowing null");
      }

      if (buffer.length != encodeLength) {
        throw new HadoopIllegalArgumentException(
            "Invalid buffer not of length " + encodeLength);
      }
    }
  }
}

View on GitHub (pinned to 2add963021)

Solutions

1. Ensure every input and output byte[] is allocated before encode; there is no legal null in the encode path
2. If a data chunk is unavailable, fail the encode explicitly rather than passing null
3. Add a null scan over both arrays right before calling encode

Example fix

// before
byte[][] inputs = new byte[numDataUnits][]; // some slots still null
encoder.encode(inputs, outputs); // throws

// after
for (int i = 0; i < inputs.length; i++) {
  if (inputs[i] == null) throw new IllegalStateException("missing input " + i);
}
encoder.encode(inputs, outputs);

for (byte[] b : inputs) {
  if (b == null) throw new IllegalStateException("encode input null");
}
for (byte[] b : outputs) {
  if (b == null) throw new IllegalStateException("encode output null");
}
encoder.encode(inputs, outputs);

try {
  encoder.encode(inputs, outputs);
} catch (HadoopIllegalArgumentException e) {
  if (e.getMessage().contains("not allowing null")) {
    // fill the null slot with a real buffer (encode allows no nulls) and retry
  }
}

When it happens

Trigger: Calling RawErasureEncoder.encode(byte[][], byte[][]) with a null element in either array — e.g., an input slot left null because its read failed, or outputs allocated as an array of nulls.

Common situations: Reusing decode-style null placeholders in encode calls; buffer pools returning null under pressure; partially filled arrays passed before all chunks were fetched.

Related errors

• Invalid buffer found, not allowing null
• Invalid inputs are found, all being null
• Invalid buffer, not of length {}
• Invalid buffer not of length {}
• Invalid buffer not of length {}

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/9db1d042a39d1479. Report an issue: GitHub.