What causes "key can not be null" in apache/hadoop?

Strip nulls from the input before the loop: keys.removeIf(Objects::isNull)

apache/hadoop · error · NullPointerException

key can not be null

Error message

key can not be null

What it means

RetouchedBloomFilter.add(Key) both sets bits and appends the key to the per-bit keyVector that later retouching (selectiveClearing) walks. It throws NullPointerException for a null key before hashing because both operations need the key object.

Source

Thrown at hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/bloom/RetouchedBloomFilter.java:121

  
  /**
   * Constructor
   * @param vectorSize The vector size of <i>this</i> filter.
   * @param nbHash The number of hash function to consider.
   * @param hashType type of the hashing function (see
   * {@link org.apache.hadoop.util.hash.Hash}).
   */
  public RetouchedBloomFilter(int vectorSize, int nbHash, int hashType) {
    super(vectorSize, nbHash, hashType);

    this.rand = null;
    createVector();
  }

  @Override
  public void add(Key key) {
    if (key == null) {
      throw new NullPointerException("key can not be null");
    }

    int[] h = hash.hash(key);
    hash.clear();

    for (int i = 0; i < nbHash; i++) {
      bits.set(h[i]);
      keyVector[h[i]].add(key);
    }
  }

  /**
   * Adds a false positive information to <i>this</i> retouched Bloom filter.
   * <p>
   * <b>Invariant</b>: if the false positive is <code>null</code>, nothing happens.
   * @param key The false positive key to add.
   */
  public void addFalsePositive(Key key) {

View on GitHub (pinned to 2add963021)

Solutions

Strip nulls from the input before the loop: keys.removeIf(Objects::isNull)
Guard each element: if (k != null) rbf.add(k)
Fix the producer so null keys never reach the filter

Example fix

// before
for (Key k : keys) {
  rbf.add(k); // NPE when k == null
}

// after
for (Key k : keys) {
  if (k != null) {
    rbf.add(k);
  }
}

Defensive patterns

Strategy: validation

Validate before calling

for (Key k : keys) {
  if (k != null) {
    rbf.add(k);
  }
}

Type guard

static boolean isAddableKey(Key k) {
  return k != null && k.getBytes() != null;
}

Prevention

Remove nulls once at the boundary: keys.removeIf(Objects::isNull)
Use Objects.requireNonNull(key) in your own wrapper so failures point at your layer, not the filter
Do not overload null with control-flow meaning (end-of-input) in filter feeds

When it happens

Trigger: rbf.add(null); batch loops iterating a collection that contains null elements; passing a Key variable that an upstream branch never assigned.

Common situations: Ingestion loops over nullable records; tests using Arrays.asList(null, key); code that uses null as a 'no key' sentinel.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/dae3acd923234872. Report an issue: GitHub.