What causes "Key can not be null" in apache/hadoop?

Null-check keys as they leave the queue and stop or skip on null

apache/hadoop · error · NullPointerException

Key can not be null

Error message

Key can not be null

What it means

selectiveClearing(Key k, short scheme) retouches the filter for one false positive: it hashes the key and clears one bit according to the scheme. The null check throws NullPointerException before hashing because there is no key to retouch.

Source

Thrown at hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/bloom/RetouchedBloomFilter.java:201

   */
  public void addFalsePositive(Key[] keys) {
    if (keys == null) {
      throw new NullPointerException("Key[] can not be null");
    }

    for (int i = 0; i < keys.length; i++) {
      addFalsePositive(keys[i]);
    }
  }

  /**
   * Performs the selective clearing for a given key.
   * @param k The false positive key to remove from <i>this</i> retouched Bloom filter.
   * @param scheme The selective clearing scheme to apply.
   */
  public void selectiveClearing(Key k, short scheme) {
    if (k == null) {
      throw new NullPointerException("Key can not be null");
    }

    if (!membershipTest(k)) {
      throw new IllegalArgumentException("Key is not a member");
    }

    int index = 0;
    int[] h = hash.hash(k);

    switch(scheme) {

    case RANDOM:
      index = randomRemove();
      break;
    
    case MINIMUM_FN:
      index = minimumFnRemove(h);
      break;

View on GitHub (pinned to 2add963021)

Solutions

Null-check keys as they leave the queue and stop or skip on null
Use an explicit shutdown sentinel object instead of null
Guard at the call site: if (k != null) rbf.selectiveClearing(k, scheme)

Example fix

// before
rbf.selectiveClearing(queue.take(), RemoveScheme.RANDOM); // NPE if take() returns null

// after
Key k = queue.take();
if (k != null) {
  rbf.selectiveClearing(k, RemoveScheme.RANDOM);
}

Defensive patterns

Strategy: validation

Validate before calling

Key k = queue.poll();
if (k == null) {
  break; // or continue, per your pipeline semantics
}
rbf.selectiveClearing(k, RemoveScheme.RATIO);

Prevention

Use an explicit shutdown sentinel object instead of null in queues
Guard keys at the point they leave the producer, not deep in the consumer
Validate the whole key source in unit tests with null-inclusive fixtures

When it happens

Trigger: rbf.selectiveClearing(null, RemoveScheme.RANDOM); retouch loops pulling keys from a queue that uses null as a poison pill; calling with a lookup result without a presence check.

Common situations: Producer/consumer retouch pipelines; tests that stub the key source with null; null-means-done conventions carried over from older loop code.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/0b659c2ede54fbf7. Report an issue: GitHub.