What causes "Key is not a member" in apache/hadoop?

Call rbf.membershipTest(k) first and skip non-members

apache/hadoop · error · IllegalArgumentException

Key is not a member

Error message

Key is not a member

What it means

selectiveClearing first runs membershipTest(k); a key whose bits are not all set cannot be retouched, so the call is refused with IllegalArgumentException. This guards the filter invariant that you may only remove keys the filter currently claims as members (true members and false positives both pass that test).

Source

Thrown at hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/bloom/RetouchedBloomFilter.java:205

    }

    for (int i = 0; i < keys.length; i++) {
      addFalsePositive(keys[i]);
    }
  }

  /**
   * Performs the selective clearing for a given key.
   * @param k The false positive key to remove from <i>this</i> retouched Bloom filter.
   * @param scheme The selective clearing scheme to apply.
   */
  public void selectiveClearing(Key k, short scheme) {
    if (k == null) {
      throw new NullPointerException("Key can not be null");
    }

    if (!membershipTest(k)) {
      throw new IllegalArgumentException("Key is not a member");
    }

    int index = 0;
    int[] h = hash.hash(k);

    switch(scheme) {

    case RANDOM:
      index = randomRemove();
      break;
    
    case MINIMUM_FN:
      index = minimumFnRemove(h);
      break;
    
    case MAXIMUM_FP:
      index = maximumFpRemove(h);
      break;

View on GitHub (pinned to 2add963021)

Solutions

Call rbf.membershipTest(k) first and skip non-members
Verify the Key bytes are byte-identical to those used at add() time (same encoding, same hash type)
Regenerate the false-positive list from the current filter generation instead of reusing an old one

Example fix

// before
rbf.selectiveClearing(fpKey, RemoveScheme.RATIO); // IllegalArgumentException if not a member

// after
if (rbf.membershipTest(fpKey)) {
  rbf.selectiveClearing(fpKey, RemoveScheme.RATIO);
} else {
  LOG.debug("skipping {}: not claimed by this filter", fpKey);
}

Defensive patterns

Strategy: validation

Validate before calling

if (rbf.membershipTest(fpKey)) {
  rbf.selectiveClearing(fpKey, RemoveScheme.RATIO);
} else {
  LOG.debug("not claimed by this filter; skipping {}", fpKey);
}

Type guard

static boolean isRetouchable(RetouchedBloomFilter f, Key k) {
  return k != null && f.membershipTest(k);
}

Try / catch

try {
  rbf.selectiveClearing(k, RemoveScheme.RATIO);
} catch (IllegalArgumentException e) {
  LOG.debug("key {} is not a member; skipping retouch", k);
}

Prevention

Always gate selectiveClearing with membershipTest - it is cheap compared to the retouch
Keep the byte encoding used for Keys stable across add and retouch stages
Rebuild false-positive lists from the same filter generation you retouch

When it happens

Trigger: Calling selectiveClearing for a key never added via add(); a key whose bits were already cleared by an earlier retouch (now a false negative); a key re-encoded differently (re-serialized byte[]) than at add() time; replaying an old false-positive list against a rebuilt filter.

Common situations: Retouch pipelines feeding candidate keys from external storage without verifying membership; schema/encoding drift changing the bytes used to build the Key; regenerating the filter but reusing stale false-positive logs.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/e4d81636f399e529. Report an issue: GitHub.