apache/hadoop · error · IllegalArgumentException

filters cannot be and-ed

Error message

filters cannot be and-ed

What it means

BloomFilter.and(Filter) merges another filter's bits into this one and requires that the operand is a BloomFilter (not null, not a subclass of another Filter type) and was constructed with the identical vectorSize and nbHash. Merging differently shaped filters would be meaningless (different bit universes, different hash count), so mismatches throw IllegalArgumentException with this message.

Source

Thrown at hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/bloom/BloomFilter.java:136

    if(key == null) {
      throw new NullPointerException("key cannot be null");
    }

    int[] h = hash.hash(key);
    hash.clear();

    for(int i = 0; i < nbHash; i++) {
      bits.set(h[i]);
    }
  }

  @Override
  public void and(Filter filter) {
    if(filter == null
        || !(filter instanceof BloomFilter)
        || filter.vectorSize != this.vectorSize
        || filter.nbHash != this.nbHash) {
      throw new IllegalArgumentException("filters cannot be and-ed");
    }

    this.bits.and(((BloomFilter) filter).bits);
  }

  @Override
  public boolean membershipTest(Key key) {
    if(key == null) {
      throw new NullPointerException("key cannot be null");
    }

    int[] h = hash.hash(key);
    hash.clear();
    for(int i = 0; i < nbHash; i++) {
      if(!bits.get(h[i])) {
        return false;
      }
    }

View on GitHub (pinned to 2add963021)

Solutions

1. Construct all filters you intend to combine with identical vectorSize and nbHash, ideally from one shared configuration constant.
2. Check compatibility before combining: filter != null && filter instanceof BloomFilter && filter.vectorSize == this.vectorSize && filter.nbHash == this.nbHash.
3. If shapes differ, rebuild one side or re-add elements from the raw data instead of merging bitsets.
4. Catch IllegalArgumentException around merge loops to identify the mismatched producer.

Example fix

// before
BloomFilter a = new BloomFilter(1024, 4, Hash.MURMUR_HASH);
BloomFilter b = new BloomFilter(640, 4, Hash.MURMUR_HASH);
a.and(b); // throws: filters cannot be and-ed

// after
BloomFilter b = new BloomFilter(1024, 4, Hash.MURMUR_HASH); // same shape
a.and(b);

static boolean isMergeable(BloomFilter target, Filter other) {
  return other instanceof BloomFilter
      && other.vectorSize == target.vectorSize
      && other.nbHash == target.nbHash;
}

if (!isMergeable(target, other)) {
  throw new IllegalStateException(
      "Cannot merge filter with shape " + shapeOf(other)
      + " into " + shapeOf(target));
}
target.and(other);

static boolean isMergeableBloomFilter(Filter f, int vectorSize, int nbHash) {
  return f != null
      && f instanceof BloomFilter
      && f.vectorSize == vectorSize
      && f.nbHash == nbHash;
}

try {
  target.and(other);
} catch (IllegalArgumentException e) {
  LOG.error("Filter shape mismatch during AND; rebuild from raw data", e);
  rebuildFromSource(target);
}

When it happens

Trigger: filterA.and(null); filterA.and(someRemoveFilter); filterA.and(filterB) where filterB was built with new BloomFilter(640, 8, Hash.JENKINS_HASH) but filterA used vectorSize 1024 — any vectorSize or nbHash difference between the two constructors triggers it.

Common situations: Merging per-shard or per-worker bloom filters that were configured from different settings; one component defaulting to a different vector size after a config change; aggregating filters built at different code versions with different defaults.

Related errors

• filters cannot be or-ed
• filters cannot be xor-ed
• filters cannot be and-ed
• Input files cannot be merged as they have different Key and
• Input files cannot be merged as they have different Key clas

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/b26ad5febb3ee2b5. Report an issue: GitHub.