apache/hadoop · error · IllegalArgumentException

filters cannot be and-ed

Error message

filters cannot be and-ed

What it means

CountingBloomFilter.and(Filter) intersects the bucket arrays of two counting filters and enforces the same compatibility contract as the other combining ops: the operand must be a non-null CountingBloomFilter with identical vectorSize and nbHash. Intersecting differently shaped counters is undefined (different bucket layouts and hash counts), so mismatches throw IllegalArgumentException('filters cannot be and-ed').

Source

Thrown at hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/bloom/CountingBloomFilter.java:168

      
      long bucketMask = 15L << bucketShift;
      long bucketValue = (buckets[wordNum] & bucketMask) >>> bucketShift;
      
      // only decrement if the count in the bucket is between 0 and BUCKET_MAX_VALUE
      if(bucketValue >= 1 && bucketValue < BUCKET_MAX_VALUE) {
        // decrement by 1
        buckets[wordNum] = (buckets[wordNum] & ~bucketMask) | ((bucketValue - 1) << bucketShift);
      }
    }
  }

  @Override
  public void and(Filter filter) {
    if(filter == null
        || !(filter instanceof CountingBloomFilter)
        || filter.vectorSize != this.vectorSize
        || filter.nbHash != this.nbHash) {
      throw new IllegalArgumentException("filters cannot be and-ed");
    }
    CountingBloomFilter cbf = (CountingBloomFilter)filter;
    
    int sizeInWords = buckets2words(vectorSize);
    for(int i = 0; i < sizeInWords; i++) {
      this.buckets[i] &= cbf.buckets[i];
    }
  }

  @Override
  public boolean membershipTest(Key key) {
    if(key == null) {
      throw new NullPointerException("Key may not be null");
    }

    int[] h = hash.hash(key);
    hash.clear();

View on GitHub (pinned to 2add963021)

Solutions

1. Construct all CountingBloomFilters meant to be combined with identical constructor parameters from shared constants.
2. Check instanceof CountingBloomFilter plus vectorSize/nbHash equality before and().
3. Re-add underlying elements from one side into a correctly shaped filter if shapes differ.
4. Catch IllegalArgumentException and report which producer's configuration diverged.

Example fix

// before
CountingBloomFilter a = new CountingBloomFilter(1024, 4, Hash.MURMUR_HASH);
CountingBloomFilter b = new CountingBloomFilter(512, 4, Hash.MURMUR_HASH);
a.and(b); // throws: filters cannot be and-ed

// after
CountingBloomFilter b = new CountingBloomFilter(1024, 4, Hash.MURMUR_HASH);
a.and(b);

static boolean mergeable(CountingBloomFilter target, Filter other) {
  return other instanceof CountingBloomFilter
      && other.vectorSize == target.vectorSize
      && other.nbHash == target.nbHash;
}

if (!mergeable(a, b)) {
  throw new IllegalStateException(
      "CountingBloomFilter shape mismatch: "
      + a.vectorSize + "/" + a.nbHash + " vs "
      + (b == null ? "null" : b.vectorSize + "/" + b.nbHash));
}
a.and(b);

static boolean isSameShapeCountingFilter(Filter f,
    int vectorSize, int nbHash) {
  return f instanceof CountingBloomFilter
      && f.vectorSize == vectorSize
      && f.nbHash == nbHash;
}

try {
  a.and(b);
} catch (IllegalArgumentException e) {
  LOG.error("Cannot intersect counting filters of different shape", e);
  rebuild(a, sourceData);
}

When it happens

Trigger: cbfA.and(cbfB) where cbfB was constructed with a different vectorSize or nbHash; cbfA.and(null); cbfA.and(new BloomFilter(...)) — a plain BloomFilter is rejected by the instanceof CountingBloomFilter check.

Common situations: Intersecting per-source counting filters (e.g. 'seen in both datasets') when producers were configured independently; one producer on older defaults for vector size; mixing a regular BloomFilter into counting-filter code paths.

Related errors

• filters cannot be and-ed
• filters cannot be or-ed
• filters cannot be xor-ed
• key can not be null
• Key may not be null

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/56e58493683b05ff. Report an issue: GitHub.