{"record":{"id":"56e58493683b05ff","repo":"apache/hadoop","slug":"filters-cannot-be-and-ed-56e584","errorCode":null,"errorMessage":"filters cannot be and-ed","messagePattern":"filters cannot be and-ed","errorType":"exception","errorClass":"IllegalArgumentException","httpStatus":null,"severity":"error","filePath":"hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/bloom/CountingBloomFilter.java","lineNumber":168,"sourceCode":"      \n      long bucketMask = 15L << bucketShift;\n      long bucketValue = (buckets[wordNum] & bucketMask) >>> bucketShift;\n      \n      // only decrement if the count in the bucket is between 0 and BUCKET_MAX_VALUE\n      if(bucketValue >= 1 && bucketValue < BUCKET_MAX_VALUE) {\n        // decrement by 1\n        buckets[wordNum] = (buckets[wordNum] & ~bucketMask) | ((bucketValue - 1) << bucketShift);\n      }\n    }\n  }\n\n  @Override\n  public void and(Filter filter) {\n    if(filter == null\n        || !(filter instanceof CountingBloomFilter)\n        || filter.vectorSize != this.vectorSize\n        || filter.nbHash != this.nbHash) {\n      throw new IllegalArgumentException(\"filters cannot be and-ed\");\n    }\n    CountingBloomFilter cbf = (CountingBloomFilter)filter;\n    \n    int sizeInWords = buckets2words(vectorSize);\n    for(int i = 0; i < sizeInWords; i++) {\n      this.buckets[i] &= cbf.buckets[i];\n    }\n  }\n\n  @Override\n  public boolean membershipTest(Key key) {\n    if(key == null) {\n      throw new NullPointerException(\"Key may not be null\");\n    }\n\n    int[] h = hash.hash(key);\n    hash.clear();\n","sourceCodeStart":150,"sourceCodeEnd":186,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/bloom/CountingBloomFilter.java#L150-L186","documentation":"CountingBloomFilter.and(Filter) intersects the bucket arrays of two counting filters and enforces the same compatibility contract as the other combining ops: the operand must be a non-null CountingBloomFilter with identical vectorSize and nbHash. Intersecting differently shaped counters is undefined (different bucket layouts and hash counts), so mismatches throw IllegalArgumentException('filters cannot be and-ed').","triggerScenarios":"cbfA.and(cbfB) where cbfB was constructed with a different vectorSize or nbHash; cbfA.and(null); cbfA.and(new BloomFilter(...)) — a plain BloomFilter is rejected by the instanceof CountingBloomFilter check.","commonSituations":"Intersecting per-source counting filters (e.g. 'seen in both datasets') when producers were configured independently; one producer on older defaults for vector size; mixing a regular BloomFilter into counting-filter code paths.","solutions":["Construct all CountingBloomFilters meant to be combined with identical constructor parameters from shared constants.","Check instanceof CountingBloomFilter plus vectorSize/nbHash equality before and().","Re-add underlying elements from one side into a correctly shaped filter if shapes differ.","Catch IllegalArgumentException and report which producer's configuration diverged."],"exampleFix":"// before\nCountingBloomFilter a = new CountingBloomFilter(1024, 4, Hash.MURMUR_HASH);\nCountingBloomFilter b = new CountingBloomFilter(512, 4, Hash.MURMUR_HASH);\na.and(b); // throws: filters cannot be and-ed\n\n// after\nCountingBloomFilter b = new CountingBloomFilter(1024, 4, Hash.MURMUR_HASH);\na.and(b);","handlingStrategy":"validation","validationCode":"static boolean mergeable(CountingBloomFilter target, Filter other) {\n  return other instanceof CountingBloomFilter\n      && other.vectorSize == target.vectorSize\n      && other.nbHash == target.nbHash;\n}\n\nif (!mergeable(a, b)) {\n  throw new IllegalStateException(\n      \"CountingBloomFilter shape mismatch: \"\n      + a.vectorSize + \"/\" + a.nbHash + \" vs \"\n      + (b == null ? \"null\" : b.vectorSize + \"/\" + b.nbHash));\n}\na.and(b);","typeGuard":"static boolean isSameShapeCountingFilter(Filter f,\n    int vectorSize, int nbHash) {\n  return f instanceof CountingBloomFilter\n      && f.vectorSize == vectorSize\n      && f.nbHash == nbHash;\n}","tryCatchPattern":"try {\n  a.and(b);\n} catch (IllegalArgumentException e) {\n  LOG.error(\"Cannot intersect counting filters of different shape\", e);\n  rebuild(a, sourceData);\n}","preventionTips":["Share one constructor-parameters constant across all counting filters that combine.","Keep plain BloomFilter and CountingBloomFilter pipelines separate; instanceof check rejects mixing.","Validate shape of imported/persisted filters before intersection."],"tags":["bloom-filter","counting-bloom-filter","incompatible-parameters","merge","hadoop-common"],"backgroundTag":"bloom-filter-incompatible","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}