apache/druid · error · IllegalArgumentException

bf1Length %d does not match bf2Length %d

Error message

bf1Length %d does not match bf2Length %d

What it means

BloomKFilter.mergeBloomFilterBytes() merges two serialized bloom filter byte arrays by OR-ing their bitsets, which is only valid when both filters share identical serialization parameters (num hash functions, bitset size) and therefore identical byte lengths. When bf1Length != bf2Length it throws IllegalArgumentException reporting both lengths; a following header check also rejects mismatched hash/bit parameters.

Source

Thrown at extensions-core/druid-bloom-filter/src/main/java/org/apache/druid/query/filter/BloomKFilter.java:206

   *
   * @param bf1Bytes
   * @param bf1Start
   * @param bf1Length
   * @param bf2Bytes
   * @param bf2Start
   * @param bf2Length
   */
  public static void mergeBloomFilterBytes(
      byte[] bf1Bytes,
      int bf1Start,
      int bf1Length,
      byte[] bf2Bytes,
      int bf2Start,
      int bf2Length
  )
  {
    if (bf1Length != bf2Length) {
      throw new IllegalArgumentException("bf1Length " + bf1Length + " does not match bf2Length " + bf2Length);
    }

    // Validation on the bitset size/3 hash functions.
    for (int idx = 0; idx < START_OF_SERIALIZED_LONGS; ++idx) {
      if (bf1Bytes[bf1Start + idx] != bf2Bytes[bf2Start + idx]) {
        throw new IllegalArgumentException("bf1 NumHashFunctions/NumBits does not match bf2");
      }
    }

    // Just bitwise-OR the bits together - size/# functions should be the same,
    // rest of the data is serialized long values for the bitset which are supposed to be bitwise-ORed.
    for (int idx = START_OF_SERIALIZED_LONGS; idx < bf1Length; ++idx) {
      bf1Bytes[bf1Start + idx] |= bf2Bytes[bf2Start + idx];
    }
  }

  public static void serialize(ByteBuffer out, BloomKFilter bloomFilter)
  {

View on GitHub (pinned to 9b90983fd2)

Solutions

  1. Use the same maxNumEntries (and numHashFunctions) for every bloom filter aggregation being merged, across all queries and subqueries
  2. Verify byte offsets: pass the full serialized length for each buffer and correct start positions
  3. Re-generate mismatched stored/serialized filters with the current parameters
  4. Catch IllegalArgumentException and treat as incompatible filters if merging user-supplied filters of unknown provenance

Example fix

// before
BloomKFilter f1 = new BloomKFilter(1000);
BloomKFilter f2 = new BloomKFilter(1500); // different bitset size
BloomKFilter.mergeBloomFilterBytes(f1.toByteBuffer().array(), 0, len1, f2.toByteBuffer().array(), 0, len2); // throws
// after
int maxNumEntries = 1500; // shared constant for all filters
BloomKFilter f1 = new BloomKFilter(maxNumEntries);
BloomKFilter f2 = new BloomKFilter(maxNumEntries);
BloomKFilter.mergeBloomFilterBytes(f1.toByteBuffer().array(), 0, len1, f2.toByteBuffer().array(), 0, len2);
Defensive patterns

Strategy: validation

Validate before calling

// Validate compatibility before merging
static boolean mergeable(byte[] f1, int l1, byte[] f2, int l2) {
  return f1 != null && f2 != null && l1 == l2;
}

Try / catch

try {
  BloomKFilter.mergeBloomFilterBytes(bf1, s1, l1, bf2, s2, l2);
} catch (IllegalArgumentException e) {
  if (e.getMessage().contains("does not match") || e.getMessage().contains("does not match bf2")) {
    throw new IllegalStateException("Incompatible bloom filter parameters (maxNumEntries/hash functions)", e);
  }
  throw e;
}

Prevention

When it happens

Trigger: Merging bloom filters created with different maxNumEntries/numBits or different numHashFunctions, e.g. combining query results from segments or subqueries configured with different maxNumEntries; passing truncated or corrupted serialized buffers; merging a partially-read buffer whose length slice is wrong.

Common situations: Aggregating bloom filters across subqueries where one used default maxNumEntries (1500) and another a custom value; hand-slicing serialized byte arrays with incorrect start/length offsets; upgrading configs so old stored filters no longer match new query filters.

Understand the failure class

Background: "Must be a positive integer", "Invalid value", "Unsupported": the invalid-argument-value error family, when a library rejects the value you pass — this error's family across 35 libraries.

Related errors

AI-assisted analysis of apache/druid@9b90983fd2 (2026-09-07). Data as JSON: /api/errors/0e1cf2273b4dc286. Report an issue: GitHub.