apache/hadoop · error · NullPointerException

key can not be null

Error message

key can not be null

What it means

CountingBloomFilter.add(Key) hashes the key and increments 4-bit counters in its long[] buckets array; a null key cannot be hashed and the method throws NullPointerException('key can not be null') as an explicit precondition (note the wording differs slightly from BloomFilter's 'key cannot be null'). Fail-fast up front avoids a JVM NPE from inside the hash implementation.

Source

Thrown at hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/bloom/CountingBloomFilter.java:107

   * @param nbHash The number of hash function to consider.
   * @param hashType type of the hashing function (see
   * {@link org.apache.hadoop.util.hash.Hash}).
   */
  public CountingBloomFilter(int vectorSize, int nbHash, int hashType) {
    super(vectorSize, nbHash, hashType);
    buckets = new long[buckets2words(vectorSize)];
  }

  /** returns the number of 64 bit words it would take to hold vectorSize buckets */
  private static int buckets2words(int vectorSize) {
   return ((vectorSize - 1) >>> 4) + 1;
  }


  @Override
  public void add(Key key) {
    if(key == null) {
      throw new NullPointerException("key can not be null");
    }

    int[] h = hash.hash(key);
    hash.clear();

    for(int i = 0; i < nbHash; i++) {
      // find the bucket
      int wordNum = h[i] >> 4;          // div 16
      int bucketShift = (h[i] & 0x0f) << 2;  // (mod 16) * 4
      
      long bucketMask = 15L << bucketShift;
      long bucketValue = (buckets[wordNum] & bucketMask) >>> bucketShift;
      
      // only increment if the count in the bucket is less than BUCKET_MAX_VALUE
      if(bucketValue < BUCKET_MAX_VALUE) {
        // increment by 1
        buckets[wordNum] = (buckets[wordNum] & ~bucketMask) | ((bucketValue + 1) << bucketShift);
      }

View on GitHub (pinned to 2add963021)

Solutions

1. Null-check the Key before add() and skip/reject the record.
2. Fix the producer of nulls (defaults, Optional, filter(Objects::nonNull)).
3. Add a boundary requireNonNull at data ingestion so failures name the real culprit.

Example fix

// before
cbf.add(parseKey(line)); // parseKey returned null for a malformed line
// throws NullPointerException: key can not be null

// after
Key k = parseKey(line);
if (k != null) {
  cbf.add(k);
}

Key k = parseKey(record);
if (k == null) {
  metrics.incrMalformed();
  return;
}
cbf.add(k);

try {
  cbf.add(key);
} catch (NullPointerException e) {
  LOG.warn("Rejected null key for counting bloom filter", e);
}

When it happens

Trigger: countingBloomFilter.add(null); adding keys pulled from a nullable source (map.get on absent key, sparsely populated array); pipelines where the key extractor returns null for malformed records.

Common situations: Frequency-count or duplicate-detection use of counting bloom filters on dirty data; record parsing where one field being absent yields a null Key; concurrent maps returning null under eviction races.

Related errors

• Key may not be null
• key cannot be null
• Key is not a member
• filters cannot be and-ed
• Value length unknown.

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/f85df2ad60abd93f. Report an issue: GitHub.