What causes "value can not be null" in apache/hadoop?

Null-check the byte[] where it enters your code and skip/log the record instead of constructing a Key

apache/hadoop · error · IllegalArgumentException

value can not be null

Error message

value can not be null

What it means

org.apache.hadoop.util.bloom.Key is the byte-array wrapper that all Hadoop Bloom filter classes (BloomFilter, RetouchedBloomFilter, CountingBloomFilter, DynamicBloomFilter) hash. Both constructors delegate to set(byte[], double), which rejects a null byte array with IllegalArgumentException because the hash functions must read actual bytes. A null value here is always a caller bug, never filter state.

Source

Thrown at hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/bloom/Key.java:108

  /**
   * Constructor.
   * <p>
   * Builds a key with a specified weight.
   * @param value The value of <i>this</i> key.
   * @param weight The weight associated to <i>this</i> key.
   */
  public Key(byte[] value, double weight) {
    set(value, weight);
  }

  /**
   * @param value value.
   * @param weight weight.
   */
  public void set(byte[] value, double weight) {
    if (value == null) {
      throw new IllegalArgumentException("value can not be null");
    }
    this.bytes = value;
    this.weight = weight;
  }
  
  /** @return byte[] The value of <i>this</i> key. */
  public byte[] getBytes() {
    return this.bytes;
  }

  /** @return Returns the weight associated to <i>this</i> key. */
  public double getWeight() {
    return weight;
  }

  /**
   * Increments the weight of <i>this</i> key with a specified value. 
   * @param weight The increment.

View on GitHub (pinned to 2add963021)

Solutions

Null-check the byte[] where it enters your code and skip/log the record instead of constructing a Key
Trace the producer of the null (map.get() miss, unset field after readFields) and fix it there; the Key class is only the messenger
If an 'empty' key is meaningful in your model, encode it explicitly as new byte[0] rather than null

Example fix

// before
filter.add(new Key(bytes)); // throws IllegalArgumentException when bytes == null

// after
if (bytes == null) {
  LOG.warn("null key bytes; skipping record");
  continue;
}
filter.add(new Key(bytes));

Defensive patterns

Strategy: validation

Validate before calling

if (bytes == null) {
  LOG.warn("null key bytes; skipping record");
  continue;
}
filter.add(new Key(bytes));

Type guard

static boolean hasKeyBytes(byte[] bytes) {
  return bytes != null;
}

Try / catch

try {
  filter.add(new Key(bytes));
} catch (IllegalArgumentException e) {
  throw new IllegalArgumentException("null key bytes for record " + recordId, e);
}

Prevention

Never pass raw map.get() results straight into new Key()
Validate key bytes at record-parse time, closest to the data source
Log and count null keys as a data-quality signal instead of crashing the filter build

When it happens

Trigger: new Key(null), new Key(null, 1.0), or key.set(null, w); feeding filter.add()/membershipTest() with a byte[] that came back null from a map lookup, an unset deserialized field, or a failed conversion.

Common situations: Bulk-building MapFile/SetFile filters from records where a field is missing; Writable round-trips that skip writing empty arrays so readFields leaves the field null; refactors that turn a lookup into a null-returning one.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/d001ed15bef8a903. Report an issue: GitHub.