apache/beam · error · CoderException

cannot encode a null KV

Error message

cannot encode a null KV

What it means

KvCoder.encode refuses the null KV passed to it: it immediately delegates to keyCoder/valueCoder on kv.getKey()/getValue(), which would NPE, so it fails fast with CoderException instead. Use NullableCoder.of(KvCoder.of(...)) if null KVs are expected in the data.

Source

Thrown at sdks/java/core/src/main/java/org/apache/beam/sdk/coders/KvCoder.java:70

  private final Coder<K> keyCoder;
  private final Coder<V> valueCoder;

  private KvCoder(Coder<K> keyCoder, Coder<V> valueCoder) {
    this.keyCoder = keyCoder;
    this.valueCoder = valueCoder;
  }

  @Override
  public void encode(KV<K, V> kv, OutputStream outStream) throws IOException, CoderException {
    encode(kv, outStream, Context.NESTED);
  }

  @Override
  public void encode(KV<K, V> kv, OutputStream outStream, Context context)
      throws IOException, CoderException {
    if (kv == null) {
      throw new CoderException("cannot encode a null KV");
    }
    keyCoder.encode(kv.getKey(), outStream);
    valueCoder.encode(kv.getValue(), outStream, context);
  }

  @Override
  public KV<K, V> decode(InputStream inStream) throws IOException, CoderException {
    return decode(inStream, Context.NESTED);
  }

  @Override
  public KV<K, V> decode(InputStream inStream, Context context) throws IOException, CoderException {
    K key = keyCoder.decode(inStream);
    V value = valueCoder.decode(inStream, context);
    return KV.of(key, value);
  }

  @Override

View on GitHub (pinned to 12126d8942)

Solutions

1. Filter out null KV elements (Filter.by(kv -> kv != null)) before grouping/encoding stages.
2. Emit an empty/sentinel KV instead of null from producing transforms.
3. Model optionality with Optional<KV<K,V>> and filter None cases out of the coded PCollection.

Example fix

// before
return hit == null ? null : KV.of(key, hit); // null KV later throws
// after
return hit == null ? null /* filtered below */ : KV.of(key, hit);
.apply(Filter.by(kv -> kv != null));

if (kv != null) { coder.encode(kv, out, context); }

When it happens

Trigger: Invoking KvCoder.encode(null, out, context) directly, or a PCollection<KV<K,V>> element that is null reaching an encoding/shuffle stage.

Common situations: DoFns that map lookup misses to null KV instead of skipping them; building records where an optional key-value pair was modeled as a null KV.

Related errors

• cannot encode a null Instant
• cannot encode a null {iterableName}
• cannot encode a null Iterable
• cannot encode a null Integer
• cannot encode a null Long

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/b01072d0e84fc7bc. Report an issue: GitHub.