apache/beam · error · CoderException

cannot encode a null Count-min Sketch

Error message

cannot encode a null Count-min Sketch

What it means

The CountMinSketchCoder's encode() method throws CoderException when asked to encode a null Sketch value. Beam coders must serialize values for shuffle/stage handoff, and this coder deliberately rejects nulls because a null sketch has no serialized representation. The null should never reach encode() in a well-formed pipeline.

Source

Thrown at sdks/java/extensions/sketching/src/main/java/org/apache/beam/sdk/extensions/sketching/SketchFrequencies.java:507

    }

    /**
     * Utility class to retrieve the estimate frequency of an element from a {@link CountMinSketch}.
     */
    public long estimateCount(T element, Coder<T> coder) {
      return sketch().estimateCount(hashElement(element, coder));
    }
  }

  /** Coder for {@link CountMinSketch} class. */
  static class CountMinSketchCoder<T> extends CustomCoder<Sketch<T>> {

    private static final ByteArrayCoder BYTE_ARRAY_CODER = ByteArrayCoder.of();

    @Override
    public void encode(Sketch<T> value, OutputStream outStream) throws IOException {
      if (value == null) {
        throw new CoderException("cannot encode a null Count-min Sketch");
      }
      BYTE_ARRAY_CODER.encode(CountMinSketch.serialize(value.sketch()), outStream);
    }

    @Override
    public Sketch<T> decode(InputStream inStream) throws IOException {
      byte[] sketchBytes = BYTE_ARRAY_CODER.decode(inStream);
      CountMinSketch sketch = CountMinSketch.deserialize(sketchBytes);
      return Sketch.create(sketch);
    }

    @Override
    public boolean isRegisterByteSizeObserverCheap(Sketch<T> value) {
      return true;
    }

    @Override
    protected long getEncodedElementByteSize(Sketch<T> value) throws IOException {

View on GitHub (pinned to 12126d8942)

Solutions

1. Ensure the code producing Sketch<T> values never emits null — return a new empty CountMinSketch instead.
2. In CombineFn combine/extractOutput, guard against null accumulators and create one via createAccumulator().
3. If nulls are legitimately possible, filter or map them to empty sketches before the coder boundary.
4. Catch CoderException at the transform boundary only to aid debugging; it signals a bug, not expected input.

Example fix

// before
context.output(value == null ? null : sketch);

// after
context.output(value == null ? CountMinSketch.builder().build(CODER) : sketch);

if (sketch == null) {
  sketch = CountMinSketch.builder().build(); // or the fn's createAccumulator()
}

boolean isUsableSketch(Sketch<?> s) { return s != null; }

try {
  coded.apply(GroupByKey.create());
} catch (CoderException e) {
  if (e.getMessage() != null && e.getMessage().contains("cannot encode a null")) {
    throw new IllegalStateException("Null sketch emitted upstream — fix producing DoFn/CombineFn", e);
  }
  throw e;
}

When it happens

Trigger: A null Sketch<T> value flows into Beam's coder pipeline (e.g. during shuffle of a PCollection of Sketch<T>, GBK output, or state/storage) so Coder.encode is invoked with null — typically when a DoFn emits null sketches or an accumulator is null.

Common situations: A CombineFn accumulator path emitting null; user DoFns outputting null sketches instead of empty ones; deserialized/state-backed values that were never initialized.

Related errors

• cannot encode a null T-Digest sketch
• cannot encode a null Integer
• cannot encode a null Long
• cannot encode a null Short
• cannot encode a null BitSet

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/b519a70c8bd8c3c2. Report an issue: GitHub.