apache/beam · error · CoderException

cannot encode a null T-Digest sketch

Error message

cannot encode a null T-Digest sketch

What it means

The T-Digest coder's encode() throws CoderException when the MergingDigest value is null, since a null digest has no byte representation for Beam's serialization pipeline. This mirrors the Count-Min Sketch coder's null policy: nulls are programmer errors and are rejected loudly rather than silently encoded as empty.

Source

Thrown at sdks/java/extensions/sketching/src/main/java/org/apache/beam/sdk/extensions/sketching/TDigestQuantiles.java:320

      return new MergingDigestCoder();
    }

    @Override
    public void populateDisplayData(DisplayData.Builder builder) {
      super.populateDisplayData(builder);
      builder.add(DisplayData.item("compression", compression).withLabel("Compression factor"));
    }
  }

  /** Coder for {@link MergingDigest} class. */
  static class MergingDigestCoder extends CustomCoder<MergingDigest> {

    private static final ByteArrayCoder BYTE_ARRAY_CODER = ByteArrayCoder.of();

    @Override
    public void encode(MergingDigest value, OutputStream outStream) throws IOException {
      if (value == null) {
        throw new CoderException("cannot encode a null T-Digest sketch");
      }
      ByteBuffer buf = ByteBuffer.allocate(value.byteSize());
      value.asBytes(buf);
      BYTE_ARRAY_CODER.encode(buf.array(), outStream);
    }

    @Override
    public MergingDigest decode(InputStream inStream) throws IOException {
      byte[] bytes = BYTE_ARRAY_CODER.decode(inStream);
      ByteBuffer buf = ByteBuffer.wrap(bytes);
      return MergingDigest.fromBytes(buf);
    }

    @Override
    public boolean isRegisterByteSizeObserverCheap(MergingDigest value) {
      return true;
    }

View on GitHub (pinned to 12126d8942)

Solutions

1. Return a fresh MergingDigest (TDigestQuantilesFn.createAccumulator()) instead of null wherever a digest may be absent.
2. Audit CombineFn.createAccumulator/mergeAccumulators for null returns.
3. Initialize state cells before first read in StateSpec usage.
4. Map nulls to empty digests before the coder boundary if third-party code may produce them.

Example fix

// before
return accumulator == null ? null : accumulator;

// after
return accumulator == null ? new MergingDigest(compression) : accumulator;

if (digest == null) {
  digest = tdigestFn.createAccumulator(); // fresh empty MergingDigest
}

boolean hasDigest(MergingDigest d) { return d != null; }

try {
  coded.apply(Combine.globally(tdigestFn));
} catch (CoderException e) {
  if (e.getMessage() != null && e.getMessage().contains("cannot encode a null T-Digest")) {
    throw new IllegalStateException("Null MergingDigest emitted — check createAccumulator/mergeAccumulators", e);
  }
  throw e;
}

When it happens

Trigger: A null MergingDigest reaches Beam's coder layer — e.g. a CombineFn that emits/extracts a null accumulator, a DoFn outputting null, or an uninitialized state value being shuffled or stored with TDigestQuantiles' coder.

Common situations: GlobalWindows/GBK paths serializing digests after a combine produced null; StateSpec-backed digests read before first write; refactored pipelines where a null return replaced an empty digest.

Related errors

• cannot encode a null Count-min Sketch
• cannot encode a null Integer
• cannot encode a null Long
• cannot encode a null Short
• cannot encode a null BitSet

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/67c1354a0c204db7. Report an issue: GitHub.