apache/beam · error · IllegalArgumentException

Coder must be deterministic to perform this sketch.${e.getMe

Error message

Coder must be deterministic to perform this sketch.${e.getMessage()}

What it means

SketchFrequencies.CountMinSketchFn.create requires a deterministic Coder because Count-Min Sketch hashes the encoded bytes of elements; a non-deterministic encoding would hash equal elements differently and break frequency counting. NonDeterministicException from verifyDeterministic() is converted into this IllegalArgumentException.

Source

Thrown at sdks/java/extensions/sketching/src/main/java/org/apache/beam/sdk/extensions/sketching/SketchFrequencies.java:365

    private CountMinSketchFn(final Coder<InputT> coder, double eps, double confidence) {
      this.epsilon = eps;
      this.confidence = confidence;
      this.width = (int) Math.ceil(2 / eps);
      this.depth = (int) Math.ceil(-Math.log(1 - confidence) / Math.log(2));
      this.inputCoder = coder;
    }

    /**
     * Returns a {@link CountMinSketchFn} combiner with the given input coder. <br>
     * <b>Warning :</b> the coder must be deterministic.
     *
     * @param coder the coder that encodes the elements' type
     */
    public static <InputT> CountMinSketchFn<InputT> create(Coder<InputT> coder) {
      try {
        coder.verifyDeterministic();
      } catch (Coder.NonDeterministicException e) {
        throw new IllegalArgumentException(
            "Coder must be deterministic to perform this sketch." + e.getMessage(), e);
      }
      return new CountMinSketchFn<>(coder, 0.01, 0.999);
    }

    /**
     * Returns a new {@link CountMinSketchFn} combiner with new precision accuracy parameters {@code
     * epsilon} and {@code confidence}.
     *
     * <p>Keep in mind that the lower the {@code epsilon} value, the greater the width, and the
     * greater the confidence, the greater the depth.
     *
     * @param epsilon the error relative to the total number of distinct elements
     * @param confidence the confidence in the result to not exceed the relative error
     */
    public CountMinSketchFn<InputT> withAccuracy(double epsilon, double confidence) {
      if (epsilon <= 0D) {
        throw new IllegalArgumentException("The relative error must be positive");

View on GitHub (pinned to 12126d8942)

Solutions

1. Provide a deterministic Coder (AvroCoder, custom Coder with stable field ordering, protobuf coder).
2. Map elements to a deterministically encodable type (String, Long, byte[]) before the transform.
3. If your custom coder is genuinely deterministic, override verifyDeterministic() to assert it.

Example fix

// before
.apply(SketchFrequencies.<Map<String,Integer>>perElement()
    .create(MapCoder.of(StringUtf8Coder.of(), VarIntCoder.of()))); // throws
// after
.apply(MapElements.via(new SimpleFunction<Map<String,Integer>, String>() {
  public String apply(Map<String,Integer> m) { return m.entrySet().stream()
      .map(e -> e.getKey()+":"+e.getValue()).sorted().collect(Collectors.joining(",")); }
}))
.apply(SketchFrequencies.<String>perElement().create(StringUtf8Coder.of()));

coder.verifyDeterministic(); // run before create to get a precise failure point

static boolean isDeterministic(Coder<?> c) { try { c.verifyDeterministic(); return true; } catch (Coder.NonDeterministicException e) { return false; } }

try { CountMinSketchFn.create(coder); } catch (IllegalArgumentException e) { /* switch to deterministic coder */ }

When it happens

Trigger: Calling SketchFrequencies.perElement().create(coder) with a coder for List/Map/custom POJO types whose verifyDeterministic() throws (e.g. ListCoder, MapCoder, or a default Bean coder).

Common situations: Counting frequencies of collections or structs; users reusing default coders without providing a deterministic one; refactor changing element type to a non-deterministic one.

Understand the failure class

Background: "Must be a positive integer", "Invalid value", "Unsupported": the invalid-argument-value error family, when a library rejects the value you pass — this error's family across 35 libraries.

Related errors

• Coder must be deterministic to perform this sketch.${e.getMe
• The relative error must be positive
• The confidence must be between 0 and 1
• The accumulators cannot be merged:${e.getMessage()}
• NonDeterministicException(target, message, e)

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/e7b9433d4a71d601. Report an issue: GitHub.