apache/beam · error · IllegalArgumentException

Coder must be deterministic to perform this sketch.${e.getMe

Error message

Coder must be deterministic to perform this sketch.${e.getMessage()}

What it means

ApproximateDistinct.ApproximateDistinctFn.create requires the input Coder to be deterministic because HyperLogLog sketches hash the encoded bytes of elements; non-deterministic encodings would give different hashes for equal elements and corrupt the estimate. create calls coder.verifyDeterministic() and converts NonDeterministicException into this IllegalArgumentException.

Source

Thrown at sdks/java/extensions/sketching/src/main/java/org/apache/beam/sdk/extensions/sketching/ApproximateDistinct.java:406

    private final Coder<InputT> inputCoder;

    private ApproximateDistinctFn(int p, int sp, Coder<InputT> coder) {
      this.p = p;
      this.sp = sp;
      inputCoder = coder;
    }

    /**
     * Returns an {@link ApproximateDistinctFn} combiner with the given input coder.
     *
     * @param coder the coder that encodes the elements' type
     */
    public static <InputT> ApproximateDistinctFn<InputT> create(Coder<InputT> coder) {
      try {
        coder.verifyDeterministic();
      } catch (Coder.NonDeterministicException e) {
        throw new IllegalArgumentException(
            "Coder must be deterministic to perform this sketch." + e.getMessage(), e);
      }
      return new ApproximateDistinctFn<>(12, 0, coder);
    }

    /**
     * Returns an {@link ApproximateDistinctFn} combiner with a new precision {@code p}.
     *
     * <p>Keep in mind that {@code p} cannot be lower than 4, because the estimation would be too
     * inaccurate.
     *
     * <p>See {@link ApproximateDistinct#precisionForRelativeError(double)} and {@link
     * ApproximateDistinct#relativeErrorForPrecision(int)} to have more information about the
     * relationship between precision and relative error.
     *
     * @param p the precision value for the normal representation
     */
    public ApproximateDistinctFn<InputT> withPrecision(int p) {

View on GitHub (pinned to 12126d8942)

Solutions

Supply a deterministic coder for the input type (e.g. wrap elements in AvroCoder or implement a custom deterministic Coder).
Pre-map elements to a deterministically encodable type (String, byte[], Long) before applying ApproximateDistinct.
If confident your coder is deterministic despite the default check, override verifyDeterministic() in your custom coder to declare determinism.

Example fix

// before
PCollection<List<String>> words = ...;
words.apply(ApproximateDistinct.<List<String>>globally()
    .create(ListCoder.of(StringUtf8Coder.of()))); // throws: ListCoder non-deterministic
// after
words.apply(MapElements.via(new SimpleFunction<List<String>, String>() {
  public String apply(List<String> in) { return String.join("\u0001", in); }
}))
 .apply(ApproximateDistinct.<String>globally().create(StringUtf8Coder.of()));

Defensive patterns

Strategy: validation

Validate before calling

// Before calling create, verify determinism yourself
coder.verifyDeterministic(); // throws Coder.NonDeterministicException if not

Type guard

static boolean isDeterministic(Coder<?> c) { try { c.verifyDeterministic(); return true; } catch (Coder.NonDeterministicException e) { return false; } }

Try / catch

try { ApproximateDistinctFn.create(coder); } catch (IllegalArgumentException e) { /* fall back to a deterministic representation */ }

Prevention

Prefer AvroCoder/protobuf coders for complex element types used in sketches
Avoid sketching List/Map/Bean-typed PCollections directly
Document deterministic-coder requirements next to sketch usage

When it happens

Trigger: Calling ApproximateDistinct.globally().withRepresentativeValueFn(...).create(coder) — i.e. ApproximateDistinctFn.create — with a coder like for a List, Map, or custom type whose verifyDeterministic() throws (e.g. new ListCoder(nonDeterministicCoder)).

Common situations: Sketching collections, structs, or POJOs whose default coders (ListCoder, MapCoder, Java Bean coders with nulls) are documented non-deterministic; users forgetting to register a custom deterministic coder.

Understand the failure class

Background: "Must be a positive integer", "Invalid value", "Unsupported": the invalid-argument-value error family, when a library rejects the value you pass — this error's family across 35 libraries.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/c6aaa30884d1388a. Report an issue: GitHub.