apache/beam · error · NonDeterministicException

Ordering of elements in a set may be non-deterministic.

Error message

Ordering of elements in a set may be non-deterministic.

What it means

SetCoder.verifyDeterministic() always throws NonDeterministicException because Sets have no defined iteration order, yet SetCoder encodes elements in arbitrary iteration order. Equal sets can therefore encode to different byte sequences, which breaks determinism requirements in Beam grouping operations.

Source

Thrown at sdks/java/core/src/main/java/org/apache/beam/sdk/coders/SetCoder.java:47

 *
 * @param <T> the type of the elements of the set
 */
public class SetCoder<T> extends IterableLikeCoder<T, Set<T>> {

  /** Produces a {@link SetCoder} with the given {@code elementCoder}. */
  public static <T> SetCoder<T> of(Coder<T> elementCoder) {
    return new SetCoder<>(elementCoder);
  }

  /**
   * {@inheritDoc}
   *
   * @throws NonDeterministicException always. Sets are not ordered, but they are encoded in the
   *     order of an arbitrary iteration.
   */
  @Override
  public void verifyDeterministic() throws NonDeterministicException {
    throw new NonDeterministicException(
        this, "Ordering of elements in a set may be non-deterministic.");
  }

  @Override
  public TypeDescriptor<Set<T>> getEncodedTypeDescriptor() {
    return new TypeDescriptor<Set<T>>() {}.where(
        new TypeParameter<T>() {}, getElemCoder().getEncodedTypeDescriptor());
  }

  /////////////////////////////////////////////////////////////////////////////
  // Internal operations below here.

  /**
   * {@inheritDoc}
   *
   * @return A new {@link Set} built from the elements in the {@link List} decoded by {@link
   *     IterableLikeCoder}.
   */

View on GitHub (pinned to 12126d8942)

Solutions

1. Use SortedSet/TreeSet with ListCoder or an ordered collection coder so iteration order is fixed
2. Encode as a deterministic structure: convert the set to a sorted List and use ListCoder
3. Use a primitive/array key representation instead of a Set for keyed operations
4. Call SetCoder.verifyDeterministic() only if you switch to an ordered Set implementation with deterministic element coder

Example fix

// before
PCollection<Set<String>> keys; // SetCoder -> NonDeterministicException on GroupByKey
// after
PCollection<List<String>> keys = sets.apply(MapElements.into(new TypeDescriptor<List<String>>() {}).via(s -> new ArrayList<>(new TreeSet<>(s))));

try { setCoder.verifyDeterministic(); } catch (NonDeterministicException e) { /* use sorted/list representation */ }

null

try { coder.verifyDeterministic(); } catch (NonDeterministicException e) { convert to sorted List / TreeSet before encoding }

When it happens

Trigger: Using a Set (via SetCoder.of(elementCoder)) as a PCollection element/key in GroupByKey, Combine, or any operation that calls verifyDeterministic().

Common situations: Grouping records keyed by a HashSet; validation at pipeline submit time; tests exercising Coder determinism checks.

Understand the failure class

Background: UnsupportedOperationException and "is not supported" errors: when a library deliberately refuses a call — this error's family across 30 libraries.

Related errors

• NonDeterministicException(target, message, e)
• Floating point encodings are not guaranteed to be determinis
• Ordering of entries in a Map may be non-deterministic.
• Java Serialization may be non-deterministic.
• Ordering of entries in a Map may be non-deterministic.

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/2861a3ba797ae282. Report an issue: GitHub.