apache/beam · error · IllegalStateException

the secondary key coder of SortValues must be deterministic

Error message

the secondary key coder of SortValues must be deterministic

What it means

SortValues sorts per-key data by serializing secondary keys, and Beam's sorting requires a total, stable order across workers. If the secondary key Coder is non-deterministic (encoding the same object yields different bytes), equal keys could sort inconsistently, so expand() calls verifyDeterministic() and wraps any NonDeterministicException in an IllegalStateException.

Solutions

Make the secondary key coder deterministic: encode fields in a fixed order (e.g., sort map keys before writing) in a custom coder and implement verifyDeterministic as a no-op/pass
Convert keys to a deterministic representation before sorting (e.g., a String or protobuf with canonical encoding)
Use a deterministic built-in coder (StringUtf8Coder, BigIntegerCoder, etc.) for the secondary key
If the coder is actually deterministic, override verifyDeterministic() in the custom coder to state that explicitly

Example fix

// before
// custom coder writes map entries in HashMap iteration order -> non-deterministic
// after
// custom coder: sorted entries before encoding
entries.sort(Comparator.comparing(e -> e.getKey()));
@Override public void verifyDeterministic() throws NonDeterministicException {}

Defensive patterns

Strategy: validation

Validate before calling

try { secondaryKeyCoder.verifyDeterministic(); } catch (Coder.NonDeterministicException e) { throw new IllegalStateException("Replace non-deterministic secondary key coder", e); }

Type guard

static <T> boolean deterministic(Coder<T> c) { try { c.verifyDeterministic(); return true; } catch (Coder.NonDeterministicException e) { return false; } }

Try / catch

try { return SortValues.perKey(); } catch (IllegalStateException e) { if (e.getMessage().contains("deterministic")) { /* swap coder and retry */ } throw e; }

Prevention

Prefer built-in deterministic coders (StringUtf8Coder, VarLongCoder) for sort keys
Implement verifyDeterministic() explicitly in custom coders with deterministic encoding
Avoid HashMap/set-based keys in sorted pipelines
Run verifyDeterministic at pipeline-construction time in tests

When it happens

Trigger: Applying SortValues.perKey() to a PCollection<KV<P, Iterable<KV<S, V>>>> where the KvCoder's secondary-key coder is non-deterministic — most commonly a custom coder for a POJO/map/JSONObject, or Coder of type Map/Struct whose field iteration order varies.

Common situations: Using a JsonCoder, AvroGenericRecordCoder with non-deterministic map fields, or a hand-written coder without implementing verifyDeterministic; keys containing HashMaps or unordered sets.

Understand the failure class

Background: Schema validation failed / invalid input schema: payload rejected because its shape doesn't match the expected schema — this error's family across 28 libraries.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/7ec10cc53dff82c7. Report an issue: GitHub.

Appendix: source

Thrown at sdks/java/extensions/sorter/src/main/java/org/apache/beam/sdk/extensions/sorter/SortValues.java:86

   * @param <SecondaryKeyT> the type of the secondary (sort) keys of the input and output {@code
   *     PCollection}s
   * @param <ValueT> the type of the values of the input and output {@code PCollection}s
   */
  public static <PrimaryKeyT, SecondaryKeyT, ValueT>
      SortValues<PrimaryKeyT, SecondaryKeyT, ValueT> create(
          BufferedExternalSorter.Options sorterOptions) {
    return new SortValues<>(sorterOptions);
  }

  @Override
  public PCollection<KV<PrimaryKeyT, Iterable<KV<SecondaryKeyT, ValueT>>>> expand(
      PCollection<KV<PrimaryKeyT, Iterable<KV<SecondaryKeyT, ValueT>>>> input) {

    Coder<SecondaryKeyT> secondaryKeyCoder = getSecondaryKeyCoder(input.getCoder());
    try {
      secondaryKeyCoder.verifyDeterministic();
    } catch (Coder.NonDeterministicException e) {
      throw new IllegalStateException(
          "the secondary key coder of SortValues must be deterministic", e);
    }

    return input
        .apply(
            ParDo.of(
                new SortValuesDoFn<>(
                    sorterOptions, secondaryKeyCoder, getValueCoder(input.getCoder()))))
        .setCoder(input.getCoder());
  }

  /** Retrieves the {@link Coder} for the secondary key-value pairs. */
  @SuppressWarnings("unchecked")
  private static <PrimaryKeyT, SecondaryKeyT, ValueT>
      KvCoder<SecondaryKeyT, ValueT> getSecondaryKeyValueCoder(
          Coder<KV<PrimaryKeyT, Iterable<KV<SecondaryKeyT, ValueT>>>> inputCoder) {
    if (!(inputCoder instanceof KvCoder)) {
      throw new IllegalStateException("SortValues requires its input to use KvCoder");

View on GitHub (pinned to 12126d8942)