apache/beam · error · IllegalStateException

the secondary key coder of SortValues must be deterministic

Error message

the secondary key coder of SortValues must be deterministic

What it means

SortValues sorts per-key data by serializing secondary keys, and Beam's sorting requires a total, stable order across workers. If the secondary key Coder is non-deterministic (encoding the same object yields different bytes), equal keys could sort inconsistently, so expand() calls verifyDeterministic() and wraps any NonDeterministicException in an IllegalStateException.

Source

Thrown at sdks/java/extensions/sorter/src/main/java/org/apache/beam/sdk/extensions/sorter/SortValues.java:86

   * @param <SecondaryKeyT> the type of the secondary (sort) keys of the input and output {@code
   *     PCollection}s
   * @param <ValueT> the type of the values of the input and output {@code PCollection}s
   */
  public static <PrimaryKeyT, SecondaryKeyT, ValueT>
      SortValues<PrimaryKeyT, SecondaryKeyT, ValueT> create(
          BufferedExternalSorter.Options sorterOptions) {
    return new SortValues<>(sorterOptions);
  }

  @Override
  public PCollection<KV<PrimaryKeyT, Iterable<KV<SecondaryKeyT, ValueT>>>> expand(
      PCollection<KV<PrimaryKeyT, Iterable<KV<SecondaryKeyT, ValueT>>>> input) {

    Coder<SecondaryKeyT> secondaryKeyCoder = getSecondaryKeyCoder(input.getCoder());
    try {
      secondaryKeyCoder.verifyDeterministic();
    } catch (Coder.NonDeterministicException e) {
      throw new IllegalStateException(
          "the secondary key coder of SortValues must be deterministic", e);
    }

    return input
        .apply(
            ParDo.of(
                new SortValuesDoFn<>(
                    sorterOptions, secondaryKeyCoder, getValueCoder(input.getCoder()))))
        .setCoder(input.getCoder());
  }

  /** Retrieves the {@link Coder} for the secondary key-value pairs. */
  @SuppressWarnings("unchecked")
  private static <PrimaryKeyT, SecondaryKeyT, ValueT>
      KvCoder<SecondaryKeyT, ValueT> getSecondaryKeyValueCoder(
          Coder<KV<PrimaryKeyT, Iterable<KV<SecondaryKeyT, ValueT>>>> inputCoder) {
    if (!(inputCoder instanceof KvCoder)) {
      throw new IllegalStateException("SortValues requires its input to use KvCoder");

View on GitHub (pinned to 12126d8942)

Solutions

1. Make the secondary key coder deterministic: encode fields in a fixed order (e.g., sort map keys before writing) in a custom coder and implement verifyDeterministic as a no-op/pass
2. Convert keys to a deterministic representation before sorting (e.g., a String or protobuf with canonical encoding)
3. Use a deterministic built-in coder (StringUtf8Coder, BigIntegerCoder, etc.) for the secondary key
4. If the coder is actually deterministic, override verifyDeterministic() in the custom coder to state that explicitly

Example fix

// before
// custom coder writes map entries in HashMap iteration order -> non-deterministic
// after
// custom coder: sorted entries before encoding
entries.sort(Comparator.comparing(e -> e.getKey()));
@Override public void verifyDeterministic() throws NonDeterministicException {}

try { secondaryKeyCoder.verifyDeterministic(); } catch (Coder.NonDeterministicException e) { throw new IllegalStateException("Replace non-deterministic secondary key coder", e); }

static <T> boolean deterministic(Coder<T> c) { try { c.verifyDeterministic(); return true; } catch (Coder.NonDeterministicException e) { return false; } }

try { return SortValues.perKey(); } catch (IllegalStateException e) { if (e.getMessage().contains("deterministic")) { /* swap coder and retry */ } throw e; }

When it happens

Trigger: Applying SortValues.perKey() to a PCollection<KV<P, Iterable<KV<S, V>>>> where the KvCoder's secondary-key coder is non-deterministic — most commonly a custom coder for a POJO/map/JSONObject, or Coder of type Map/Struct whose field iteration order varies.

Common situations: Using a JsonCoder, AvroGenericRecordCoder with non-deterministic map fields, or a hand-written coder without implementing verifyDeterministic; keys containing HashMaps or unordered sets.

Understand the failure class

Background: Schema validation failed / invalid input schema: payload rejected because its shape doesn't match the expected schema — this error's family across 28 libraries.

Related errors

• the keyCoder of a GroupByEncryptedKey must be deterministic
• the keyCoder of a GroupByKey must be deterministic
• Window coders must be deterministic.
• SortValues requires its input to use KvCoder
• SortValues requires the values be encoded with IterableCoder

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/7ec10cc53dff82c7. Report an issue: GitHub.