apache/beam · error · IllegalStateException

SortValues requires the values be encoded with IterableCoder

Error message

SortValues requires the values be encoded with IterableCoder

What it means

After confirming the input coder is a KvCoder, SortValues requires the value side to be an Iterable of KV pairs encoded with IterableCoder, since per-key sorting operates on collections of secondary key-value pairs. getSecondaryKeyValueCoder throws IllegalStateException when kvCoder.getValueCoder() is not an IterableCoder because it cannot extract the element coder of the sorted collection.

Solutions

1. Set the value coder to IterableCoder.of(KvCoder.of(secondaryKeyCoder, valueCoder)) via setCoder
2. Produce the values through CoGroupByKey/GroupByKey-style pipelines that infer IterableCoder
3. If the value is a List, wrap/convert it to a coder-compatible Iterable representation (IterableCoder) or buffer into a re-batched PCollection with the correct coder
4. Additionally ensure each element coder of the Iterable is itself a KvCoder (the next check in the same method)

Example fix

// before
input.setCoder(KvCoder.of(pkCoder, ListCoder.of(KvCoder.of(skCoder, vCoder))));
// after
input.setCoder(KvCoder.of(pkCoder, IterableCoder.of(KvCoder.of(skCoder, vCoder))));

Coder<?> valueCoder = ((KvCoder<?, ?>) input.getCoder()).getValueCoder();
if (!(valueCoder instanceof IterableCoder)) { throw new IllegalStateException("value must be IterableCoder, got " + valueCoder); }

static boolean hasIterableValueCoder(PCollection<?> pc) { return pc.getCoder() instanceof KvCoder && ((KvCoder<?, ?>) pc.getCoder()).getValueCoder() instanceof IterableCoder; }

try { SortValues.perKey(); } catch (IllegalStateException e) { if (e.getMessage().contains("IterableCoder")) { input.setCoder(KvCoder.of(pkCoder, IterableCoder.of(kvElemCoder))); } else { throw e; } }

When it happens

Trigger: Applying SortValues.perKey() where the value coder is ListCoder, CollectionCoder, a custom iterable coder, or any non-IterableCoder — e.g., input.setCoder(KvCoder.of(pkCoder, ListCoder.of(kvCoder))) instead of IterableCoder.

Common situations: Manually setting the value coder to ListCoder or SetCoder; keys grouped through a custom combine that emits lists; using a coder registry that maps the value type to a non-Iterable coder.

Understand the failure class

Background: "is not a compatible type" / "cannot merge" errors: when a value's type doesn't match what the library requires — this error's family across 65 libraries.

Related errors

• SortValues requires its input to use KvCoder
• GroupByKey requires its input to use KvCoder
• the secondary key coder of SortValues must be deterministic
• ApproximateUnique.PerKey requires its input to use KvCoder
• AvroCoder for GenericRecord requires a schema

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/40fb56ead08040a8. Report an issue: GitHub.

Appendix: source

Thrown at sdks/java/extensions/sorter/src/main/java/org/apache/beam/sdk/extensions/sorter/SortValues.java:111

                new SortValuesDoFn<>(
                    sorterOptions, secondaryKeyCoder, getValueCoder(input.getCoder()))))
        .setCoder(input.getCoder());
  }

  /** Retrieves the {@link Coder} for the secondary key-value pairs. */
  @SuppressWarnings("unchecked")
  private static <PrimaryKeyT, SecondaryKeyT, ValueT>
      KvCoder<SecondaryKeyT, ValueT> getSecondaryKeyValueCoder(
          Coder<KV<PrimaryKeyT, Iterable<KV<SecondaryKeyT, ValueT>>>> inputCoder) {
    if (!(inputCoder instanceof KvCoder)) {
      throw new IllegalStateException("SortValues requires its input to use KvCoder");
    }
    @SuppressWarnings("unchecked")
    KvCoder<PrimaryKeyT, Iterable<KV<SecondaryKeyT, ValueT>>> kvCoder =
        (KvCoder<PrimaryKeyT, Iterable<KV<SecondaryKeyT, ValueT>>>) inputCoder;

    if (!(kvCoder.getValueCoder() instanceof IterableCoder)) {
      throw new IllegalStateException(
          "SortValues requires the values be encoded with IterableCoder");
    }
    IterableCoder<KV<SecondaryKeyT, ValueT>> iterableCoder =
        (IterableCoder<KV<SecondaryKeyT, ValueT>>) kvCoder.getValueCoder();

    if (!(iterableCoder.getElemCoder() instanceof KvCoder)) {
      throw new IllegalStateException(
          "SortValues requires the secondary key-value pairs to use KvCoder");
    }
    return (KvCoder<SecondaryKeyT, ValueT>) iterableCoder.getElemCoder();
  }

  /** Retrieves the {@link Coder} for the secondary keys. */
  private static <PrimaryKeyT, SecondaryKeyT, ValueT> Coder<SecondaryKeyT> getSecondaryKeyCoder(
      Coder<KV<PrimaryKeyT, Iterable<KV<SecondaryKeyT, ValueT>>>> inputCoder) {
    return getSecondaryKeyValueCoder(inputCoder).getKeyCoder();
  }

View on GitHub (pinned to 12126d8942)