apache/beam · error · IllegalStateException

SortValues requires the values be encoded with IterableCoder

Error message

SortValues requires the values be encoded with IterableCoder

What it means

After confirming the input coder is a KvCoder, SortValues requires the value side to be an Iterable of KV pairs encoded with IterableCoder, since per-key sorting operates on collections of secondary key-value pairs. getSecondaryKeyValueCoder throws IllegalStateException when kvCoder.getValueCoder() is not an IterableCoder because it cannot extract the element coder of the sorted collection.

Source

Thrown at sdks/java/extensions/sorter/src/main/java/org/apache/beam/sdk/extensions/sorter/SortValues.java:111

                new SortValuesDoFn<>(
                    sorterOptions, secondaryKeyCoder, getValueCoder(input.getCoder()))))
        .setCoder(input.getCoder());
  }

  /** Retrieves the {@link Coder} for the secondary key-value pairs. */
  @SuppressWarnings("unchecked")
  private static <PrimaryKeyT, SecondaryKeyT, ValueT>
      KvCoder<SecondaryKeyT, ValueT> getSecondaryKeyValueCoder(
          Coder<KV<PrimaryKeyT, Iterable<KV<SecondaryKeyT, ValueT>>>> inputCoder) {
    if (!(inputCoder instanceof KvCoder)) {
      throw new IllegalStateException("SortValues requires its input to use KvCoder");
    }
    @SuppressWarnings("unchecked")
    KvCoder<PrimaryKeyT, Iterable<KV<SecondaryKeyT, ValueT>>> kvCoder =
        (KvCoder<PrimaryKeyT, Iterable<KV<SecondaryKeyT, ValueT>>>) inputCoder;

    if (!(kvCoder.getValueCoder() instanceof IterableCoder)) {
      throw new IllegalStateException(
          "SortValues requires the values be encoded with IterableCoder");
    }
    IterableCoder<KV<SecondaryKeyT, ValueT>> iterableCoder =
        (IterableCoder<KV<SecondaryKeyT, ValueT>>) kvCoder.getValueCoder();

    if (!(iterableCoder.getElemCoder() instanceof KvCoder)) {
      throw new IllegalStateException(
          "SortValues requires the secondary key-value pairs to use KvCoder");
    }
    return (KvCoder<SecondaryKeyT, ValueT>) iterableCoder.getElemCoder();
  }

  /** Retrieves the {@link Coder} for the secondary keys. */
  private static <PrimaryKeyT, SecondaryKeyT, ValueT> Coder<SecondaryKeyT> getSecondaryKeyCoder(
      Coder<KV<PrimaryKeyT, Iterable<KV<SecondaryKeyT, ValueT>>>> inputCoder) {
    return getSecondaryKeyValueCoder(inputCoder).getKeyCoder();
  }

View on GitHub (pinned to 12126d8942)

Solutions

1. Set the value coder to IterableCoder.of(KvCoder.of(secondaryKeyCoder, valueCoder)) via setCoder
2. Produce the values through CoGroupByKey/GroupByKey-style pipelines that infer IterableCoder
3. If the value is a List, wrap/convert it to a coder-compatible Iterable representation (IterableCoder) or buffer into a re-batched PCollection with the correct coder
4. Additionally ensure each element coder of the Iterable is itself a KvCoder (the next check in the same method)

Example fix

// before
input.setCoder(KvCoder.of(pkCoder, ListCoder.of(KvCoder.of(skCoder, vCoder))));
// after
input.setCoder(KvCoder.of(pkCoder, IterableCoder.of(KvCoder.of(skCoder, vCoder))));

Coder<?> valueCoder = ((KvCoder<?, ?>) input.getCoder()).getValueCoder();
if (!(valueCoder instanceof IterableCoder)) { throw new IllegalStateException("value must be IterableCoder, got " + valueCoder); }

static boolean hasIterableValueCoder(PCollection<?> pc) { return pc.getCoder() instanceof KvCoder && ((KvCoder<?, ?>) pc.getCoder()).getValueCoder() instanceof IterableCoder; }

try { SortValues.perKey(); } catch (IllegalStateException e) { if (e.getMessage().contains("IterableCoder")) { input.setCoder(KvCoder.of(pkCoder, IterableCoder.of(kvElemCoder))); } else { throw e; } }

When it happens

Trigger: Applying SortValues.perKey() where the value coder is ListCoder, CollectionCoder, a custom iterable coder, or any non-IterableCoder — e.g., input.setCoder(KvCoder.of(pkCoder, ListCoder.of(kvCoder))) instead of IterableCoder.

Common situations: Manually setting the value coder to ListCoder or SetCoder; keys grouped through a custom combine that emits lists; using a coder registry that maps the value type to a non-Iterable coder.

Understand the failure class

Background: "is not a compatible type" / "cannot merge" errors: when a value's type doesn't match what the library requires — this error's family across 65 libraries.

Related errors

• SortValues requires its input to use KvCoder
• GroupByKey requires its input to use KvCoder
• the secondary key coder of SortValues must be deterministic
• cannot encode a null Integer
• cannot encode a null Long

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/40fb56ead08040a8. Report an issue: GitHub.