apache/beam · error · IllegalStateException

SortValues requires the secondary key-value pairs to use…

Error message

SortValues requires the secondary key-value pairs to use KvCoder

What it means

SortValues only works on PCollections whose values are encoded as an iterable of KV pairs wrapped in a KvCoder, because it must obtain typed coders for the secondary key and value to feed the external sort. getSecondaryKeyValueCoder casts the element coder of the IterableCoder to KvCoder and throws IllegalStateException when the cast fails. This guards the downstream sorting transform against a coder it cannot introspect.

Solutions

1. Ensure the value PCollection's element coder is KvCoder.of(secondaryKeyCoder, valueCoder) via pipeline.apply(..., PCollection.setCoder(KvCoder.of(...))) or CoderRegistry inference.
2. Check what the values actually are: SortValues expects values of type Iterable<KV<SecondaryKeyT, ValueT>>; fix upstream types so iterable elements are genuine KV pairs.
3. If using a custom coder wrapper, switch to KvCoder or supply coders explicitly through SortValues' typed construction instead of relying on inference.

Example fix

// before
PCollection<KV<byte[], Iterable<KV<String, Integer>>>> sorted =
    input.apply("sort", SortValues.create()); // elem coder not KvCoder

// after
PCollection<KV<byte[], Iterable<KV<String, Integer>>>> sorted =
    input.setCoder(KvCoder.of(
        ByteArrayCoder.of(),
        IterableCoder.of(KvCoder.of(StringUtf8Coder.of(), VarIntCoder.of()))))
      .apply("sort", SortValues.create());

Coder<?> valueCoder = input.getCoder();
if (!(valueCoder instanceof KvCoder)) throw new IllegalArgumentException(
    "SortValues input must be encoded with KvCoder, got: " + valueCoder);
KvCoder<?, ?> kv = (KvCoder<?, ?>) valueCoder;
if (!(kv.getValueCoder() instanceof IterableCoder)) throw new IllegalArgumentException(
    "SortValues values must be IterableCoder");
IterableCoder<?, ?> it = (IterableCoder<?, ?>) kv.getValueCoder();
if (!(it.getElemCoder() instanceof KvCoder)) throw new IllegalArgumentException(
    "SortValues secondary pairs must use KvCoder, got: " + it.getElemCoder());

When it happens

Trigger: Applying SortValues to a PCollection created from a source whose values use IterableCoder<KV<K,V>> but whose element coder is not a KvCoder — e.g. a custom Coder, a CoGbkResult-like coder, or a coder inferred as a generic iterable coder rather than KvCoder.of(k,v).

Common situations: Developers pipe in results of a CoGroupByKey or build PCollections manually and set an explicit custom coder for the KV pairs; or a DoFn output's default coder resolves to something other than KvCoder (e.g. after a map that erases type information).

Understand the failure class

Background: Type mismatch errors: IllegalArgumentException, TypeError and type guards across 150 open-source libraries — this error's family across 150 libraries.

Related errors

• ApproximateUnique.PerKey requires its input to use KvCoder
• AvroCoder for GenericRecord requires a schema
• cannot encode a null BitSet
• cannot encode a null byte[]
• cannot encode a null Integer

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/acb8fe3996f761f8. Report an issue: GitHub.

Appendix: source

Thrown at sdks/java/extensions/sorter/src/main/java/org/apache/beam/sdk/extensions/sorter/SortValues.java:118

  private static <PrimaryKeyT, SecondaryKeyT, ValueT>
      KvCoder<SecondaryKeyT, ValueT> getSecondaryKeyValueCoder(
          Coder<KV<PrimaryKeyT, Iterable<KV<SecondaryKeyT, ValueT>>>> inputCoder) {
    if (!(inputCoder instanceof KvCoder)) {
      throw new IllegalStateException("SortValues requires its input to use KvCoder");
    }
    @SuppressWarnings("unchecked")
    KvCoder<PrimaryKeyT, Iterable<KV<SecondaryKeyT, ValueT>>> kvCoder =
        (KvCoder<PrimaryKeyT, Iterable<KV<SecondaryKeyT, ValueT>>>) inputCoder;

    if (!(kvCoder.getValueCoder() instanceof IterableCoder)) {
      throw new IllegalStateException(
          "SortValues requires the values be encoded with IterableCoder");
    }
    IterableCoder<KV<SecondaryKeyT, ValueT>> iterableCoder =
        (IterableCoder<KV<SecondaryKeyT, ValueT>>) kvCoder.getValueCoder();

    if (!(iterableCoder.getElemCoder() instanceof KvCoder)) {
      throw new IllegalStateException(
          "SortValues requires the secondary key-value pairs to use KvCoder");
    }
    return (KvCoder<SecondaryKeyT, ValueT>) iterableCoder.getElemCoder();
  }

  /** Retrieves the {@link Coder} for the secondary keys. */
  private static <PrimaryKeyT, SecondaryKeyT, ValueT> Coder<SecondaryKeyT> getSecondaryKeyCoder(
      Coder<KV<PrimaryKeyT, Iterable<KV<SecondaryKeyT, ValueT>>>> inputCoder) {
    return getSecondaryKeyValueCoder(inputCoder).getKeyCoder();
  }

  /** Returns the {@code Coder} of the values associated with the secondary keys. */
  private static <PrimaryKeyT, SecondaryKeyT, ValueT> Coder<ValueT> getValueCoder(
      Coder<KV<PrimaryKeyT, Iterable<KV<SecondaryKeyT, ValueT>>>> inputCoder) {
    return getSecondaryKeyValueCoder(inputCoder).getValueCoder();
  }

  private static <T> T elementOf(Coder<T> coder, byte[] bytes) throws CoderException {

View on GitHub (pinned to 12126d8942)