apache/beam · error · IllegalStateException

SortValues requires its input to use KvCoder

Error message

SortValues requires its input to use KvCoder

What it means

SortValues expects its input PCollection to be keyed data whose coder is a KvCoder<P, Iterable<...>> so it can statically extract the secondary key/value coders. If the input coder is not a KvCoder (e.g., an inferred coder like a custom or StructuralCoder), getSecondaryKeyValueCoder throws IllegalStateException because it cannot decompose the element type.

Source

Thrown at sdks/java/extensions/sorter/src/main/java/org/apache/beam/sdk/extensions/sorter/SortValues.java:104

      throw new IllegalStateException(
          "the secondary key coder of SortValues must be deterministic", e);
    }

    return input
        .apply(
            ParDo.of(
                new SortValuesDoFn<>(
                    sorterOptions, secondaryKeyCoder, getValueCoder(input.getCoder()))))
        .setCoder(input.getCoder());
  }

  /** Retrieves the {@link Coder} for the secondary key-value pairs. */
  @SuppressWarnings("unchecked")
  private static <PrimaryKeyT, SecondaryKeyT, ValueT>
      KvCoder<SecondaryKeyT, ValueT> getSecondaryKeyValueCoder(
          Coder<KV<PrimaryKeyT, Iterable<KV<SecondaryKeyT, ValueT>>>> inputCoder) {
    if (!(inputCoder instanceof KvCoder)) {
      throw new IllegalStateException("SortValues requires its input to use KvCoder");
    }
    @SuppressWarnings("unchecked")
    KvCoder<PrimaryKeyT, Iterable<KV<SecondaryKeyT, ValueT>>> kvCoder =
        (KvCoder<PrimaryKeyT, Iterable<KV<SecondaryKeyT, ValueT>>>) inputCoder;

    if (!(kvCoder.getValueCoder() instanceof IterableCoder)) {
      throw new IllegalStateException(
          "SortValues requires the values be encoded with IterableCoder");
    }
    IterableCoder<KV<SecondaryKeyT, ValueT>> iterableCoder =
        (IterableCoder<KV<SecondaryKeyT, ValueT>>) kvCoder.getValueCoder();

    if (!(iterableCoder.getElemCoder() instanceof KvCoder)) {
      throw new IllegalStateException(
          "SortValues requires the secondary key-value pairs to use KvCoder");
    }
    return (KvCoder<SecondaryKeyT, ValueT>) iterableCoder.getElemCoder();
  }

View on GitHub (pinned to 12126d8942)

Solutions

1. Ensure the input uses KvCoder: pCollection.setCoder(KvCoder.of(primaryCoder, IterableCoder.of(KvCoder.of(secondaryKeyCoder, valueCoder))))
2. Build the input with keyed transforms (WithKeys, GroupByKey) so Beam infers KvCoder naturally
3. If using a custom coder, make it a KvCoder subclass exposing getKeyCoder/getValueCoder
4. Inspect the actual coder with input.getCoder() and log it to find where the wrong coder is assigned

Example fix

// before
input.setCoder(new MyKvLikeCoder<P, Iterable<KV<S,V>>>());
// after
input.setCoder(KvCoder.of(
    primaryCoder,
    IterableCoder.of(KvCoder.of(secondaryKeyCoder, valueCoder))));

if (!(input.getCoder() instanceof KvCoder)) { throw new IllegalStateException("SortValues input must use KvCoder; got " + input.getCoder()); }

static boolean hasKvCoder(PCollection<?> pc) { return pc.getCoder() instanceof KvCoder; }

try { SortValues.perKey(); } catch (IllegalStateException e) { if (e.getMessage().contains("KvCoder")) { input.setCoder(KvCoder.of(pkCoder, IterableCoder.of(KvCoder.of(skCoder, vCoder)))); } else { throw e; } }

When it happens

Trigger: Applying SortValues.perKey() to a PCollection<KV<P, Iterable<KV<S,V>>>> whose coder was set/inferred as something other than KvCoder — e.g., setCoder(myCustomKvLikeCoder), or input built through transforms that lose the KvCoder type information.

Common situations: Manually calling setCoder with a wrong coder; piping through a transform that re-infers coders; deserializing KV from a source that assigns a generic coder; test pipelines that construct PCollections without KvCoder.

Understand the failure class

Background: "is not a compatible type" / "cannot merge" errors: when a value's type doesn't match what the library requires — this error's family across 65 libraries.

Related errors

• SortValues requires the values be encoded with IterableCoder
• GroupByKey requires its input to use KvCoder
• the secondary key coder of SortValues must be deterministic
• cannot encode a null Integer
• cannot encode a null Long

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/cbc09393888c7a98. Report an issue: GitHub.