apache/beam · error · IllegalStateException

the GroupByKey requires its output coder to be %s but found

Error message

the GroupByKey requires its output coder to be %s but found %s.

What it means

During validate(), GroupByKey recomputes the KvCoder it expects its output PCollection to carry (KvCoder<K, Iterable<V>> derived from the input coder) and compares it with the coder actually set on the output. A mismatch means something changed the output coder incorrectly, which would break downstream coders/serialization, so Beam throws IllegalStateException.

Source

Thrown at sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/GroupByKey.java:209

                  + "https://s.apache.org/finishing-triggers-drop-data "
                  + "for details.",
              windowingStrategy.getTrigger()));
    }
  }

  @Override
  public void validate(
      @Nullable PipelineOptions options,
      Map<TupleTag<?>, PCollection<?>> inputs,
      Map<TupleTag<?>, PCollection<?>> outputs) {
    PCollection<?> input = Iterables.getOnlyElement(inputs.values());
    KvCoder<K, V> inputCoder = getInputKvCoder(input.getCoder());

    // Ensure that the output coder key and value types aren't different.
    Coder<?> outputCoder = Iterables.getOnlyElement(outputs.values()).getCoder();
    KvCoder<?, ?> expectedOutputCoder = getOutputKvCoder(inputCoder);
    if (!expectedOutputCoder.equals(outputCoder)) {
      throw new IllegalStateException(
          String.format(
              "the GroupByKey requires its output coder to be %s but found %s.",
              expectedOutputCoder, outputCoder));
    }
  }

  // Note that Never trigger finishes *at* GC time so it is OK, and
  // AfterWatermark.fromEndOfWindow() finishes at end-of-window time so it is
  // OK if there is no allowed lateness.
  private static boolean triggerIsSafe(WindowingStrategy<?, ?> windowingStrategy) {
    if (!windowingStrategy.getTrigger().mayFinish()) {
      return true;
    }

    if (windowingStrategy.getTrigger() instanceof NeverTrigger) {
      return true;
    }

View on GitHub (pinned to 12126d8942)

Solutions

1. Remove any explicit setCoder() on GroupByKey's output and let Beam infer KvCoder<K, Iterable<V>>
2. If a custom coder is needed, change the input coder so getOutputKvCoder derives the desired output coder
3. Ensure any coder overrides registered pipeline-wide match exactly KvCoder.of(inputKeyCoder, IterableCoder.of(inputValueCoder))
4. Compare the two coders in the message: align the found coder's structural types (key coder, iterable value coder) with the expected one

Example fix

// before
PCollection<KV<K, Iterable<V>>> grouped = input.apply(GroupByKey.create());
grouped.setCoder(KvCoder.of(keyCoder, ListCoder.of(valueCoder))); // wrong shape
// after
PCollection<KV<K, Iterable<V>>> grouped = input.apply(GroupByKey.create());
// no explicit setCoder; Beam derives KvCoder<K, IterableCoder<V>> from the input coder

KvCoder<?, ?> expected = KvCoder.of(
    ((KvCoder<?, ?>) input.getCoder()).getKeyCoder(),
    IterableCoder.of(((KvCoder<?, ?>) input.getCoder()).getValueCoder()));
if (!expected.equals(grouped.getCoder())) { /* remove or fix setCoder */ }

static <K,V> boolean outputCoderIsValid(PCollection<KV<K, Iterable<V>>> out) {
  return out.getCoder() instanceof KvCoder<?, ?>
      && ((KvCoder<?, ?>) out.getCoder()).getValueCoder() instanceof IterableCoder<?>;
}

try {
  pipeline.run();
} catch (IllegalStateException e) {
  if (e.getMessage() != null && e.getMessage().contains("requires its output coder")) { /* drop the manual setCoder */ }
  else throw e;
}

When it happens

Trigger: A user or intermediate transform calls output.setCoder(...) on GroupByKey's output with a coder different from getOutputKvCoder(inputCoder) — e.g. set a KvCoder with a different value coder after applying GroupByKey, or an output-coder-override registry supplies the wrong coder.

Common situations: Manually overriding output coders to 'optimize' serialization; wrapping GroupByKey in a composite that resets coders; custom pipeline-level coder registration that substitutes a coder for KV<K, Iterable<V>>.

Understand the failure class

Background: Type mismatch errors: IllegalArgumentException, TypeError and type guards across 150 open-source libraries — this error's family across 150 libraries.

Related errors

• GroupByEncryptedKey requires its input to use KvCoder
• the keyCoder of a GroupByKey must be deterministic
• GroupByKey requires its input to use KvCoder
• cannot encode a null Integer
• cannot encode a null Long

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/cdf2e4f25a3ed9a4. Report an issue: GitHub.