apache/beam · error · IllegalStateException

the keyCoder of a GroupByKey must be deterministic

Error message

the keyCoder of a GroupByKey must be deterministic

What it means

Like all grouping transforms, GroupByKey needs key encodings to be canonical because encoded key bytes decide grouping. In expand() it calls keyCoder.verifyDeterministic() and wraps a NonDeterministicException into an IllegalStateException with this message.

Source

Thrown at sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/GroupByKey.java:264

  public WindowingStrategy<?, ?> updateWindowingStrategy(WindowingStrategy<?, ?> inputStrategy) {
    // If the WindowFn was merging, set the bit to indicate it is already merged.
    // Switch to the continuation trigger associated with the current trigger.
    return inputStrategy
        .withAlreadyMerged(!inputStrategy.getWindowFn().isNonMerging())
        .withTrigger(inputStrategy.getTrigger().getContinuationTrigger());
  }

  @Override
  public PCollection<KV<K, Iterable<V>>> expand(PCollection<KV<K, V>> input) {
    applicableTo(input);

    // Verify that the input Coder<KV<K, V>> is a KvCoder<K, V>, and that
    // the key coder is deterministic.
    Coder<K> keyCoder = getKeyCoder(input.getCoder());
    try {
      keyCoder.verifyDeterministic();
    } catch (NonDeterministicException e) {
      throw new IllegalStateException("the keyCoder of a GroupByKey must be deterministic", e);
    }

    PipelineOptions options = input.getPipeline().getOptions();
    String gbekOveride = options.getGbek();
    if (!this.insideGBEK && gbekOveride != null && !gbekOveride.trim().isEmpty()) {
      this.surroundsGBEK = true;
      Secret hmacSecret = Secret.parseSecretOption(gbekOveride);
      GroupByKey<byte[], KV<byte[], byte[]>> gbk = GroupByKey.create();
      if (this.fewKeys) {
        gbk = GroupByKey.createWithFewKeys();
      }
      gbk.setInsideGBEK();
      GroupByEncryptedKey<K, V> gbek = GroupByEncryptedKey.createWithCustomGbk(hmacSecret, gbk);
      return input.apply(gbek);
    }

    // This primitive operation groups by the combination of key and window,
    // merging windows as needed, using the windows assigned to the

View on GitHub (pinned to 12126d8942)

Solutions

1. Use a deterministic key type (String, Long, byte[] with fixed encoding) or canonicalize the key before grouping
2. Implement/fix the custom coder so it produces identical bytes for equal keys and does not throw in verifyDeterministic
3. Sort or flatten unordered fields in a deterministic order before encoding, then group on that representation
4. Set a deterministic KvCoder explicitly with input.setCoder(...) before GroupByKey
5. Note: when the GroupByKey is wrapped by GroupByEncryptedKey/GBEK mode, the same determinism requirement applies to the pre-encryption key coder

Example fix

// before
PCollection<KV<Map<String,Integer>, V>> grouped = input.apply(GroupByKey.create()); // MapCoder non-deterministic
// after
PCollection<KV<String, V>> withCanonicalKeys = input
    .apply(MapElements.into(strings()).via(kv -> KV.of(canonicalEncode(kv.getKey()), kv.getValue())))
    .apply(GroupByKey.create());

KvCoder<K,V> kv = (KvCoder<K,V>) input.getCoder();
try { kv.getKeyCoder().verifyDeterministic(); }
catch (Coder.NonDeterministicException e) {
  input = input.apply(MapElements.into(kvType()).via(kv2 -> KV.of(canonical(kv2.getKey()), kv2.getValue())));
}

static <K,V> boolean groupSafe(PCollection<KV<K,V>> pc) {
  return pc.getCoder() instanceof KvCoder<?, ?>
      && ((KvCoder<?, ?>) pc.getCoder()).getKeyCoder() instanceof DeterministicKeyCoderMarker;
}

try {
  grouped = input.apply(GroupByKey.create());
} catch (IllegalStateException e) {
  if (e.getMessage().contains("keyCoder of a GroupByKey must be deterministic")) { /* canonicalize keys */ }
  else throw e;
}

When it happens

Trigger: Applying GroupByKey (or a transform that contains it, e.g. Combine.perKey when not inside GBEK) to a PCollection<KV<K,V>> whose key coder fails verifyDeterministic — typical for coders of types containing doubles (NaN), unordered collections, or arbitrary-precision/serialization-order-dependent data.

Common situations: Grouping on Avro/Protobuf records with map fields without a deterministic coder; grouping on TableRow/Map keys; custom coders that didn't implement canonical serialization; upgrading Beam where a previously lenient path now surfaces the check.

Understand the failure class

Background: Schema validation failed / invalid input schema: payload rejected because its shape doesn't match the expected schema — this error's family across 28 libraries.

Related errors

• the keyCoder of a GroupByEncryptedKey must be deterministic
• the GroupByKey requires its output coder to be %s but found
• GroupByKey requires its input to use KvCoder
• Window coders must be deterministic.
• the secondary key coder of SortValues must be deterministic

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/be67857b35228447. Report an issue: GitHub.