apache/beam · error · IllegalStateException

the keyCoder of a GroupByEncryptedKey must be deterministic

Error message

the keyCoder of a GroupByEncryptedKey must be deterministic

What it means

GroupByEncryptedKey groups elements by an HMAC-derived, encrypted key, so the underlying key coder must be deterministic: the same key value must always encode to the same bytes, otherwise keys would encrypt to different values and fail to group. The transform calls keyCoder.verifyDeterministic() during expand and throws IllegalStateException when the coder reports itself non-deterministic.

Source

Thrown at sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/GroupByEncryptedKey.java:113

              PCollection<KV<byte[], KV<byte[], byte[]>>>,
              PCollection<KV<byte[], Iterable<KV<byte[], byte[]>>>>>
          gbk) {
    return new GroupByEncryptedKey<>(hmacKey, gbk);
  }

  @Override
  public PCollection<KV<K, Iterable<V>>> expand(PCollection<KV<K, V>> input) {
    Coder<KV<K, V>> inputCoder = input.getCoder();
    if (!(inputCoder instanceof KvCoder)) {
      throw new IllegalStateException("GroupByEncryptedKey requires its input to use KvCoder");
    }
    KvCoder<K, V> inputKvCoder = (KvCoder<K, V>) inputCoder;
    Coder<K> keyCoder = inputKvCoder.getKeyCoder();

    try {
      keyCoder.verifyDeterministic();
    } catch (NonDeterministicException e) {
      throw new IllegalStateException(
          "the keyCoder of a GroupByEncryptedKey must be deterministic", e);
    }

    Coder<V> valueCoder = inputKvCoder.getValueCoder();

    PCollection<KV<byte[], Iterable<KV<byte[], byte[]>>>> grouped =
        input
            .apply(
                "EncryptMessage",
                ParDo.of(new EncryptMessage<>(this.hmacKey, keyCoder, valueCoder)))
            .apply(this.gbk);

    return grouped
        .apply("DecryptMessage", ParDo.of(new DecryptMessage<>(this.hmacKey, keyCoder, valueCoder)))
        .setCoder(KvCoder.of(keyCoder, IterableCoder.of(valueCoder)));
  }

  /**

View on GitHub (pinned to 12126d8942)

Solutions

1. Make the key coder deterministic (sort map/set entries, canonicalize numeric formats) so verifyDeterministic() passes
2. Use a canonical key type such as String or Long with a deterministic coder
3. If the existing coder is correct but non-canonical, wrap values into a canonical representation before grouping
4. Explicitly set a deterministic key coder via input.setCoder(KvCoder.of(deterministicKeyCoder, valueCoder)) before applying the transform

Example fix

// before
PCollection<KV<MyKey, V>> input = ...; // MyKeyCoder is non-deterministic
input.apply(GroupByEncryptedKey.of(hmacKey));
// after
input.apply(MapElements.into(typeDescriptorOfKV()).via(kv -> KV.of(kv.getKey().canonicalize(), kv.getValue())))
     .apply(GroupByEncryptedKey.of(hmacKey));

if (keyCoder instanceof StructuredCoder) {
  try { keyCoder.verifyDeterministic(); }
  catch (Coder.NonDeterministicException e) { throw new IllegalStateException("Replace key coder: " + keyCoder, e); }
}

static <K,V> boolean hasDeterministicKeyCoder(PCollection<KV<K,V>> pc) {
  return pc.getCoder() instanceof KvCoder<?, ?>
      && ((KvCoder<?, ?>) pc.getCoder()).getKeyCoder() instanceof StructuredCoder;
}

try {
  input.apply(GroupByEncryptedKey.of(hmacKey));
} catch (IllegalStateException e) {
  if (e.getMessage().contains("must be deterministic")) { /* switch key coder / canonicalize keys */ }
  else throw e;
}

When it happens

Trigger: Applying GroupByEncryptedKey to a PCollection<KV<K,V>> whose KvCoder key coder overrides verifyDeterministic to throw NonDeterministicException — e.g. a custom coder that serializes unordered Maps/Sets, double values, or non-canonical encodings.

Common situations: Custom key types with hand-written coders that don't canonicalize field order; keys containing floating-point NaN or unordered collections; migrating a pipeline from a different grouping transform that tolerated non-deterministic coders.

Understand the failure class

Background: Schema validation failed / invalid input schema: payload rejected because its shape doesn't match the expected schema — this error's family across 28 libraries.

Related errors

• the keyCoder of a GroupByKey must be deterministic
• Window coders must be deterministic.
• the secondary key coder of SortValues must be deterministic
• DatadogWriteError can hold arbitrary instances, which may be
• SplunkEvent can hold arbitrary instances, which may be non-d

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/e2192db085165c84. Report an issue: GitHub.