{"record":{"id":"e2192db085165c84","repo":"apache/beam","slug":"the-keycoder-of-a-groupbyencryptedkey-must-be-deterministic","errorCode":null,"errorMessage":"the keyCoder of a GroupByEncryptedKey must be deterministic","messagePattern":"the keyCoder of a GroupByEncryptedKey must be deterministic","errorType":"exception","errorClass":"IllegalStateException","httpStatus":null,"severity":"error","filePath":"sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/GroupByEncryptedKey.java","lineNumber":113,"sourceCode":"              PCollection<KV<byte[], KV<byte[], byte[]>>>,\n              PCollection<KV<byte[], Iterable<KV<byte[], byte[]>>>>>\n          gbk) {\n    return new GroupByEncryptedKey<>(hmacKey, gbk);\n  }\n\n  @Override\n  public PCollection<KV<K, Iterable<V>>> expand(PCollection<KV<K, V>> input) {\n    Coder<KV<K, V>> inputCoder = input.getCoder();\n    if (!(inputCoder instanceof KvCoder)) {\n      throw new IllegalStateException(\"GroupByEncryptedKey requires its input to use KvCoder\");\n    }\n    KvCoder<K, V> inputKvCoder = (KvCoder<K, V>) inputCoder;\n    Coder<K> keyCoder = inputKvCoder.getKeyCoder();\n\n    try {\n      keyCoder.verifyDeterministic();\n    } catch (NonDeterministicException e) {\n      throw new IllegalStateException(\n          \"the keyCoder of a GroupByEncryptedKey must be deterministic\", e);\n    }\n\n    Coder<V> valueCoder = inputKvCoder.getValueCoder();\n\n    PCollection<KV<byte[], Iterable<KV<byte[], byte[]>>>> grouped =\n        input\n            .apply(\n                \"EncryptMessage\",\n                ParDo.of(new EncryptMessage<>(this.hmacKey, keyCoder, valueCoder)))\n            .apply(this.gbk);\n\n    return grouped\n        .apply(\"DecryptMessage\", ParDo.of(new DecryptMessage<>(this.hmacKey, keyCoder, valueCoder)))\n        .setCoder(KvCoder.of(keyCoder, IterableCoder.of(valueCoder)));\n  }\n\n  /**","sourceCodeStart":95,"sourceCodeEnd":131,"githubUrl":"https://github.com/apache/beam/blob/12126d8942aaf848030c478b4c6a28c6af861c66/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/GroupByEncryptedKey.java#L95-L131","documentation":"GroupByEncryptedKey groups elements by an HMAC-derived, encrypted key, so the underlying key coder must be deterministic: the same key value must always encode to the same bytes, otherwise keys would encrypt to different values and fail to group. The transform calls keyCoder.verifyDeterministic() during expand and throws IllegalStateException when the coder reports itself non-deterministic.","triggerScenarios":"Applying GroupByEncryptedKey to a PCollection<KV<K,V>> whose KvCoder key coder overrides verifyDeterministic to throw NonDeterministicException — e.g. a custom coder that serializes unordered Maps/Sets, double values, or non-canonical encodings.","commonSituations":"Custom key types with hand-written coders that don't canonicalize field order; keys containing floating-point NaN or unordered collections; migrating a pipeline from a different grouping transform that tolerated non-deterministic coders.","solutions":["Make the key coder deterministic (sort map/set entries, canonicalize numeric formats) so verifyDeterministic() passes","Use a canonical key type such as String or Long with a deterministic coder","If the existing coder is correct but non-canonical, wrap values into a canonical representation before grouping","Explicitly set a deterministic key coder via input.setCoder(KvCoder.of(deterministicKeyCoder, valueCoder)) before applying the transform"],"exampleFix":"// before\nPCollection<KV<MyKey, V>> input = ...; // MyKeyCoder is non-deterministic\ninput.apply(GroupByEncryptedKey.of(hmacKey));\n// after\ninput.apply(MapElements.into(typeDescriptorOfKV()).via(kv -> KV.of(kv.getKey().canonicalize(), kv.getValue())))\n     .apply(GroupByEncryptedKey.of(hmacKey));","handlingStrategy":"validation","validationCode":"if (keyCoder instanceof StructuredCoder) {\n  try { keyCoder.verifyDeterministic(); }\n  catch (Coder.NonDeterministicException e) { throw new IllegalStateException(\"Replace key coder: \" + keyCoder, e); }\n}","typeGuard":"static <K,V> boolean hasDeterministicKeyCoder(PCollection<KV<K,V>> pc) {\n  return pc.getCoder() instanceof KvCoder<?, ?>\n      && ((KvCoder<?, ?>) pc.getCoder()).getKeyCoder() instanceof StructuredCoder;\n}","tryCatchPattern":"try {\n  input.apply(GroupByEncryptedKey.of(hmacKey));\n} catch (IllegalStateException e) {\n  if (e.getMessage().contains(\"must be deterministic\")) { /* switch key coder / canonicalize keys */ }\n  else throw e;\n}","preventionTips":["Only group on keys with canonical, deterministic encodings (String, Long, fixed-width bytes)","Always call verifyDeterministic() on custom coders in unit tests","Avoid Map/Set/double fields in grouping keys"],"tags":["apache-beam","java","coder","determinism","grouping"],"backgroundTag":"schema-validation-failed","analyzedSha":"12126d8942aaf848030c478b4c6a28c6af861c66","analyzedAt":"2026-09-13T01:50:10.254Z","contentChangedAt":"2026-09-13T01:50:10.254Z","schemaVersion":2},"datasetVersion":"2026-09-14T16:17:12.679Z"}