{"record":{"id":"be67857b35228447","repo":"apache/beam","slug":"the-keycoder-of-a-groupbykey-must-be-deterministic","errorCode":null,"errorMessage":"the keyCoder of a GroupByKey must be deterministic","messagePattern":"the keyCoder of a GroupByKey must be deterministic","errorType":"exception","errorClass":"IllegalStateException","httpStatus":null,"severity":"error","filePath":"sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/GroupByKey.java","lineNumber":264,"sourceCode":"  public WindowingStrategy<?, ?> updateWindowingStrategy(WindowingStrategy<?, ?> inputStrategy) {\n    // If the WindowFn was merging, set the bit to indicate it is already merged.\n    // Switch to the continuation trigger associated with the current trigger.\n    return inputStrategy\n        .withAlreadyMerged(!inputStrategy.getWindowFn().isNonMerging())\n        .withTrigger(inputStrategy.getTrigger().getContinuationTrigger());\n  }\n\n  @Override\n  public PCollection<KV<K, Iterable<V>>> expand(PCollection<KV<K, V>> input) {\n    applicableTo(input);\n\n    // Verify that the input Coder<KV<K, V>> is a KvCoder<K, V>, and that\n    // the key coder is deterministic.\n    Coder<K> keyCoder = getKeyCoder(input.getCoder());\n    try {\n      keyCoder.verifyDeterministic();\n    } catch (NonDeterministicException e) {\n      throw new IllegalStateException(\"the keyCoder of a GroupByKey must be deterministic\", e);\n    }\n\n    PipelineOptions options = input.getPipeline().getOptions();\n    String gbekOveride = options.getGbek();\n    if (!this.insideGBEK && gbekOveride != null && !gbekOveride.trim().isEmpty()) {\n      this.surroundsGBEK = true;\n      Secret hmacSecret = Secret.parseSecretOption(gbekOveride);\n      GroupByKey<byte[], KV<byte[], byte[]>> gbk = GroupByKey.create();\n      if (this.fewKeys) {\n        gbk = GroupByKey.createWithFewKeys();\n      }\n      gbk.setInsideGBEK();\n      GroupByEncryptedKey<K, V> gbek = GroupByEncryptedKey.createWithCustomGbk(hmacSecret, gbk);\n      return input.apply(gbek);\n    }\n\n    // This primitive operation groups by the combination of key and window,\n    // merging windows as needed, using the windows assigned to the","sourceCodeStart":246,"sourceCodeEnd":282,"githubUrl":"https://github.com/apache/beam/blob/12126d8942aaf848030c478b4c6a28c6af861c66/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/GroupByKey.java#L246-L282","documentation":"Like all grouping transforms, GroupByKey needs key encodings to be canonical because encoded key bytes decide grouping. In expand() it calls keyCoder.verifyDeterministic() and wraps a NonDeterministicException into an IllegalStateException with this message.","triggerScenarios":"Applying GroupByKey (or a transform that contains it, e.g. Combine.perKey when not inside GBEK) to a PCollection<KV<K,V>> whose key coder fails verifyDeterministic — typical for coders of types containing doubles (NaN), unordered collections, or arbitrary-precision/serialization-order-dependent data.","commonSituations":"Grouping on Avro/Protobuf records with map fields without a deterministic coder; grouping on TableRow/Map keys; custom coders that didn't implement canonical serialization; upgrading Beam where a previously lenient path now surfaces the check.","solutions":["Use a deterministic key type (String, Long, byte[] with fixed encoding) or canonicalize the key before grouping","Implement/fix the custom coder so it produces identical bytes for equal keys and does not throw in verifyDeterministic","Sort or flatten unordered fields in a deterministic order before encoding, then group on that representation","Set a deterministic KvCoder explicitly with input.setCoder(...) before GroupByKey","Note: when the GroupByKey is wrapped by GroupByEncryptedKey/GBEK mode, the same determinism requirement applies to the pre-encryption key coder"],"exampleFix":"// before\nPCollection<KV<Map<String,Integer>, V>> grouped = input.apply(GroupByKey.create()); // MapCoder non-deterministic\n// after\nPCollection<KV<String, V>> withCanonicalKeys = input\n    .apply(MapElements.into(strings()).via(kv -> KV.of(canonicalEncode(kv.getKey()), kv.getValue())))\n    .apply(GroupByKey.create());","handlingStrategy":"validation","validationCode":"KvCoder<K,V> kv = (KvCoder<K,V>) input.getCoder();\ntry { kv.getKeyCoder().verifyDeterministic(); }\ncatch (Coder.NonDeterministicException e) {\n  input = input.apply(MapElements.into(kvType()).via(kv2 -> KV.of(canonical(kv2.getKey()), kv2.getValue())));\n}","typeGuard":"static <K,V> boolean groupSafe(PCollection<KV<K,V>> pc) {\n  return pc.getCoder() instanceof KvCoder<?, ?>\n      && ((KvCoder<?, ?>) pc.getCoder()).getKeyCoder() instanceof DeterministicKeyCoderMarker;\n}","tryCatchPattern":"try {\n  grouped = input.apply(GroupByKey.create());\n} catch (IllegalStateException e) {\n  if (e.getMessage().contains(\"keyCoder of a GroupByKey must be deterministic\")) { /* canonicalize keys */ }\n  else throw e;\n}","preventionTips":["Unit-test custom key coders with verifyDeterministic()","Ban non-deterministic field types in grouping-key classes","Prefer primitive/UTF-8 keys for grouping"],"tags":["apache-beam","java","coder","determinism","groupbykey"],"backgroundTag":"schema-validation-failed","analyzedSha":"12126d8942aaf848030c478b4c6a28c6af861c66","analyzedAt":"2026-09-13T01:50:10.254Z","contentChangedAt":"2026-09-13T01:50:10.254Z","schemaVersion":2},"datasetVersion":"2026-09-14T16:17:12.679Z"}