{"record":{"id":"7ec10cc53dff82c7","repo":"apache/beam","slug":"the-secondary-key-coder-of-sortvalues-must-be-deterministic","errorCode":null,"errorMessage":"the secondary key coder of SortValues must be deterministic","messagePattern":"the secondary key coder of SortValues must be deterministic","errorType":"exception","errorClass":"IllegalStateException","httpStatus":null,"severity":"error","filePath":"sdks/java/extensions/sorter/src/main/java/org/apache/beam/sdk/extensions/sorter/SortValues.java","lineNumber":86,"sourceCode":"   * @param <SecondaryKeyT> the type of the secondary (sort) keys of the input and output {@code\n   *     PCollection}s\n   * @param <ValueT> the type of the values of the input and output {@code PCollection}s\n   */\n  public static <PrimaryKeyT, SecondaryKeyT, ValueT>\n      SortValues<PrimaryKeyT, SecondaryKeyT, ValueT> create(\n          BufferedExternalSorter.Options sorterOptions) {\n    return new SortValues<>(sorterOptions);\n  }\n\n  @Override\n  public PCollection<KV<PrimaryKeyT, Iterable<KV<SecondaryKeyT, ValueT>>>> expand(\n      PCollection<KV<PrimaryKeyT, Iterable<KV<SecondaryKeyT, ValueT>>>> input) {\n\n    Coder<SecondaryKeyT> secondaryKeyCoder = getSecondaryKeyCoder(input.getCoder());\n    try {\n      secondaryKeyCoder.verifyDeterministic();\n    } catch (Coder.NonDeterministicException e) {\n      throw new IllegalStateException(\n          \"the secondary key coder of SortValues must be deterministic\", e);\n    }\n\n    return input\n        .apply(\n            ParDo.of(\n                new SortValuesDoFn<>(\n                    sorterOptions, secondaryKeyCoder, getValueCoder(input.getCoder()))))\n        .setCoder(input.getCoder());\n  }\n\n  /** Retrieves the {@link Coder} for the secondary key-value pairs. */\n  @SuppressWarnings(\"unchecked\")\n  private static <PrimaryKeyT, SecondaryKeyT, ValueT>\n      KvCoder<SecondaryKeyT, ValueT> getSecondaryKeyValueCoder(\n          Coder<KV<PrimaryKeyT, Iterable<KV<SecondaryKeyT, ValueT>>>> inputCoder) {\n    if (!(inputCoder instanceof KvCoder)) {\n      throw new IllegalStateException(\"SortValues requires its input to use KvCoder\");","sourceCodeStart":68,"sourceCodeEnd":104,"githubUrl":"https://github.com/apache/beam/blob/12126d8942aaf848030c478b4c6a28c6af861c66/sdks/java/extensions/sorter/src/main/java/org/apache/beam/sdk/extensions/sorter/SortValues.java#L68-L104","documentation":"SortValues sorts per-key data by serializing secondary keys, and Beam's sorting requires a total, stable order across workers. If the secondary key Coder is non-deterministic (encoding the same object yields different bytes), equal keys could sort inconsistently, so expand() calls verifyDeterministic() and wraps any NonDeterministicException in an IllegalStateException.","triggerScenarios":"Applying SortValues.perKey() to a PCollection<KV<P, Iterable<KV<S, V>>>> where the KvCoder's secondary-key coder is non-deterministic — most commonly a custom coder for a POJO/map/JSONObject, or Coder of type Map/Struct whose field iteration order varies.","commonSituations":"Using a JsonCoder, AvroGenericRecordCoder with non-deterministic map fields, or a hand-written coder without implementing verifyDeterministic; keys containing HashMaps or unordered sets.","solutions":["Make the secondary key coder deterministic: encode fields in a fixed order (e.g., sort map keys before writing) in a custom coder and implement verifyDeterministic as a no-op/pass","Convert keys to a deterministic representation before sorting (e.g., a String or protobuf with canonical encoding)","Use a deterministic built-in coder (StringUtf8Coder, BigIntegerCoder, etc.) for the secondary key","If the coder is actually deterministic, override verifyDeterministic() in the custom coder to state that explicitly"],"exampleFix":"// before\n// custom coder writes map entries in HashMap iteration order -> non-deterministic\n// after\n// custom coder: sorted entries before encoding\nentries.sort(Comparator.comparing(e -> e.getKey()));\n@Override public void verifyDeterministic() throws NonDeterministicException {}","handlingStrategy":"validation","validationCode":"try { secondaryKeyCoder.verifyDeterministic(); } catch (Coder.NonDeterministicException e) { throw new IllegalStateException(\"Replace non-deterministic secondary key coder\", e); }","typeGuard":"static <T> boolean deterministic(Coder<T> c) { try { c.verifyDeterministic(); return true; } catch (Coder.NonDeterministicException e) { return false; } }","tryCatchPattern":"try { return SortValues.perKey(); } catch (IllegalStateException e) { if (e.getMessage().contains(\"deterministic\")) { /* swap coder and retry */ } throw e; }","preventionTips":["Prefer built-in deterministic coders (StringUtf8Coder, VarLongCoder) for sort keys","Implement verifyDeterministic() explicitly in custom coders with deterministic encoding","Avoid HashMap/set-based keys in sorted pipelines","Run verifyDeterministic at pipeline-construction time in tests"],"tags":["java","apache-beam","sorter","determinism","coder"],"backgroundTag":"schema-validation-failed","analyzedSha":"12126d8942aaf848030c478b4c6a28c6af861c66","analyzedAt":"2026-09-13T01:50:10.254Z","contentChangedAt":"2026-09-13T01:50:10.254Z","schemaVersion":2},"datasetVersion":"2026-09-14T16:17:12.679Z"}