{"record":{"id":"c9b5efee7d33adf4","repo":"apache/beam","slug":"hadoop-writable-may-be-non-deterministic","errorCode":null,"errorMessage":"Hadoop Writable may be non-deterministic.","messagePattern":"Hadoop Writable may be non-deterministic\\.","errorType":"validation","errorClass":"NonDeterministicException","httpStatus":null,"severity":"error","filePath":"sdks/java/io/hadoop-common/src/main/java/org/apache/beam/sdk/io/hadoop/WritableCoder.java","lineNumber":104,"sourceCode":"      T t = type.getDeclaredConstructor().newInstance();\n      t.readFields(new DataInputStream(inStream));\n      return t;\n    } catch (InstantiationException\n        | IllegalAccessException\n        | NoSuchMethodException\n        | InvocationTargetException e) {\n      throw new CoderException(\"unable to deserialize record\", e);\n    }\n  }\n\n  @Override\n  public List<Coder<?>> getCoderArguments() {\n    return Collections.emptyList();\n  }\n\n  @Override\n  public void verifyDeterministic() throws NonDeterministicException {\n    throw new NonDeterministicException(this, \"Hadoop Writable may be non-deterministic.\");\n  }\n\n  @Override\n  public boolean equals(@Nullable Object other) {\n    if (other == this) {\n      return true;\n    }\n    if (!(other instanceof WritableCoder)) {\n      return false;\n    }\n    WritableCoder<?> that = (WritableCoder<?>) other;\n    return Objects.equals(this.type, that.type);\n  }\n\n  @Override\n  public int hashCode() {\n    return type.hashCode();\n  }","sourceCodeStart":86,"sourceCodeEnd":122,"githubUrl":"https://github.com/apache/beam/blob/12126d8942aaf848030c478b4c6a28c6af861c66/sdks/java/io/hadoop-common/src/main/java/org/apache/beam/sdk/io/hadoop/WritableCoder.java#L86-L122","documentation":"WritableCoder.verifyDeterministic always throws NonDeterministicException because Hadoop Writable serialization offers no guarantee of byte-for-byte deterministic output. Beam requires deterministic coders for operations like grouping keys and state; using a Writable in such positions is rejected up front.","triggerScenarios":"Using a WritableCoder-coded type as a GroupByKey key, state/timer key, or any Beam operation that calls verifyDeterministic.","commonSituations":"Pipelines migrating MapReduce code to Beam that reuse Writable types as keys, hitting the deterministic-coder requirement at graph validation time.","solutions":["Use an Avro/POJO/protobuf coder or beam codable type for keys instead of a Writable","Wrap the Writable value in a deterministic container (e.g. encode to byte[] with a custom deterministic Coder) if you can guarantee determinism yourself","Keep Writables only as values, never as keys, in grouped operations"],"exampleFix":"// before\napply(GroupByKey.<Text, LongWritable>create());\n// after\nPCollection<KV<String, Long>> kv = pairs.apply(\n    MapElements.into(typeOf(KV<String,Long>())).via(w -> KV.of(w.key, w.value.get())));\nkv.apply(GroupByKey.create());","handlingStrategy":"fallback","validationCode":"try {\n  coder.verifyDeterministic();\n  deterministic = true;\n} catch (Coder.NonDeterministicException e) {\n  deterministic = false;\n}","typeGuard":null,"tryCatchPattern":"try {\n  coder.verifyDeterministic();\n} catch (Coder.NonDeterministicException e) {\n  // switch to a deterministic coder or convert keys to a codable type\n}","preventionTips":["Never use Writable-typed keys in GroupByKey or stateful DoFns","Convert Writable values to Beam-native types (Avro/POJO) at pipeline boundaries","Reserve WritableCoder for value positions only"],"tags":["java","hadoop","writable","determinism","coder"],"backgroundTag":"nondeterministic-coder","analyzedSha":"12126d8942aaf848030c478b4c6a28c6af861c66","analyzedAt":"2026-09-13T01:50:10.254Z","contentChangedAt":"2026-09-13T01:50:10.254Z","schemaVersion":2},"datasetVersion":"2026-09-20T03:17:13.778Z"}