What causes "Hadoop Writable may be non-deterministic." in apache/beam?

Use an Avro/POJO/protobuf coder or beam codable type for keys instead of a Writable

apache/beam · error · NonDeterministicException

Hadoop Writable may be non-deterministic.

Error message

Hadoop Writable may be non-deterministic.

What it means

WritableCoder.verifyDeterministic always throws NonDeterministicException because Hadoop Writable serialization offers no guarantee of byte-for-byte deterministic output. Beam requires deterministic coders for operations like grouping keys and state; using a Writable in such positions is rejected up front.

Solutions

Use an Avro/POJO/protobuf coder or beam codable type for keys instead of a Writable
Wrap the Writable value in a deterministic container (e.g. encode to byte[] with a custom deterministic Coder) if you can guarantee determinism yourself
Keep Writables only as values, never as keys, in grouped operations

Example fix

// before
apply(GroupByKey.<Text, LongWritable>create());
// after
PCollection<KV<String, Long>> kv = pairs.apply(
    MapElements.into(typeOf(KV<String,Long>())).via(w -> KV.of(w.key, w.value.get())));
kv.apply(GroupByKey.create());

Defensive patterns

Strategy: fallback

Validate before calling

try {
  coder.verifyDeterministic();
  deterministic = true;
} catch (Coder.NonDeterministicException e) {
  deterministic = false;
}

Try / catch

try {
  coder.verifyDeterministic();
} catch (Coder.NonDeterministicException e) {
  // switch to a deterministic coder or convert keys to a codable type
}

Prevention

Never use Writable-typed keys in GroupByKey or stateful DoFns
Convert Writable values to Beam-native types (Avro/POJO) at pipeline boundaries
Reserve WritableCoder for value positions only

When it happens

Trigger: Using a WritableCoder-coded type as a GroupByKey key, state/timer key, or any Beam operation that calls verifyDeterministic.

Common situations: Pipelines migrating MapReduce code to Beam that reuse Writable types as keys, hitting the deterministic-coder requirement at graph validation time.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/c9b5efee7d33adf4. Report an issue: GitHub.

Appendix: source

Thrown at sdks/java/io/hadoop-common/src/main/java/org/apache/beam/sdk/io/hadoop/WritableCoder.java:104

      T t = type.getDeclaredConstructor().newInstance();
      t.readFields(new DataInputStream(inStream));
      return t;
    } catch (InstantiationException
        | IllegalAccessException
        | NoSuchMethodException
        | InvocationTargetException e) {
      throw new CoderException("unable to deserialize record", e);
    }
  }

  @Override
  public List<Coder<?>> getCoderArguments() {
    return Collections.emptyList();
  }

  @Override
  public void verifyDeterministic() throws NonDeterministicException {
    throw new NonDeterministicException(this, "Hadoop Writable may be non-deterministic.");
  }

  @Override
  public boolean equals(@Nullable Object other) {
    if (other == this) {
      return true;
    }
    if (!(other instanceof WritableCoder)) {
      return false;
    }
    WritableCoder<?> that = (WritableCoder<?>) other;
    return Objects.equals(this.type, that.type);
  }

  @Override
  public int hashCode() {
    return type.hashCode();
  }

View on GitHub (pinned to 12126d8942)