apache/beam · error · IllegalArgumentException

Key coder must be deterministic

Error message

Key coder %s must be deterministic

What it means

Beam requires key coders to be deterministic because keyed data may be shuffled/grouped and must decode identically byte-for-byte. Watch.Growth verifies the output key coder determinism at expansion; if outputKeyCoder.verifyDeterministic() throws Coder.NonDeterministicException, Beam rethrows it as an IllegalArgumentException with this message.

Solutions

  1. Use a deterministic key type (String, Long, byte[]) for the output key
  2. Provide a deterministic custom coder via withOutputKeyCoder() (e.g. an Avro or protobuf-based coder)
  3. Implement Coder.verifyDeterministic() with no exception for a coder you know is stable
  4. Canonicalize keys (e.g. sorted-field serialization) before emitting them

Example fix

// before
Watch.growthOf(pollFn).withOutputKeyCoder(SerializableCoder.of(MyPojoKey.class)); // non-deterministic
// after
Watch.growthOf(pollFn).withOutputKeyCoder(StringUtf8Coder.of()); // key = MyPojoKey#canonicalString()
Defensive patterns

Strategy: validation

Validate before calling

try {
  coder.verifyDeterministic();
} catch (Coder.NonDeterministicException e) {
  throw new IllegalStateException("Key coder must be deterministic: " + coder, e);
}

Try / catch

try { pc = input.apply(growth); } catch (IllegalArgumentException e) { if (e.getMessage().contains("must be deterministic")) { /* swap key coder */ } }

Prevention

When it happens

Trigger: Using Watch.growthOf with an output key function whose KeyT coder is non-deterministic — e.g. a key type using SerializableCoder over a class without stable serialization (HashMap, POJO with unordered fields), or Avro/JSON coders with map fields.

Common situations: Choosing POJO or map-based key types for Watch output keys; switching a key type from String to a struct-like class; Beam upgrades tightening determinism checks.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/5279f2f0b7c4ec0d. Report an issue: GitHub.

Appendix: source

Thrown at sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/Watch.java:819

        outputKeyFn = (SerializableFunction) SerializableFunctions.identity();
      } else {
        if (outputKeyCoder == null) {
          // If a coder was not specified explicitly, infer it from the OutputT type parameter
          // of the output key fn.
          TypeDescriptor<KeyT> keyT = TypeDescriptors.outputOf(getOutputKeyFn());
          try {
            outputKeyCoder = input.getPipeline().getCoderRegistry().getCoder(keyT);
          } catch (CannotProvideCoderException e) {
            throw new RuntimeException(
                "Unable to infer coder for KeyT ("
                    + keyT
                    + "). Specify it explicitly using withOutputKeyCoder().");
          }
        }
        try {
          outputKeyCoder.verifyDeterministic();
        } catch (Coder.NonDeterministicException e) {
          throw new IllegalArgumentException(
              "Key coder " + outputKeyCoder + " must be deterministic");
        }
      }

      PCollection<KV<InputT, List<TimestampedValue<OutputT>>>> polledPc =
          input
              .apply(
                  ParDo.of(new WatchGrowthFn<>(this, outputCoder, outputKeyFn, outputKeyCoder))
                      .withSideInputs(getPollFn().getRequirements().getSideInputs()))
              .setCoder(
                  KvCoder.of(
                      input.getCoder(),
                      ListCoder.of(TimestampedValue.TimestampedValueCoder.of(outputCoder))));
      return polledPc
          .apply(ParDo.of(new PollResultSplitFn<>()))
          .setCoder(KvCoder.of(input.getCoder(), outputCoder));
    }
  }

View on GitHub (pinned to 12126d8942)