apache/beam · error · NonDeterministicException

CheckpointCoder uses Java Serialization, which may be…

Error message

CheckpointCoder uses Java Serialization, which may be non-deterministic.

What it means

CheckpointCoder (used by UnboundedReadFromBoundedSource to checkpoint UnboundedReader state) relies on Java serialization, which is not guaranteed deterministic across JVMs/runs. Its verifyDeterministic override unconditionally throws Coder.NonDeterministicException to signal this, since Beam requires deterministic coders for key operations like grouping and caching.

Solutions

Do not use CheckpointCoder in contexts requiring deterministic coders; restructure so checkpoint state isn't coder-validated
Provide a deterministic coder for the checkpoint state if the runner supports customization
If the check is incidental, avoid operations (like GroupByKey) on PCollections coded with CheckpointCoder

Example fix

// before
checkpointCoder.verifyDeterministic(); // always throws
// after: use a deterministic coder implementation for the state
Coder<MyState> coder = new MyDeterministicStateCoder();

Defensive patterns

Strategy: try-catch

Try / catch

try {
  coder.verifyDeterministic();
} catch (Coder.NonDeterministicException e) {
  // CheckpointCoder (or its element coder) is non-deterministic; do not use it
  // for GroupByKey/GBK-like operations requiring deterministic coders.
}

Prevention

Never use CheckpointCoder for PCollections that undergo key-based operations
Substitute a deterministic coder for checkpoint state when the runner requires one

When it happens

Trigger: Calling verifyDeterministic() on a CheckpointCoder instance — e.g. when the pipeline framework validates coder determinism during Apply witnesses, GBK, or coder verification of a checkpointed unbounded read.

Common situations: Runners or pipeline validators that enforce deterministic coders encountering an unbounded source checkpoint; debugging coder determinism of BoundedSource-based reads.

Understand the failure class

Background: UnsupportedOperationException and "is not supported" errors: when a library deliberately refuses a call — this error's family across 30 libraries.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/5bfac28e1b751a95. Report an issue: GitHub.

Appendix: source

Thrown at sdks/java/core/src/main/java/org/apache/beam/sdk/util/construction/UnboundedReadFromBoundedSource.java:267

          throws CoderException, IOException {
        elemsCoder.encode(value.residualElements, outStream);
        sourceCoder.encode(value.residualSource, outStream);
      }

      @SuppressWarnings("unchecked")
      @Override
      public Checkpoint<T> decode(InputStream inStream) throws CoderException, IOException {
        return new Checkpoint<>(elemsCoder.decode(inStream), sourceCoder.decode(inStream));
      }

      @Override
      public List<Coder<?>> getCoderArguments() {
        return Arrays.asList(elemCoder);
      }

      @Override
      public void verifyDeterministic() throws NonDeterministicException {
        throw new NonDeterministicException(
            this, "CheckpointCoder uses Java Serialization, which may be non-deterministic.");
      }
    }

    /**
     * An {@code UnboundedReader<T>} that wraps a {@code BoundedSource<T>} into {@link
     * ResidualElements} and {@link ResidualSource}.
     *
     * <p>In the initial state, {@link ResidualElements} is null and {@link ResidualSource} contains
     * the {@code BoundedSource<T>}. After the first checkpoint, the {@code BoundedSource<T>} will
     * be split into {@link ResidualElements} and {@link ResidualSource}.
     */
    @VisibleForTesting
    class Reader extends UnboundedReader<T> {
      // Initialized in init()
      private @Nullable ResidualElements residualElements;
      private @Nullable ResidualSource residualSource;
      private final PipelineOptions options;

View on GitHub (pinned to 12126d8942)