apache/beam · warning · NonDeterministicException

DatadogEvent can hold arbitrary instances, which may be…

Error message

DatadogEvent can hold arbitrary instances, which may be non-deterministic.

What it means

DatadogEventCoder.verifyDeterministic() unconditionally throws NonDeterministicException because DatadogEvent can carry arbitrary Object instances whose serialized bytes may differ between runs. Beam requires deterministic coders for operations like GroupByKey/reshuffle, so this coder declares itself non-deterministic up front.

Solutions

Avoid GroupByKey/Combine on PCollection<DatadogEvent>; send events directly from each element (DatadogIO.write is designed for that).
Convert events to a deterministic, byte-stable type (e.g. the serialized JSON String or a fixed-field POJO) before shuffling.
Register/override a custom deterministic Coder<DatadogEvent> if you control the event contents.

Example fix

// before
events.apply(GroupByKey.create()); // throws NonDeterministicException
// after
events.apply(MapElements.into(TypeDescriptor.of(String.class))
        .via(event -> event.serializeToJson()))
    .apply(GroupByKey.create());

Defensive patterns

Strategy: fallback

Try / catch

try { events.apply(GroupByKey.create()); } catch (NonDeterministicException e) { /* switch to mapping events to deterministic Strings before shuffling */ }

Prevention

Never GroupByKey/Combine directly on PCollection<DatadogEvent>.
Convert events to byte-stable representations (JSON String, fixed POJO) before any shuffle.
Review coder requirements when adding aggregation steps to event pipelines.

When it happens

Trigger: Using DatadogEvent in an operation that requires deterministic coding, e.g. PCollection<DatadogEvent> followed by GroupByKey, Combine, or a reshuffle that triggers coder.verifyDeterministic().

Common situations: Aggregating or deduplicating events before sending to Datadog; adding a shuffle step in a pipeline that previously wrote events directly; Beam upgrading and becoming stricter about coder determinism checks.

Understand the failure class

Background: UnsupportedOperationException and "is not supported" errors: when a library deliberately refuses a call — this error's family across 30 libraries.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/4382091ba939e5d7. Report an issue: GitHub.

Appendix: source

Thrown at sdks/java/io/datadog/src/main/java/org/apache/beam/sdk/io/datadog/DatadogEventCoder.java:91

      builder.withService(service);
    }

    String message = STRING_NULLABLE_CODER.decode(in);
    if (message != null) {
      builder.withMessage(message);
    }

    return builder.build();
  }

  @Override
  public TypeDescriptor<DatadogEvent> getEncodedTypeDescriptor() {
    return TYPE_DESCRIPTOR;
  }

  @Override
  public void verifyDeterministic() throws NonDeterministicException {
    throw new NonDeterministicException(
        this, "DatadogEvent can hold arbitrary instances, which may be non-deterministic.");
  }
}

View on GitHub (pinned to 12126d8942)