apache/beam · error · NonDeterministicException

DatadogWriteError can hold arbitrary instances, which may be

Error message

DatadogWriteError can hold arbitrary instances, which may be non-deterministic.

What it means

The Coder for DatadogWriteError explicitly declares itself non-deterministic. DatadogWriteError may wrap arbitrary payload objects whose encodings can differ between JVMs (e.g. HashMap iteration order), so Beam throws NonDeterministicException when a deterministic coder is required, such as for GroupByKey or stateful transforms.

Source

Thrown at sdks/java/io/datadog/src/main/java/org/apache/beam/sdk/io/datadog/DatadogWriteErrorCoder.java:85

      builder.withStatusMessage(statusMessage);
    }

    String payload = STRING_NULLABLE_CODER.decode(in);
    if (payload != null) {
      builder.withPayload(payload);
    }

    return builder.build();
  }

  @Override
  public TypeDescriptor<DatadogWriteError> getEncodedTypeDescriptor() {
    return TYPE_DESCRIPTOR;
  }

  @Override
  public void verifyDeterministic() throws NonDeterministicException {
    throw new NonDeterministicException(
        this, "DatadogWriteError can hold arbitrary instances, which may be non-deterministic.");
  }
}

View on GitHub (pinned to 12126d8942)

Solutions

1. Do not group or key on DatadogWriteError elements; restructure the pipeline so only deterministic types (e.g. String key/value) feed grouping transforms.
2. Extract deterministic fields (statusCode, statusMessage) into your own class with a deterministic coder before aggregating.
3. Wrap the payload in a type with a deterministic custom coder (e.g. Avro/Proto based) instead of the default arbitrary-instance coder.

Example fix

// before
PCollection<DatadogWriteError> failures = ...;
failures.apply(GroupByKey.create()); // throws NonDeterministicException
// after
PCollection<KV<String, String>> failures = errors.apply(
    MapElements.into(TypeDescriptor.of(KV.class))
        .via(e -> KV.of(e.statusMessage(), String.valueOf(e.statusCode()))));

// Prefer deterministic types before grouping
boolean safe = payload instanceof String || payload instanceof java.io.Serializable;
if (!safe) throw new IllegalStateException("DatadogWriteError payload must be deterministic before GroupByKey");

static boolean deterministicPayload(DatadogWriteError e) {
  return e.statusMessage() instanceof String && e.statusCode() == null || e.statusCode() instanceof Integer;
}

try {
  coder.verifyDeterministic();
} catch (Coder.NonDeterministicException e) {
  // re-map elements to a deterministic representation before the grouping stage
}

When it happens

Trigger: Using DatadogWriteError elements upstream of a GroupByKey, or calling verifyDeterministic() on DatadogWriteErrorCoder directly; also when a pipeline stage requires key-encoded determinism and the error type flows into a key.

Common situations: Developers branching Datadog write failures into a side output and then aggregating/grouping the failed records, or writing failed records to a sink that requires deterministic encoding.

Understand the failure class

Background: UnsupportedOperationException and "is not supported" errors: when a library deliberately refuses a call — this error's family across 30 libraries.

Related errors

• the keyCoder of a GroupByEncryptedKey must be deterministic
• the keyCoder of a GroupByKey must be deterministic
• Window coders must be deterministic.
• the secondary key coder of SortValues must be deterministic
• SplunkEvent can hold arbitrary instances, which may be non-d

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/38ec32080e2546d2. Report an issue: GitHub.