{"record":{"id":"38ec32080e2546d2","repo":"apache/beam","slug":"datadogwriteerror-can-hold-arbitrary-instances-which-may-be","errorCode":null,"errorMessage":"DatadogWriteError can hold arbitrary instances, which may be non-deterministic.","messagePattern":"DatadogWriteError can hold arbitrary instances, which may be non-deterministic\\.","errorType":"exception","errorClass":"NonDeterministicException","httpStatus":null,"severity":"error","filePath":"sdks/java/io/datadog/src/main/java/org/apache/beam/sdk/io/datadog/DatadogWriteErrorCoder.java","lineNumber":85,"sourceCode":"      builder.withStatusMessage(statusMessage);\n    }\n\n    String payload = STRING_NULLABLE_CODER.decode(in);\n    if (payload != null) {\n      builder.withPayload(payload);\n    }\n\n    return builder.build();\n  }\n\n  @Override\n  public TypeDescriptor<DatadogWriteError> getEncodedTypeDescriptor() {\n    return TYPE_DESCRIPTOR;\n  }\n\n  @Override\n  public void verifyDeterministic() throws NonDeterministicException {\n    throw new NonDeterministicException(\n        this, \"DatadogWriteError can hold arbitrary instances, which may be non-deterministic.\");\n  }\n}\n","sourceCodeStart":67,"sourceCodeEnd":89,"githubUrl":"https://github.com/apache/beam/blob/12126d8942aaf848030c478b4c6a28c6af861c66/sdks/java/io/datadog/src/main/java/org/apache/beam/sdk/io/datadog/DatadogWriteErrorCoder.java#L67-L89","documentation":"The Coder for DatadogWriteError explicitly declares itself non-deterministic. DatadogWriteError may wrap arbitrary payload objects whose encodings can differ between JVMs (e.g. HashMap iteration order), so Beam throws NonDeterministicException when a deterministic coder is required, such as for GroupByKey or stateful transforms.","triggerScenarios":"Using DatadogWriteError elements upstream of a GroupByKey, or calling verifyDeterministic() on DatadogWriteErrorCoder directly; also when a pipeline stage requires key-encoded determinism and the error type flows into a key.","commonSituations":"Developers branching Datadog write failures into a side output and then aggregating/grouping the failed records, or writing failed records to a sink that requires deterministic encoding.","solutions":["Do not group or key on DatadogWriteError elements; restructure the pipeline so only deterministic types (e.g. String key/value) feed grouping transforms.","Extract deterministic fields (statusCode, statusMessage) into your own class with a deterministic coder before aggregating.","Wrap the payload in a type with a deterministic custom coder (e.g. Avro/Proto based) instead of the default arbitrary-instance coder."],"exampleFix":"// before\nPCollection<DatadogWriteError> failures = ...;\nfailures.apply(GroupByKey.create()); // throws NonDeterministicException\n// after\nPCollection<KV<String, String>> failures = errors.apply(\n    MapElements.into(TypeDescriptor.of(KV.class))\n        .via(e -> KV.of(e.statusMessage(), String.valueOf(e.statusCode()))));","handlingStrategy":"try-catch","validationCode":"// Prefer deterministic types before grouping\nboolean safe = payload instanceof String || payload instanceof java.io.Serializable;\nif (!safe) throw new IllegalStateException(\"DatadogWriteError payload must be deterministic before GroupByKey\");","typeGuard":"static boolean deterministicPayload(DatadogWriteError e) {\n  return e.statusMessage() instanceof String && e.statusCode() == null || e.statusCode() instanceof Integer;\n}","tryCatchPattern":"try {\n  coder.verifyDeterministic();\n} catch (Coder.NonDeterministicException e) {\n  // re-map elements to a deterministic representation before the grouping stage\n}","preventionTips":["Never feed coder-non-deterministic types into GroupByKey/stateful DoFns.","Project errors down to plain String fields before aggregation.","Document custom payload types with deterministic coders if needed."],"tags":["java","apache-beam","coder","determinism"],"backgroundTag":"unsupported-operation","analyzedSha":"12126d8942aaf848030c478b4c6a28c6af861c66","analyzedAt":"2026-09-13T01:50:10.254Z","contentChangedAt":"2026-09-13T01:50:10.254Z","schemaVersion":2},"datasetVersion":"2026-09-14T16:17:12.679Z"}