apache/beam · error · NonDeterministicException

SplunkEvent can hold arbitrary instances, which may be non-d

Error message

SplunkEvent can hold arbitrary instances, which may be non-deterministic.

What it means

SplunkEventCoder.verifyDeterministic() unconditionally declares that SplunkEvent coding is non-deterministic, because a SplunkEvent can contain arbitrary Java objects (nested maps/lists) whose encoded byte form may differ between encodes of equal events. Beam requires deterministic coders for grouping (GBK), state/timers, and some sinks, so this coder refuses.

Source

Thrown at sdks/java/io/splunk/src/main/java/org/apache/beam/sdk/io/splunk/SplunkEventCoder.java:203

    String sourceType = STRING_NULLABLE_CODER.decode(in);
    if (sourceType != null) {
      builder.withSourceType(sourceType);
    }

    String index = STRING_NULLABLE_CODER.decode(in);
    if (index != null) {
      builder.withIndex(index);
    }
  }

  @Override
  public TypeDescriptor<SplunkEvent> getEncodedTypeDescriptor() {
    return TYPE_DESCRIPTOR;
  }

  @Override
  public void verifyDeterministic() throws NonDeterministicException {
    throw new NonDeterministicException(
        this, "SplunkEvent can hold arbitrary instances, which may be non-deterministic.");
  }
}

View on GitHub (pinned to 12126d8942)

Solutions

1. Convert SplunkEvents to a deterministically codable type (e.g., a custom Avro/POJO/Row with ordered fields) before grouping.
2. Provide an explicit deterministic coder via .setCoder() on the PCollection with a canonical serialization (sorted keys, fixed field order).
3. If only the event's byte payload matters, extract a String/byte[] representation and use StringUtf8Coder/ByteArrayCoder semantics.

Example fix

// before
PCollection<KV<String, Iterable<SplunkEvent>>> grouped = events
    .apply(GroupByKey.create());
// after
PCollection<KV<String, String>> flat = events.apply(MapElements.into(TypeDescriptor.of(String.class))
    .via(e -> canonicalJson(e)));
PCollection<KV<String, Iterable<String>>> grouped = flat.apply(GroupByKey.create());

// Java — avoid grouping SplunkEvents; convert to a deterministic type first
PCollection<KV<String, String>> kv = events.apply(MapElements.into(TypeDescriptors.strings()).via(e -> e.value() == null ? "" : e.value()));

boolean deterministicSafe(Class<?> c) { return String.class.equals(c) || byte[].class.equals(c) || Row.class.isAssignableFrom(c); }

When it happens

Trigger: Using SplunkEvent in a PCollection that requires deterministic encoding: Apply.groupByKey(), Combine, stateful DoFns, or writing SplunkEvents through a runner stage that checks coder determinism (e.g., after SplunkIO.read() results are re-grouped).

Common situations: Users parse Splunk events then groupByKey on event fields to deduplicate or join; deterministic-coder validation fails at job submission with this NonDeterministicException.

Understand the failure class

Background: UnsupportedOperationException and "is not supported" errors: when a library deliberately refuses a call — this error's family across 30 libraries.

Related errors

• the keyCoder of a GroupByEncryptedKey must be deterministic
• the keyCoder of a GroupByKey must be deterministic
• Window coders must be deterministic.
• the secondary key coder of SortValues must be deterministic
• DatadogWriteError can hold arbitrary instances, which may be

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/61fd02b7411e70ee. Report an issue: GitHub.