apache/beam · error · org.apache.beam.sdk.coders.Coder$NonDeterministicException

Protocol Buffers message

Error message

Protocol Buffers message %s transitively includes Map field %s (from file %s). Maps cannot be deterministically encoded.

What it means

verifyDeterministic checks whether a proto coder can produce deterministic encodings, which Beam requires for keyed operations like GroupByKey on sorted input. Map fields in protobuf have unspecified iteration/serialization order, so a message that transitively contains a map field makes the coder non-deterministic, and a NonDeterministicException is thrown with a message naming the message, map field, and file.

Solutions

Replace the `map<K,V>` field with `repeated Entry` message pairs and sort the entries (e.g. by key) before encoding to restore determinism.
Sort map entries in a custom Coder's encode() implementation, or use a coder that serializes maps in sorted key order.
Exclude the map field from keyed usage — choose a different deterministic field as the key.

Example fix

// before
message Config { map<string, string> tags = 1; } // NonDeterministicException
// after
message Tag { string key = 1; string value = 2; }
message Config { repeated Tag tags = 1; } // sort by key before encoding

Defensive patterns

Strategy: validation

Validate before calling

// Java: reject map fields before using the proto as a keyed/deterministic element
static boolean hasMapField(Descriptors.Descriptor d, Set<String> visited) {
  if (!visited.add(d.getFullName())) return false;
  for (Descriptors.FieldDescriptor f : d.getFields()) {
    if (f.isMapField()) return true;
    if (f.getType() == Descriptors.FieldDescriptor.Type.MESSAGE && hasMapField(f.getMessageType(), visited)) return true;
  }
  return false;
}

Try / catch

try { ProtobufUtil.verifyDeterministic(coder); } catch (NonDeterministicException e) { /* replace/sort map fields or pick another key */ }

Prevention

Never use proto messages containing map fields as keys for GroupByKey/CoGroupByKey.
Model ordered maps as repeated Entry messages sorted by key instead of map<> fields.
Call verifyDeterministic early in pipeline construction to surface the offending field before runtime.

When it happens

Trigger: Calling ProtobufUtil.verifyDeterministic(coder) (e.g. when Beam demands a deterministic coder for the proto class) where the message or any nested message has a `map<K,V>` field.

Common situations: Using a proto with map fields as a key in GroupByKey/CoGroupByKey; switching a pipeline to require deterministic coders; adding a map field to a previously map-free proto used as a key.

Understand the failure class

Background: UnsupportedOperationException and "is not supported" errors: when a library deliberately refuses a call — this error's family across 30 libraries.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/691363586e175e3a. Report an issue: GitHub.

Appendix: source

Thrown at sdks/java/extensions/protobuf/src/main/java/org/apache/beam/sdk/extensions/protobuf/ProtobufUtil.java:86

   * deterministically encoded.
   *
   * @throws NonDeterministicException if the object cannot be encoded deterministically.
   */
  static void verifyDeterministic(ProtoCoder<?> coder) throws NonDeterministicException {
    Class<? extends Message> message = coder.getMessageType();
    ExtensionRegistry registry = coder.getExtensionRegistry();
    Set<Descriptor> descriptors = getRecursiveDescriptorsForClass(message, registry);
    for (Descriptor d : descriptors) {
      for (FieldDescriptor fd : d.getFields()) {
        // If there is a transitively reachable Protocol Buffers map field, then this object cannot
        // be encoded deterministically.
        if (fd.isMapField()) {
          String reason =
              String.format(
                  "Protocol Buffers message %s transitively includes Map field %s (from file %s)."
                      + " Maps cannot be deterministically encoded.",
                  message.getName(), fd.getFullName(), fd.getFile().getFullName());
          throw new NonDeterministicException(coder, reason);
        }
      }
    }
  }

  ////////////////////////////////////////////////////////////////////////////////////////////////
  // Disable construction of utility class
  private ProtobufUtil() {}

  private static void recursivelyAddDescriptors(
      Descriptor message, Set<Descriptor> descriptors, ExtensionRegistry registry) {
    if (descriptors.contains(message)) {
      return;
    }
    descriptors.add(message);

    for (FieldDescriptor f : message.getFields()) {
      recursivelyAddDescriptors(f, descriptors, registry);

View on GitHub (pinned to 12126d8942)