{"record":{"id":"691363586e175e3a","repo":"apache/beam","slug":"protocol-buffers-message-s-transitively-includes-map-field-s","errorCode":null,"errorMessage":"Protocol Buffers message %s transitively includes Map field %s (from file %s). Maps cannot be deterministically encoded.","messagePattern":"Protocol Buffers message (.+?) transitively includes Map field (.+?) \\(from file (.+?)\\)\\. Maps cannot be deterministically encoded\\.","errorType":"exception","errorClass":"org.apache.beam.sdk.coders.Coder$NonDeterministicException","httpStatus":null,"severity":"error","filePath":"sdks/java/extensions/protobuf/src/main/java/org/apache/beam/sdk/extensions/protobuf/ProtobufUtil.java","lineNumber":86,"sourceCode":"   * deterministically encoded.\n   *\n   * @throws NonDeterministicException if the object cannot be encoded deterministically.\n   */\n  static void verifyDeterministic(ProtoCoder<?> coder) throws NonDeterministicException {\n    Class<? extends Message> message = coder.getMessageType();\n    ExtensionRegistry registry = coder.getExtensionRegistry();\n    Set<Descriptor> descriptors = getRecursiveDescriptorsForClass(message, registry);\n    for (Descriptor d : descriptors) {\n      for (FieldDescriptor fd : d.getFields()) {\n        // If there is a transitively reachable Protocol Buffers map field, then this object cannot\n        // be encoded deterministically.\n        if (fd.isMapField()) {\n          String reason =\n              String.format(\n                  \"Protocol Buffers message %s transitively includes Map field %s (from file %s).\"\n                      + \" Maps cannot be deterministically encoded.\",\n                  message.getName(), fd.getFullName(), fd.getFile().getFullName());\n          throw new NonDeterministicException(coder, reason);\n        }\n      }\n    }\n  }\n\n  ////////////////////////////////////////////////////////////////////////////////////////////////\n  // Disable construction of utility class\n  private ProtobufUtil() {}\n\n  private static void recursivelyAddDescriptors(\n      Descriptor message, Set<Descriptor> descriptors, ExtensionRegistry registry) {\n    if (descriptors.contains(message)) {\n      return;\n    }\n    descriptors.add(message);\n\n    for (FieldDescriptor f : message.getFields()) {\n      recursivelyAddDescriptors(f, descriptors, registry);","sourceCodeStart":68,"sourceCodeEnd":104,"githubUrl":"https://github.com/apache/beam/blob/12126d8942aaf848030c478b4c6a28c6af861c66/sdks/java/extensions/protobuf/src/main/java/org/apache/beam/sdk/extensions/protobuf/ProtobufUtil.java#L68-L104","documentation":"verifyDeterministic checks whether a proto coder can produce deterministic encodings, which Beam requires for keyed operations like GroupByKey on sorted input. Map fields in protobuf have unspecified iteration/serialization order, so a message that transitively contains a map field makes the coder non-deterministic, and a NonDeterministicException is thrown with a message naming the message, map field, and file.","triggerScenarios":"Calling ProtobufUtil.verifyDeterministic(coder) (e.g. when Beam demands a deterministic coder for the proto class) where the message or any nested message has a `map<K,V>` field.","commonSituations":"Using a proto with map fields as a key in GroupByKey/CoGroupByKey; switching a pipeline to require deterministic coders; adding a map field to a previously map-free proto used as a key.","solutions":["Replace the `map<K,V>` field with `repeated Entry` message pairs and sort the entries (e.g. by key) before encoding to restore determinism.","Sort map entries in a custom Coder's encode() implementation, or use a coder that serializes maps in sorted key order.","Exclude the map field from keyed usage — choose a different deterministic field as the key."],"exampleFix":"// before\nmessage Config { map<string, string> tags = 1; } // NonDeterministicException\n// after\nmessage Tag { string key = 1; string value = 2; }\nmessage Config { repeated Tag tags = 1; } // sort by key before encoding","handlingStrategy":"validation","validationCode":"// Java: reject map fields before using the proto as a keyed/deterministic element\nstatic boolean hasMapField(Descriptors.Descriptor d, Set<String> visited) {\n  if (!visited.add(d.getFullName())) return false;\n  for (Descriptors.FieldDescriptor f : d.getFields()) {\n    if (f.isMapField()) return true;\n    if (f.getType() == Descriptors.FieldDescriptor.Type.MESSAGE && hasMapField(f.getMessageType(), visited)) return true;\n  }\n  return false;\n}","typeGuard":null,"tryCatchPattern":"try { ProtobufUtil.verifyDeterministic(coder); } catch (NonDeterministicException e) { /* replace/sort map fields or pick another key */ }","preventionTips":["Never use proto messages containing map fields as keys for GroupByKey/CoGroupByKey.","Model ordered maps as repeated Entry messages sorted by key instead of map<> fields.","Call verifyDeterministic early in pipeline construction to surface the offending field before runtime."],"tags":["protobuf","determinism","coder","java"],"backgroundTag":"unsupported-operation","analyzedSha":"12126d8942aaf848030c478b4c6a28c6af861c66","analyzedAt":"2026-09-13T01:50:10.254Z","contentChangedAt":"2026-09-13T01:50:10.254Z","schemaVersion":2},"datasetVersion":"2026-09-20T03:17:13.778Z"}