apache/beam · error · ValueError

cannot be made deterministic for ' '.

Error message

%s cannot be made deterministic for '%s'.

What it means

as_deterministic_coder returns the coder unchanged when is_deterministic() is true, otherwise it raises ValueError (with an optional custom message naming the coder and pipeline step) because the operation requires deterministic encoding — e.g. GroupByKey correctness depends on stable serialized bytes.

Solutions

1. Provide a deterministic coder for the key type and register it (coders registry) or wrap via coder.as_deterministic_coder with a real deterministic implementation
2. Encode keys into deterministic primitives (e.g. sorted JSON bytes, protobuf) before grouping
3. Pass a custom error_message or a fallback coder where the API allows

Example fix

// before
pcoll | beam.GroupByKey()  # key coded with default/pickle coder
// after
pcoll | beam.Map(lambda kv: (json.dumps(kv[0], sort_keys=True).encode(), kv[1])) | beam.GroupByKey()

if not coder.is_deterministic():
    coder = deterministic_alternative(coder)  # e.g. bytes/protobuf key coder

def key_is_deterministic(coder):
    return coder.is_deterministic()

try:
    coder = coder.as_deterministic_coder(step_label)
except ValueError as e:
    log.error('need a deterministic coder for %s: %s', step_label, e)
    coder = fallback_deterministic_coder

When it happens

Trigger: deterministic_coder(coder, step_label) is called for a step that requires deterministic coders (GroupByKey on non-deterministically-coded keys) and the coder reports is_deterministic() == False and no fallback exists.

Common situations: GroupByKey/co-grouping on keys coded with a coder that serializes dicts/objects with unstable ordering (default pickle coder); users coding custom objects as keys without a deterministic coder.

Understand the failure class

Background: UnsupportedOperationException and "is not supported" errors: when a library deliberately refuses a call — this error's family across 30 libraries.

Related errors

• Attempted to encode null for non-nullable field
• Decode not implemented
• Encode not implemented
• No fallback.
• No logical type registered for URN

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/458eae2cd536a158. Report an issue: GitHub.

Appendix: source

Thrown at sdks/python/apache_beam/coders/coders.py:185

    deterministic: the ordering of picked entries in maps may vary across
    executions since there is no defined order, and such a coder is not in
    general suitable for usage as a key coder in GroupByKey operations, since
    each instance of the same key may be encoded differently.

    Returns:
      Whether coder is deterministic.
    """
    return False

  def as_deterministic_coder(self, step_label, error_message=None):
    """Returns a deterministic version of self, if possible.

    Otherwise raises a value error.
    """
    if self.is_deterministic():
      return self
    else:
      raise ValueError(
          error_message or
          "%s cannot be made deterministic for '%s'." % (self, step_label))

  def estimate_size(self, value):
    """Estimates the encoded size of the given value, in bytes.

    Dataflow estimates the encoded size of a PCollection processed in a pipeline
    step by using the estimated size of a random sample of elements in that
    PCollection.

    The default implementation encodes the given value and returns its byte
    size.  If a coder can provide a fast estimate of the encoded size of a value
    (e.g., if the encoding has a fixed size), it can provide its estimate here
    to improve performance.

    Arguments:
      value: the value whose encoded size is to be estimated.

View on GitHub (pinned to 12126d8942)