apache/beam · error · ValueError

Unexpected DoFn type

Error message

Unexpected DoFn type: %s

What it means

DoFnInfo.from_runner_api deserializes a DoFnInfo protobuf by dispatching on its URN. Only the pickled-info URN and URNs registered in StatelessDoFnInfo.REGISTERED_DOFNS are recognized; any other URN means the payload references a DoFn type this Beam version cannot interpret, so it raises ValueError.

Solutions

1. Align Beam versions: run the pipeline with the same (or compatible) apache_beam version that serialized the DoFnInfo.
2. If the DoFn is a recognized stateless built-in, ensure the SDK host includes it (upgrade apache_beam so the URN is in REGISTERED_DOFNS).
3. Re-generate the pipeline with the target SDK instead of hand-carrying runner-api protos across versions.
4. Check for stale prebuilt worker containers/images and rebuild them with the current SDK.

Example fix

null

from apache_beam.transforms.core import StatelessDoFnInfo, DoFnInfo
from apache_beam.portability import python_urns
assert spec.urn == python_urns.PICKLED_DOFN_INFO or spec.urn in StatelessDoFnInfo.REGISTERED_DOFNS, 'unsupported DoFn urn: %s' % spec.urn

def is_supported_dofn_urn(urn, registered):
    return urn == python_urns.PICKLED_DOFN_INFO or urn in registered

try:
    info = DoFnInfo.from_runner_api(spec, context)
except ValueError as e:
    logger.error('DoFn urn %s unsupported by SDK %s; align Beam versions', spec.urn, beam.__version__)
    raise

When it happens

Trigger: Deserializing a pipeline/runner-api proto whose DoFn spec.urn is not PICKLED_DOFN_INFO and not in REGISTERED_DOFNS — typically a pipeline produced by a different (newer or older) Beam version or a custom URN.

Common situations: Cross-version pipeline transport (submitting a pipeline generated by Beam X to a Beam Y SDK worker); custom transforms writing their own DoFn URNs; stale worker images during pipeline upgrade.

Understand the failure class

Background: UnsupportedOperationException and "is not supported" errors: when a library deliberately refuses a call — this error's family across 30 libraries.

Related errors

• Could not find enum descriptor
• Expected 0 or 1, got
• Faled loading session: expected dict, got
• from_runner_api_parameter does not handle empty or None…
• No logical type registered for URN

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/95376e46d2488fb1. Report an issue: GitHub.

Appendix: source

Thrown at sdks/python/apache_beam/transforms/core.py:2021

    return wrapper

  @classmethod
  def create(cls, fn, args, kwargs):
    if hasattr(fn, '_stateless_dofn_urn'):
      assert not args and not kwargs
      return StatelessDoFnInfo(fn._stateless_dofn_urn)
    else:
      return PickledDoFnInfo(cls._pickled_do_fn_info(fn, args, kwargs))

  @staticmethod
  def from_runner_api(spec, unused_context):
    if spec.urn == python_urns.PICKLED_DOFN_INFO:
      return PickledDoFnInfo(spec.payload)
    elif spec.urn in StatelessDoFnInfo.REGISTERED_DOFNS:
      return StatelessDoFnInfo(spec.urn)
    else:
      raise ValueError('Unexpected DoFn type: %s' % spec.urn)

  @staticmethod
  def _pickled_do_fn_info(fn, args, kwargs):
    # This can be cleaned up once all runners move to portability.
    return pickler.dumps((fn, args, kwargs, None, None))

  def serialized_dofn_data(self):
    raise NotImplementedError(type(self))


class PickledDoFnInfo(DoFnInfo):
  def __init__(self, serialized_data):
    self._serialized_data = serialized_data

  def serialized_dofn_data(self):
    return self._serialized_data

  def to_runner_api(self, unused_context):

View on GitHub (pinned to 12126d8942)