apache/beam · error · TypeError

can't (safely) pickle generator objects

Error message

can't (safely) pickle generator objects

What it means

dill will happily pickle generator objects, but unpickling them fails on several Python versions with 'object.__new__(generator) is not safe'. Beam therefore registers a dispatch hook in dill_pickler that raises TypeError "can't (safely) pickle generator objects" at pickling time so the failure surfaces early and clearly instead of at unpickle time on a worker.

Solutions

1. Materialize the generator to a list (or other concrete container) before it is captured by the pickled closure.
2. Replace the generator with a Beam transform (e.g. produce elements inside the DoFn, or use an iterable side input).
3. Use a function that creates the generator at runtime instead of capturing the generator instance itself.
4. If the generator was partially consumed, verify your data flow — materializing preserves a snapshot.

Example fix

// before
class MyDoFn(beam.DoFn):
    def __init__(self, items):
        self.items = (i for i in items)  # generator captured
// after
class MyDoFn(beam.DoFn):
    def __init__(self, items):
        self.items = list(items)  # materialized

import types
def reject_captured_generators(obj):
    for name, val in vars(obj).items():
        if isinstance(val, types.GeneratorType):
            raise TypeError(f'{name} is a generator; materialize with list() before pickling')

def is_generator(o) -> bool:
    import types
    return isinstance(o, types.GeneratorType)

try:
    pipeline.run()
except TypeError as e:
    if 'generator' in str(e):
        raise RuntimeError('A captured generator object cannot be pickled; materialize it to a list first') from e
    raise

When it happens

Trigger: Submitting a Beam pipeline (or otherwise dill-pickling) a DoFn/closure that captures a live generator object (a stored genexp, a generator from zip/map, a generator held as instance state or passed as a side input default).

Common situations: Storing `yield`-producing iterators or generator expressions in DoFn attributes or ParDo closures; capturing generator pipelines like `(x for x in ...)` created in __main__; passing generators in __init__ arguments that get pickled with the DoFn.

Understand the failure class

Background: UnsupportedOperationException and "is not supported" errors: when a library deliberately refuses a call — this error's family across 30 libraries.

Related errors

• cannot check importability of
• Could not find code object with path
• Could not find code object with path
• Expected Iterator in return type annotation for
• Index is out of bounds for obj.__defaults__ in path

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/1e3a69dc402b43eb. Report an issue: GitHub.

Appendix: source

Thrown at sdks/python/apache_beam/internal/dill_pickler.py:290

  return wrapper


# Monkey patch the standard pickler dispatch table entry for type objects.
# Dill, for certain types, defers to the standard pickler (including type
# objects). We wrap the standard handler using type_wrapper() because
# for nested class we want to pickle the actual enclosing class object so we
# can recreate it during unpickling.
# TODO(silviuc): Make sure we submit the fix upstream to GitHub dill project.
dill.dill.Pickler.dispatch[type] = _nested_type_wrapper(
    dill.dill.Pickler.dispatch[type])


# Dill pickles generators objects without complaint, but unpickling produces
# TypeError: object.__new__(generator) is not safe, use generator.__new__()
# on some versions of Python.
def _reject_generators(unused_pickler, unused_obj):
  raise TypeError("can't (safely) pickle generator objects")


dill.dill.Pickler.dispatch[types.GeneratorType] = _reject_generators

# This if guards against dill not being full initialized when generating docs.
if 'save_module' in dir(dill.dill):

  # Always pickle non-main modules by name.
  old_save_module = dill.dill.save_module

  @dill.dill.register(dill.dill.ModuleType)
  def save_module(pickler, obj):
    if dill.dill.is_dill(pickler) and obj is pickler._main:
      return old_save_module(pickler, obj)
    else:
      dill_log.info('M2: %s' % obj)
      # pylint: disable=protected-access
      pickler.save_reduce(

View on GitHub (pinned to 12126d8942)