{"record":{"id":"1e3a69dc402b43eb","repo":"apache/beam","slug":"can-t-safely-pickle-generator-objects","errorCode":null,"errorMessage":"can't (safely) pickle generator objects","messagePattern":"can't \\(safely\\) pickle generator objects","errorType":"exception","errorClass":"TypeError","httpStatus":null,"severity":"error","filePath":"sdks/python/apache_beam/internal/dill_pickler.py","lineNumber":290,"sourceCode":"\n  return wrapper\n\n\n# Monkey patch the standard pickler dispatch table entry for type objects.\n# Dill, for certain types, defers to the standard pickler (including type\n# objects). We wrap the standard handler using type_wrapper() because\n# for nested class we want to pickle the actual enclosing class object so we\n# can recreate it during unpickling.\n# TODO(silviuc): Make sure we submit the fix upstream to GitHub dill project.\ndill.dill.Pickler.dispatch[type] = _nested_type_wrapper(\n    dill.dill.Pickler.dispatch[type])\n\n\n# Dill pickles generators objects without complaint, but unpickling produces\n# TypeError: object.__new__(generator) is not safe, use generator.__new__()\n# on some versions of Python.\ndef _reject_generators(unused_pickler, unused_obj):\n  raise TypeError(\"can't (safely) pickle generator objects\")\n\n\ndill.dill.Pickler.dispatch[types.GeneratorType] = _reject_generators\n\n# This if guards against dill not being full initialized when generating docs.\nif 'save_module' in dir(dill.dill):\n\n  # Always pickle non-main modules by name.\n  old_save_module = dill.dill.save_module\n\n  @dill.dill.register(dill.dill.ModuleType)\n  def save_module(pickler, obj):\n    if dill.dill.is_dill(pickler) and obj is pickler._main:\n      return old_save_module(pickler, obj)\n    else:\n      dill_log.info('M2: %s' % obj)\n      # pylint: disable=protected-access\n      pickler.save_reduce(","sourceCodeStart":272,"sourceCodeEnd":308,"githubUrl":"https://github.com/apache/beam/blob/12126d8942aaf848030c478b4c6a28c6af861c66/sdks/python/apache_beam/internal/dill_pickler.py#L272-L308","documentation":"dill will happily pickle generator objects, but unpickling them fails on several Python versions with 'object.__new__(generator) is not safe'. Beam therefore registers a dispatch hook in dill_pickler that raises TypeError \"can't (safely) pickle generator objects\" at pickling time so the failure surfaces early and clearly instead of at unpickle time on a worker.","triggerScenarios":"Submitting a Beam pipeline (or otherwise dill-pickling) a DoFn/closure that captures a live generator object (a stored genexp, a generator from zip/map, a generator held as instance state or passed as a side input default).","commonSituations":"Storing `yield`-producing iterators or generator expressions in DoFn attributes or ParDo closures; capturing generator pipelines like `(x for x in ...)` created in __main__; passing generators in __init__ arguments that get pickled with the DoFn.","solutions":["Materialize the generator to a list (or other concrete container) before it is captured by the pickled closure.","Replace the generator with a Beam transform (e.g. produce elements inside the DoFn, or use an iterable side input).","Use a function that creates the generator at runtime instead of capturing the generator instance itself.","If the generator was partially consumed, verify your data flow — materializing preserves a snapshot."],"exampleFix":"// before\nclass MyDoFn(beam.DoFn):\n    def __init__(self, items):\n        self.items = (i for i in items)  # generator captured\n// after\nclass MyDoFn(beam.DoFn):\n    def __init__(self, items):\n        self.items = list(items)  # materialized","handlingStrategy":"type-guard","validationCode":"import types\ndef reject_captured_generators(obj):\n    for name, val in vars(obj).items():\n        if isinstance(val, types.GeneratorType):\n            raise TypeError(f'{name} is a generator; materialize with list() before pickling')","typeGuard":"def is_generator(o) -> bool:\n    import types\n    return isinstance(o, types.GeneratorType)","tryCatchPattern":"try:\n    pipeline.run()\nexcept TypeError as e:\n    if 'generator' in str(e):\n        raise RuntimeError('A captured generator object cannot be pickled; materialize it to a list first') from e\n    raise","preventionTips":["Never store generators in DoFn __init__ state or closures sent to workers.","Materialize with list() or tuple() before capture.","Push element production into the pipeline (source transforms) rather than pre-generated iterators.","Watch for genexp assignments: `self.x = (i for i in ...)` is a generator."],"tags":["python","apache-beam","dill","pickling","generator"],"backgroundTag":"unsupported-operation","analyzedSha":"12126d8942aaf848030c478b4c6a28c6af861c66","analyzedAt":"2026-09-13T01:50:10.254Z","contentChangedAt":"2026-09-13T01:50:10.254Z","schemaVersion":2},"datasetVersion":"2026-09-20T03:17:13.778Z"}