apache/beam · error · PicklingError

Could not pickle object as excessively deep recursion…

Error message

Could not pickle object as excessively deep recursion required.

What it means

Beam's vendored cloudpickle CloudPickler.dump delegates to the base pickler but converts RecursionError into pickle.PicklingError('Could not pickle object as excessively deep recursion required.'), because serializing the object exceeded Python's recursion limit — the object graph is too deeply nested (or cyclic in a way the pickler chases too far).

Solutions

1. Simplify/flatten the object being pickled; pass only needed data, not the whole graph.
2. Increase the recursion limit in the submitting process: sys.setrecursionlimit(10000).
3. Break cycles and remove unnecessary nested references from closures.
4. Serialize large structures to a file (pickle/json) and pass the path instead of the object.

Example fix

// before
pipeline.apply(fn, deeply_nested_obj)  # RecursionError on dump
// after
save_to_file(deeply_nested_obj, 'state.pkl')
pipeline.apply(fn, 'state.pkl')

import sys
if sys.getrecursionlimit() < 10000:
    sys.setrecursionlimit(10000)

try:
    cloudpickle.dump(obj)
except pickle.PicklingError as e:
    if 'recursion' in str(e): obj = flatten(obj)
    else: raise

When it happens

Trigger: Cloudpickling deeply nested structures (very long lists/tuples/dicts chains, deeply nested classes or closures) submitted to a Beam pipeline; mutually recursive object graphs; extremely deep inheritance or wrapper layers.

Common situations: Huge nested config objects captured in DoFn closures; recursive data structures built without normalization; frameworks stacking many decorators/wrappers before serialization; small sys.setrecursionlimit values.

Understand the failure class

Background: "JSON serialization failed", "not JSON serializable", "Failed to serialize": why JSON marshaling errors happen and how to fix them — this error's family across 46 libraries.

Related errors

• Cannot pickle files that do not map to an actual file
• Recursive ID generation detected for
• cannot check importability of
• Cannot pickle closed files
• Cannot pickle file as it cannot be read

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/9203db81f02589dd. Report an issue: GitHub.

Appendix: source

Thrown at sdks/python/apache_beam/internal/cloudpickle/cloudpickle.py:1469

      for k in ["__package__", "__name__", "__path__"]:
        if k in func.__globals__:
          base_globals[k] = func.__globals__[k]

    # Do not bind the free variables before the function is created to
    # avoid infinite recursion.
    if func.__closure__ is None:
      closure = None
    else:
      closure = tuple(_make_empty_cell() for _ in range(len(code.co_freevars)))

    return code, base_globals, None, None, closure

  def dump(self, obj):
    try:
      return super().dump(obj)
    except RecursionError as e:
      msg = "Could not pickle object as excessively deep recursion required."
      raise pickle.PicklingError(msg) from e

  def __init__(
      self,
      file,
      protocol=None,
      buffer_callback=None,
      config: CloudPickleConfig = DEFAULT_CONFIG):
    if protocol is None:
      protocol = DEFAULT_PROTOCOL
    super().__init__(file, protocol=protocol, buffer_callback=buffer_callback)
    # map functions __globals__ attribute ids, to ensure that functions
    # sharing the same global namespace at pickling time also share
    # their global namespace at unpickling time.
    self.globals_ref = {}
    self.proto = int(protocol)
    self.config = config

  if not PYPY:

View on GitHub (pinned to 12126d8942)