{"record":{"id":"9203db81f02589dd","repo":"apache/beam","slug":"could-not-pickle-object-as-excessively-deep-recursion","errorCode":null,"errorMessage":"Could not pickle object as excessively deep recursion required.","messagePattern":"Could not pickle object as excessively deep recursion required\\.","errorType":"exception","errorClass":"PicklingError","httpStatus":null,"severity":"error","filePath":"sdks/python/apache_beam/internal/cloudpickle/cloudpickle.py","lineNumber":1469,"sourceCode":"      for k in [\"__package__\", \"__name__\", \"__path__\"]:\n        if k in func.__globals__:\n          base_globals[k] = func.__globals__[k]\n\n    # Do not bind the free variables before the function is created to\n    # avoid infinite recursion.\n    if func.__closure__ is None:\n      closure = None\n    else:\n      closure = tuple(_make_empty_cell() for _ in range(len(code.co_freevars)))\n\n    return code, base_globals, None, None, closure\n\n  def dump(self, obj):\n    try:\n      return super().dump(obj)\n    except RecursionError as e:\n      msg = \"Could not pickle object as excessively deep recursion required.\"\n      raise pickle.PicklingError(msg) from e\n\n  def __init__(\n      self,\n      file,\n      protocol=None,\n      buffer_callback=None,\n      config: CloudPickleConfig = DEFAULT_CONFIG):\n    if protocol is None:\n      protocol = DEFAULT_PROTOCOL\n    super().__init__(file, protocol=protocol, buffer_callback=buffer_callback)\n    # map functions __globals__ attribute ids, to ensure that functions\n    # sharing the same global namespace at pickling time also share\n    # their global namespace at unpickling time.\n    self.globals_ref = {}\n    self.proto = int(protocol)\n    self.config = config\n\n  if not PYPY:","sourceCodeStart":1451,"sourceCodeEnd":1487,"githubUrl":"https://github.com/apache/beam/blob/12126d8942aaf848030c478b4c6a28c6af861c66/sdks/python/apache_beam/internal/cloudpickle/cloudpickle.py#L1451-L1487","documentation":"Beam's vendored cloudpickle CloudPickler.dump delegates to the base pickler but converts RecursionError into pickle.PicklingError('Could not pickle object as excessively deep recursion required.'), because serializing the object exceeded Python's recursion limit — the object graph is too deeply nested (or cyclic in a way the pickler chases too far).","triggerScenarios":"Cloudpickling deeply nested structures (very long lists/tuples/dicts chains, deeply nested classes or closures) submitted to a Beam pipeline; mutually recursive object graphs; extremely deep inheritance or wrapper layers.","commonSituations":"Huge nested config objects captured in DoFn closures; recursive data structures built without normalization; frameworks stacking many decorators/wrappers before serialization; small sys.setrecursionlimit values.","solutions":["Simplify/flatten the object being pickled; pass only needed data, not the whole graph.","Increase the recursion limit in the submitting process: sys.setrecursionlimit(10000).","Break cycles and remove unnecessary nested references from closures.","Serialize large structures to a file (pickle/json) and pass the path instead of the object."],"exampleFix":"// before\npipeline.apply(fn, deeply_nested_obj)  # RecursionError on dump\n// after\nsave_to_file(deeply_nested_obj, 'state.pkl')\npipeline.apply(fn, 'state.pkl')","handlingStrategy":"try-catch","validationCode":"import sys\nif sys.getrecursionlimit() < 10000:\n    sys.setrecursionlimit(10000)","typeGuard":null,"tryCatchPattern":"try:\n    cloudpickle.dump(obj)\nexcept pickle.PicklingError as e:\n    if 'recursion' in str(e): obj = flatten(obj)\n    else: raise","preventionTips":["Keep closures small; pass only required data","Avoid cyclic/deeply nested graphs in DoFn state","Persist large structures to files and pass paths","Raise recursionlimit for genuinely deep structures"],"tags":["python","pickle","recursion","cloudpickle"],"backgroundTag":"json-serialization-failed","analyzedSha":"12126d8942aaf848030c478b4c6a28c6af861c66","analyzedAt":"2026-09-13T01:50:10.254Z","contentChangedAt":"2026-09-13T01:50:10.254Z","schemaVersion":2},"datasetVersion":"2026-09-20T03:17:13.778Z"}