{"record":{"id":"30d082d2dfd53d0b","repo":"apache/beam","slug":"unable-to-pickle-fn-s-s-user-code-must-be-serializable","errorCode":null,"errorMessage":"Unable to pickle fn %s: %s. User code must be serializable (picklable) for distributed execution. This usually happens when lambdas or closures capture non-serializable objects like file handles, database connections, or thread locks. Try: (1) using module-level functions instead of lambdas, (2) initializing resources in setup() methods, (3) checking what your closure captures.","messagePattern":"Unable to pickle fn (.+?): (.+?)\\. User code must be serializable \\(picklable\\) for distributed execution\\. This usually happens when lambdas or closures capture non-serializable objects like file handles, database connections, or thread locks\\. Try: \\(1\\) using module-level functions instead of lambdas, \\(2\\) initializing resources in setup\\(\\) methods, \\(3\\) checking what your closure captures\\.","errorType":"exception","errorClass":"RuntimeError","httpStatus":null,"severity":"error","filePath":"sdks/python/apache_beam/transforms/ptransform.py","lineNumber":903,"sourceCode":"\n    if (any(isinstance(v, pvalue.PCollection) for v in args) or\n        any(isinstance(v, pvalue.PCollection) for v in kwargs.values())):\n      raise error.SideInputError(\n          'PCollection used directly as side input argument. Specify '\n          'AsIter(pcollection) or AsSingleton(pcollection) to indicate how the '\n          'PCollection is to be used.')\n    self.args, self.kwargs, self.side_inputs = util.remove_objects_from_args(\n        args, kwargs, pvalue.AsSideInput)\n    self.raw_side_inputs = args, kwargs\n\n    # Prevent name collisions with fns of the form '<function <lambda> at ...>'\n    self._cached_fn = self.fn\n\n    # Ensure fn and side inputs are picklable for remote execution.\n    try:\n      self.fn = pickler.roundtrip(self.fn)\n    except (RuntimeError, TypeError, Exception) as e:\n      raise RuntimeError(\n          'Unable to pickle fn %s: %s. '\n          'User code must be serializable (picklable) for distributed '\n          'execution. This usually happens when lambdas or closures capture '\n          'non-serializable objects like file handles, database connections, '\n          'or thread locks. Try: (1) using module-level functions instead of '\n          'lambdas, (2) initializing resources in setup() methods, '\n          '(3) checking what your closure captures.' % (self.fn, e)) from e\n\n    self.args = pickler.roundtrip(self.args)\n    self.kwargs = pickler.roundtrip(self.kwargs)\n\n    # For type hints, because loads(dumps(class)) != class.\n    self.fn = self._cached_fn\n\n  def with_input_types(\n      self, input_type_hint, *side_inputs_arg_hints, **side_input_kwarg_hints):\n    \"\"\"Annotates the types of main inputs and side inputs for the PTransform.\n","sourceCodeStart":885,"sourceCodeEnd":921,"githubUrl":"https://github.com/apache/beam/blob/12126d8942aaf848030c478b4c6a28c6af861c66/sdks/python/apache_beam/transforms/ptransform.py#L885-L921","documentation":"At PTransform construction time Beam pickles and unpickles (`pickler.roundtrip`) the user-supplied fn to guarantee it is serializable for distributed workers. If pickling fails (RuntimeError/TypeError etc.), this RuntimeError wraps it, because an unserializable fn would only fail later, possibly on remote workers, with a more confusing error.","triggerScenarios":"PTransform __init__ where `self.fn` cannot be pickled: lambdas defined in REPL/notebooks, closures capturing file handles, sockets, DB connections, locks, or objects defined inside functions; side inputs that are not picklable.","commonSituations":"Defining lambdas in interactive sessions (REPL/Jupyter/Databricks) where their globals aren't importable; capturing open connections in DoFn closures; using non-top-level helper classes; Windows-specific issues pickling locally-scoped functions.","solutions":["Replace lambdas with module-level functions (top-level defs) so workers can import them.","Move resource initialization into `DoFn.setup()` or use `beam.CombineFn`/manager patterns instead of capturing live objects (files, connections, locks) in the closure.","Simplify the closure: only capture plain data (str/int/dict) and attach complex objects via DoFn setup/teardown lifecycle.","If using a notebook, define the function in a `.py` module on the pipeline path, or use `dill`-based environments that Beam supports (e.g. interactive runner)."],"exampleFix":"# before\nlock = threading.Lock()\np | beam.Map(lambda x: (lock, x))\n# after\nclass AddLock(beam.DoFn):\n  def setup(self):\n    self.lock = threading.Lock()\n  def process(self, x):\n    yield (self.lock, x)","handlingStrategy":"try-catch","validationCode":"import pickle\ntry:\n  pickle.dumps(fn)\nexcept Exception as e:\n  raise TypeError(f'fn {fn!r} is not picklable: {e}')","typeGuard":"def is_picklable(obj):\n  try:\n    pickle.dumps(obj)\n    return True\n  except Exception:\n    return False","tryCatchPattern":"try:\n  result = pc | beam.Map(fn)\nexcept RuntimeError as e:\n  if 'Unable to pickle fn' in str(e):\n    log.error('fn not serializable: %s', e)\n    raise TypeError('Use a module-level function or DoFn with setup()') from e\n  raise","preventionTips":["Avoid lambdas in pipelines destined for remote runners; define module-level functions.","Never capture file handles, connections, or locks in DoFn closures; init them in setup().","Test pipelines with a local roundtrip: pickle.dumps on all fns before submitting."],"tags":["python","apache-beam","pickling","serialization"],"backgroundTag":"json-marshal-failed","analyzedSha":"12126d8942aaf848030c478b4c6a28c6af861c66","analyzedAt":"2026-09-13T01:50:10.254Z","contentChangedAt":"2026-09-13T01:50:10.254Z","schemaVersion":2},"datasetVersion":"2026-09-20T03:17:13.778Z"}