apache/beam · error · RuntimeError

Unable to pickle fn : . User code must be serializable…

Error message

Unable to pickle fn %s: %s. User code must be serializable (picklable) for distributed execution. This usually happens when lambdas or closures capture non-serializable objects like file handles, database connections, or thread locks. Try: (1) using module-level functions instead of lambdas, (2) initializing resources in setup() methods, (3) checking what your closure captures.

What it means

At PTransform construction time Beam pickles and unpickles (`pickler.roundtrip`) the user-supplied fn to guarantee it is serializable for distributed workers. If pickling fails (RuntimeError/TypeError etc.), this RuntimeError wraps it, because an unserializable fn would only fail later, possibly on remote workers, with a more confusing error.

Solutions

1. Replace lambdas with module-level functions (top-level defs) so workers can import them.
2. Move resource initialization into `DoFn.setup()` or use `beam.CombineFn`/manager patterns instead of capturing live objects (files, connections, locks) in the closure.
3. Simplify the closure: only capture plain data (str/int/dict) and attach complex objects via DoFn setup/teardown lifecycle.
4. If using a notebook, define the function in a `.py` module on the pipeline path, or use `dill`-based environments that Beam supports (e.g. interactive runner).

Example fix

# before
lock = threading.Lock()
p | beam.Map(lambda x: (lock, x))
# after
class AddLock(beam.DoFn):
  def setup(self):
    self.lock = threading.Lock()
  def process(self, x):
    yield (self.lock, x)

import pickle
try:
  pickle.dumps(fn)
except Exception as e:
  raise TypeError(f'fn {fn!r} is not picklable: {e}')

def is_picklable(obj):
  try:
    pickle.dumps(obj)
    return True
  except Exception:
    return False

try:
  result = pc | beam.Map(fn)
except RuntimeError as e:
  if 'Unable to pickle fn' in str(e):
    log.error('fn not serializable: %s', e)
    raise TypeError('Use a module-level function or DoFn with setup()') from e
  raise

When it happens

Trigger: PTransform __init__ where `self.fn` cannot be pickled: lambdas defined in REPL/notebooks, closures capturing file handles, sockets, DB connections, locks, or objects defined inside functions; side inputs that are not picklable.

Common situations: Defining lambdas in interactive sessions (REPL/Jupyter/Databricks) where their globals aren't importable; capturing open connections in DoFn closures; using non-top-level helper classes; Windows-specific issues pickling locally-scoped functions.

Understand the failure class

Background: json.Marshal / "failed to marshal" errors in Go: why "unsupported type" happens and how to fix it — this error's family across 22 libraries.

Related errors

• Attempted to encode null for non-nullable field
• can't (safely) pickle generator objects
• cannot check importability of
• Could not find code object with path
• Could not find code object with path

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/30d082d2dfd53d0b. Report an issue: GitHub.

Appendix: source

Thrown at sdks/python/apache_beam/transforms/ptransform.py:903

    if (any(isinstance(v, pvalue.PCollection) for v in args) or
        any(isinstance(v, pvalue.PCollection) for v in kwargs.values())):
      raise error.SideInputError(
          'PCollection used directly as side input argument. Specify '
          'AsIter(pcollection) or AsSingleton(pcollection) to indicate how the '
          'PCollection is to be used.')
    self.args, self.kwargs, self.side_inputs = util.remove_objects_from_args(
        args, kwargs, pvalue.AsSideInput)
    self.raw_side_inputs = args, kwargs

    # Prevent name collisions with fns of the form '<function <lambda> at ...>'
    self._cached_fn = self.fn

    # Ensure fn and side inputs are picklable for remote execution.
    try:
      self.fn = pickler.roundtrip(self.fn)
    except (RuntimeError, TypeError, Exception) as e:
      raise RuntimeError(
          'Unable to pickle fn %s: %s. '
          'User code must be serializable (picklable) for distributed '
          'execution. This usually happens when lambdas or closures capture '
          'non-serializable objects like file handles, database connections, '
          'or thread locks. Try: (1) using module-level functions instead of '
          'lambdas, (2) initializing resources in setup() methods, '
          '(3) checking what your closure captures.' % (self.fn, e)) from e

    self.args = pickler.roundtrip(self.args)
    self.kwargs = pickler.roundtrip(self.kwargs)

    # For type hints, because loads(dumps(class)) != class.
    self.fn = self._cached_fn

  def with_input_types(
      self, input_type_hint, *side_inputs_arg_hints, **side_input_kwarg_hints):
    """Annotates the types of main inputs and side inputs for the PTransform.

View on GitHub (pinned to 12126d8942)