apache/beam · error · PicklingError

Cannot pickle files that do not map to an actual file

Error message

Cannot pickle files that do not map to an actual file

What it means

Beam's vendored cloudpickle supports pickling open file objects via _file_reduce, but only files that expose both .name and .mode attributes. Objects lacking them (e.g. io.StringIO, io.BytesIO, temporary buffers) cannot be mapped back to a real file, so pickling raises pickle.PicklingError.

Solutions

1. Replace the StringIO/BytesIO reference with a real file opened by path (it will have name and mode).
2. Remove the file object from the pickled closure; open it inside the DoFn/function at runtime.
3. Pass the file path string through closure and reopen it on the worker.
4. Write buffer contents to a temp file first: tempfile.NamedTemporaryFile(delete=False).

Example fix

// before
buf = io.StringIO('data')
do_fn_capturing(buf)  # pickling fails
// after
def load():
  return io.StringIO('data')  # construct at runtime inside the function

if not (hasattr(obj, 'name') and hasattr(obj, 'mode')):
    raise TypeError('object cannot be cloudpickled as a file')

def is_picklable_file(obj):
    return hasattr(obj, 'name') and hasattr(obj, 'mode') and not isinstance(obj, (io.StringIO, io.BytesIO))

try:
    cloudpickle.dump(payload)
except pickle.PicklingError:
    payload = strip_file_objects(payload)

When it happens

Trigger: Cloudpickling a task graph / function closure that references an open file-like object without name and mode attributes, such as io.StringIO(), io.BytesIO(), or a wrapped stream.

Common situations: Capturing an in-memory buffer in a lambda or DoFn closure that Beam serializes; passing StringIO fixtures into remote-executed code; wrapping files in adapter classes that hide name/mode.

Understand the failure class

Background: "JSON serialization failed", "not JSON serializable", "Failed to serialize": why JSON marshaling errors happen and how to fix them — this error's family across 46 libraries.

Related errors

• Cannot pickle closed files
• Cannot pickle file as it cannot be read
• Cannot pickle files that are not opened for reading
• Cannot pickle files that map to tty objects
• Cannot pickle standard input

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/e120945d4ebd9f5e. Report an issue: GitHub.

Appendix: source

Thrown at sdks/python/apache_beam/internal/cloudpickle/cloudpickle.py:1071

  try:
    obj.cell_contents
  except ValueError:  # cell is empty
    return _make_empty_cell, ()
  else:
    return _make_cell, (obj.cell_contents, )


def _classmethod_reduce(obj):
  orig_func = obj.__func__
  return type(obj), (orig_func, )


def _file_reduce(obj):
  """Save a file."""
  import io

  if not hasattr(obj, "name") or not hasattr(obj, "mode"):
    raise pickle.PicklingError(
        "Cannot pickle files that do not map to an actual file")
  if obj is sys.stdout:
    return getattr, (sys, "stdout")
  if obj is sys.stderr:
    return getattr, (sys, "stderr")
  if obj is sys.stdin:
    raise pickle.PicklingError("Cannot pickle standard input")
  if obj.closed:
    raise pickle.PicklingError("Cannot pickle closed files")
  if hasattr(obj, "isatty") and obj.isatty():
    raise pickle.PicklingError("Cannot pickle files that map to tty objects")
  if "r" not in obj.mode and "+" not in obj.mode:
    raise pickle.PicklingError(
        "Cannot pickle files that are not opened for reading: %s" % obj.mode)

  name = obj.name

  retval = io.StringIO()

View on GitHub (pinned to 12126d8942)