apache/beam · error · PicklingError

Cannot pickle closed files

Error message

Cannot pickle closed files

What it means

_file_reduce in Beam's cloudpickle refuses closed file objects: a closed file has no meaningful readable state to reconstruct remotely, so pickling raises pickle.PicklingError('Cannot pickle closed files').

Solutions

1. Keep the file open (or reopen it) before the object is pickled.
2. Store the file path string instead of the handle and open it lazily at use time.
3. Remove the closed handle from the pickled object (set attribute to None or del).
4. Use a class with __getstate__/__setstate__ that serializes the path and reopens on load.

Example fix

// before
with open('f.txt') as f:
  job.state = f  # closed when pickled
// after
job.path = 'f.txt'  # reopen inside worker when needed

if hasattr(f, 'closed') and f.closed:
    raise ValueError('file is closed and cannot be pickled')

def is_open_file(obj):
    return hasattr(obj, 'closed') and not obj.closed

try:
    cloudpickle.dump(state)
except pickle.PicklingError as e:
    if 'closed files' in str(e): state = reopen_files(state)
    else: raise

When it happens

Trigger: Cloudpickling a closure or object holding a file handle after f.close() was called; context-manager-exited files retained in module state; objects caching an exhausted/closed handle.

Common situations: Using 'with open(...)' and then storing the handle in a global or class attribute that later gets serialized; long-lived workers holding stale handles; retry logic re-serializing objects whose files were closed between attempts.

Understand the failure class

Background: "JSON serialization failed", "not JSON serializable", "Failed to serialize": why JSON marshaling errors happen and how to fix them — this error's family across 46 libraries.

Related errors

• Cannot pickle file as it cannot be read
• Cannot pickle files that are not opened for reading
• Cannot pickle files that do not map to an actual file
• Cannot pickle files that map to tty objects
• Cannot pickle standard input

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/aa908678e63113d6. Report an issue: GitHub.

Appendix: source

Thrown at sdks/python/apache_beam/internal/cloudpickle/cloudpickle.py:1080

  orig_func = obj.__func__
  return type(obj), (orig_func, )


def _file_reduce(obj):
  """Save a file."""
  import io

  if not hasattr(obj, "name") or not hasattr(obj, "mode"):
    raise pickle.PicklingError(
        "Cannot pickle files that do not map to an actual file")
  if obj is sys.stdout:
    return getattr, (sys, "stdout")
  if obj is sys.stderr:
    return getattr, (sys, "stderr")
  if obj is sys.stdin:
    raise pickle.PicklingError("Cannot pickle standard input")
  if obj.closed:
    raise pickle.PicklingError("Cannot pickle closed files")
  if hasattr(obj, "isatty") and obj.isatty():
    raise pickle.PicklingError("Cannot pickle files that map to tty objects")
  if "r" not in obj.mode and "+" not in obj.mode:
    raise pickle.PicklingError(
        "Cannot pickle files that are not opened for reading: %s" % obj.mode)

  name = obj.name

  retval = io.StringIO()

  try:
    # Read the whole file
    curloc = obj.tell()
    obj.seek(0)
    contents = obj.read()
    obj.seek(curloc)
  except OSError as e:
    raise pickle.PicklingError(

View on GitHub (pinned to 12126d8942)