apache/beam · error · PicklingError

Cannot pickle file as it cannot be read

Error message

Cannot pickle file %s as it cannot be read

What it means

When pickling a file, cloudpickle tries to read its entire contents (seek(0), read(), seek back). If that raises OSError, the file cannot be captured, and _file_reduce re-raises as pickle.PicklingError('Cannot pickle file <name> as it cannot be read'), chaining the original OSError.

Solutions

1. Verify the file still exists and is readable (os.access(path, os.R_OK)); reopen it if deleted.
2. Copy the file to a stable local temp location and pickle a handle/path to that.
3. Avoid pickling handles to special files (pipes, devices); read the needed data eagerly instead.
4. Check file permissions and mount status; fix access before running the pipeline.

Example fix

// before
f = open(path)  # file deleted before serialization
defer(fn_capturing(f))
// after
shutil.copy(path, tmp_path)
f = open(tmp_path)
defer(fn_capturing(f))

import os
if not os.access(f.name, os.R_OK) or not os.path.exists(f.name):
    raise IOError(f'file {f.name} not readable at serialization time')

def is_file_readable(obj):
    import os
    return hasattr(obj, 'name') and os.path.exists(obj.name) and os.access(obj.name, os.R_OK)

try:
    cloudpickle.dump(fn)
except pickle.PicklingError as e:
    if 'cannot be read' in str(e): fn = copy_and_reopen(fn)
    else: raise

When it happens

Trigger: Pickling a file handle whose backing file is unreadable: deleted-but-open files, permission changes, special/device files that fail on seek or read, files on unmounted media, or sockets/ fifos opened with a name and read mode.

Common situations: Rotated/deleted log files still held open; files on NFS/volumes that went away between open and pickle; attempting to serialize handles to pipes or character devices; container filesystem changes between pipeline submission and serialization.

Understand the failure class

Background: "failed to read file", EACCES, ENOENT and "could not read <path>" errors: when a program can't read a file from disk — this error's family across 49 libraries.

Related errors

• Cannot pickle closed files
• Cannot pickle files that are not opened for reading
• Cannot pickle files that do not map to an actual file
• Cannot pickle files that map to tty objects
• Cannot pickle standard input

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/26252d8e7671706f. Report an issue: GitHub.

Appendix: source

Thrown at sdks/python/apache_beam/internal/cloudpickle/cloudpickle.py:1098

    raise pickle.PicklingError("Cannot pickle closed files")
  if hasattr(obj, "isatty") and obj.isatty():
    raise pickle.PicklingError("Cannot pickle files that map to tty objects")
  if "r" not in obj.mode and "+" not in obj.mode:
    raise pickle.PicklingError(
        "Cannot pickle files that are not opened for reading: %s" % obj.mode)

  name = obj.name

  retval = io.StringIO()

  try:
    # Read the whole file
    curloc = obj.tell()
    obj.seek(0)
    contents = obj.read()
    obj.seek(curloc)
  except OSError as e:
    raise pickle.PicklingError(
        "Cannot pickle file %s as it cannot be read" % name) from e
  retval.write(contents)
  retval.seek(curloc)

  retval.name = name
  return _file_reconstructor, (retval, )


def _getset_descriptor_reduce(obj):
  return getattr, (obj.__objclass__, obj.__name__)


def _mappingproxy_reduce(obj):
  return types.MappingProxyType, (dict(obj), )


def _memoryview_reduce(obj):
  return bytes, (obj.tobytes(), )

View on GitHub (pinned to 12126d8942)