apache/beam · error · PicklingError

Cannot pickle files that are not opened for reading

Error message

Cannot pickle files that are not opened for reading: %s

What it means

_file_reduce only supports pickling files open for reading; if the file's mode contains neither 'r' nor '+', it raises pickle.PicklingError('Cannot pickle files that are not opened for reading: <mode>'). The contents are captured by reading the file, which is impossible for write-only handles.

Solutions

1. Open the file in read mode ('r' or 'r+') if its contents should be serialized.
2. Pass the file path string and reopen with the desired mode inside the worker.
3. Remove the write handle from the pickled object; write output at the destination via sinks (e.g. WriteToText).
4. Use 'a+' or 'r+' if both append and read access are needed.

Example fix

// before
f = open('out.txt', 'w')
fn_capturing(f)
// after
path = 'out.txt'
def write():
  with open(path, 'w') as f: ...

if hasattr(f, 'mode') and 'r' not in f.mode and '+' not in f.mode:
    raise ValueError(f"file {f.name} not open for reading: {f.mode}")

def is_readable_file(obj):
    return hasattr(obj, 'mode') and ('r' in obj.mode or '+' in obj.mode)

try:
    cloudpickle.dump(fn)
except pickle.PicklingError as e:
    if 'not opened for reading' in str(e): fn = reopen_readable(fn)
    else: raise

When it happens

Trigger: Cloudpickling a file opened with mode 'w', 'a', 'x', 'wb' etc. that is captured in a closure or object being serialized by Beam.

Common situations: Log or output files opened for writing that leak into pickled function state; accidentally passing an output handle where an input file was intended; code refactors moving an output file into a captured scope.

Understand the failure class

Background: "JSON serialization failed", "not JSON serializable", "Failed to serialize": why JSON marshaling errors happen and how to fix them — this error's family across 46 libraries.

Related errors

• Cannot pickle closed files
• Cannot pickle file as it cannot be read
• Cannot pickle files that do not map to an actual file
• Cannot pickle files that map to tty objects
• Cannot pickle standard input

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/8ce73e7263cebe05. Report an issue: GitHub.

Appendix: source

Thrown at sdks/python/apache_beam/internal/cloudpickle/cloudpickle.py:1084

def _file_reduce(obj):
  """Save a file."""
  import io

  if not hasattr(obj, "name") or not hasattr(obj, "mode"):
    raise pickle.PicklingError(
        "Cannot pickle files that do not map to an actual file")
  if obj is sys.stdout:
    return getattr, (sys, "stdout")
  if obj is sys.stderr:
    return getattr, (sys, "stderr")
  if obj is sys.stdin:
    raise pickle.PicklingError("Cannot pickle standard input")
  if obj.closed:
    raise pickle.PicklingError("Cannot pickle closed files")
  if hasattr(obj, "isatty") and obj.isatty():
    raise pickle.PicklingError("Cannot pickle files that map to tty objects")
  if "r" not in obj.mode and "+" not in obj.mode:
    raise pickle.PicklingError(
        "Cannot pickle files that are not opened for reading: %s" % obj.mode)

  name = obj.name

  retval = io.StringIO()

  try:
    # Read the whole file
    curloc = obj.tell()
    obj.seek(0)
    contents = obj.read()
    obj.seek(curloc)
  except OSError as e:
    raise pickle.PicklingError(
        "Cannot pickle file %s as it cannot be read" % name) from e
  retval.write(contents)
  retval.seek(curloc)

View on GitHub (pinned to 12126d8942)