apache/beam · error · PicklingError

Cannot pickle files that map to tty objects

Error message

Cannot pickle files that map to tty objects

What it means

Beam's cloudpickle will not serialize file objects attached to a terminal (isatty() true), such as stdin/stdout variants or pseudo-terminals, because a tty cannot be reconstructed on a remote worker. Pickling raises pickle.PicklingError('Cannot pickle files that map to tty objects').

Solutions

1. Replace the tty reference with a real file: open a log file and capture to it instead.
2. Read terminal output in the main process; pass data, not the handle, to workers.
3. Use logging module rather than holding a tty stream in serialized state.
4. Strip the attribute before pickling (e.g. in __getstate__).

Example fix

// before
class Job: out = sys.stdout  # tty when run interactively
// after
class Job: out = open('job.log', 'w')

if hasattr(obj, 'isatty') and obj.isatty():
    raise ValueError('tty file object cannot be pickled')

def is_non_tty_file(obj):
    return not (hasattr(obj, 'isatty') and obj.isatty())

try:
    cloudpickle.dump(fn)
except pickle.PicklingError as e:
    if 'tty' in str(e): fn = replace_tty_with_file(fn)
    else: raise

When it happens

Trigger: Cloudpickling a closure capturing a tty file object, e.g. sys.stdout when running under a terminal (if not caught by the earlier identity checks), os.openpty() handles, or pty-backed streams.

Common situations: Running Beam jobs from an interactive shell where code references stdout/stderr-like tty streams; debug logging handles bound to the terminal captured in DoFns; test harnesses using pty objects.

Understand the failure class

Background: "JSON serialization failed", "not JSON serializable", "Failed to serialize": why JSON marshaling errors happen and how to fix them — this error's family across 46 libraries.

Related errors

• Cannot pickle closed files
• Cannot pickle file as it cannot be read
• Cannot pickle files that are not opened for reading
• Cannot pickle files that do not map to an actual file
• Cannot pickle standard input

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/3361636f6a4eb09a. Report an issue: GitHub.

Appendix: source

Thrown at sdks/python/apache_beam/internal/cloudpickle/cloudpickle.py:1082

def _file_reduce(obj):
  """Save a file."""
  import io

  if not hasattr(obj, "name") or not hasattr(obj, "mode"):
    raise pickle.PicklingError(
        "Cannot pickle files that do not map to an actual file")
  if obj is sys.stdout:
    return getattr, (sys, "stdout")
  if obj is sys.stderr:
    return getattr, (sys, "stderr")
  if obj is sys.stdin:
    raise pickle.PicklingError("Cannot pickle standard input")
  if obj.closed:
    raise pickle.PicklingError("Cannot pickle closed files")
  if hasattr(obj, "isatty") and obj.isatty():
    raise pickle.PicklingError("Cannot pickle files that map to tty objects")
  if "r" not in obj.mode and "+" not in obj.mode:
    raise pickle.PicklingError(
        "Cannot pickle files that are not opened for reading: %s" % obj.mode)

  name = obj.name

  retval = io.StringIO()

  try:
    # Read the whole file
    curloc = obj.tell()
    obj.seek(0)
    contents = obj.read()
    obj.seek(curloc)
  except OSError as e:
    raise pickle.PicklingError(
        "Cannot pickle file %s as it cannot be read" % name) from e
  retval.write(contents)

View on GitHub (pinned to 12126d8942)