apache/beam · error · PicklingError

Cannot pickle standard input

Error message

Cannot pickle standard input

What it means

Beam's cloudpickle _file_reduce refuses to pickle sys.stdin, because the worker process's standard input cannot be meaningfully reconstructed on another machine. Pickling any closure or object referencing sys.stdin raises pickle.PicklingError('Cannot pickle standard input').

Solutions

1. Remove the sys.stdin reference from the code being pickled; read stdin in the main process and pass data (or its path) instead.
2. Read stdin into a variable/string before defining the pickled closure.
3. Open a named file (e.g. /dev/stdin path may still fail; prefer a real temp file) and reference it by path inside the worker.
4. Restructure so worker code receives data via pipeline inputs, not stdin.

Example fix

// before
class Reader: src = sys.stdin
// after
class Reader: src = open('input.txt')  # or pass data via pipeline

import sys
if any(v is sys.stdin for v in closure_vars):
    raise ValueError('sys.stdin cannot be pickled; pass data instead')

def references_stdin(obj):
    return obj is sys.stdin

try:
    cloudpickle.dump(fn)
except pickle.PicklingError as e:
    if 'standard input' in str(e): fn = rewrite_without_stdin(fn)
    else: raise

When it happens

Trigger: Cloudpickling an object/closure that captures sys.stdin directly; a function defined at module level that closes over stdin; serializing an object whose __reduce__ walks into sys.stdin.

Common situations: Pipelines run with stdin redirection/piped input where code references sys.stdin; interactive notebooks capturing stdin; CLI tools embedding stdin readers in Beam DoFns.

Understand the failure class

Background: "JSON serialization failed", "not JSON serializable", "Failed to serialize": why JSON marshaling errors happen and how to fix them — this error's family across 46 libraries.

Related errors

• Cannot pickle closed files
• Cannot pickle file as it cannot be read
• Cannot pickle files that are not opened for reading
• Cannot pickle files that do not map to an actual file
• Cannot pickle files that map to tty objects

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/c5e3dfc68f7f3d78. Report an issue: GitHub.

Appendix: source

Thrown at sdks/python/apache_beam/internal/cloudpickle/cloudpickle.py:1078

def _classmethod_reduce(obj):
  orig_func = obj.__func__
  return type(obj), (orig_func, )


def _file_reduce(obj):
  """Save a file."""
  import io

  if not hasattr(obj, "name") or not hasattr(obj, "mode"):
    raise pickle.PicklingError(
        "Cannot pickle files that do not map to an actual file")
  if obj is sys.stdout:
    return getattr, (sys, "stdout")
  if obj is sys.stderr:
    return getattr, (sys, "stderr")
  if obj is sys.stdin:
    raise pickle.PicklingError("Cannot pickle standard input")
  if obj.closed:
    raise pickle.PicklingError("Cannot pickle closed files")
  if hasattr(obj, "isatty") and obj.isatty():
    raise pickle.PicklingError("Cannot pickle files that map to tty objects")
  if "r" not in obj.mode and "+" not in obj.mode:
    raise pickle.PicklingError(
        "Cannot pickle files that are not opened for reading: %s" % obj.mode)

  name = obj.name

  retval = io.StringIO()

  try:
    # Read the whole file
    curloc = obj.tell()
    obj.seek(0)
    contents = obj.read()
    obj.seek(curloc)

View on GitHub (pinned to 12126d8942)