apache/beam · error · BeamIOError

Copy operation failed

Error message

Copy operation failed

What it means

LocalFileSystem.copy copies each (source, destination) pair via _copy_path and records any exception in a dict keyed by the pair. If any pair failed, it raises BeamIOError('Copy operation failed', exceptions) where exception_details maps (source, destination) tuples to the underlying IOError. This is the batch-level failure signal for local file copies in Beam's FileSystem API.

Source

Thrown at sdks/python/apache_beam/io/localfilesystem.py:220

            shutil.rmtree(destination)
          else:
            os.remove(destination)
        if os.path.isdir(source):
          shutil.copytree(source, destination)
        else:
          shutil.copy2(source, destination)
      except OSError as err:
        raise IOError(err)

    exceptions = {}
    for source, destination in zip(source_file_names, destination_file_names):
      try:
        _copy_path(source, destination)
      except Exception as e:  # pylint: disable=broad-except
        exceptions[(source, destination)] = e

    if exceptions:
      raise BeamIOError("Copy operation failed", exceptions)

  def rename(self, source_file_names, destination_file_names):
    """Rename the files at the source list to the destination list.
    Source and destination lists should be of the same size.

    Args:
      source_file_names: List of file paths that need to be moved
      destination_file_names: List of destination_file_names for the files

    Raises:
      ``BeamIOError``: if any of the rename operations fail
    """
    err_msg = (
        "source_file_names and destination_file_names should "
        "be equal in length")
    assert len(source_file_names) == len(destination_file_names), err_msg

    def _rename_file(source, destination):

View on GitHub (pinned to 12126d8942)

Solutions

1. Read BeamIOError.exception_details to identify exactly which (source, destination) pairs failed and why.
2. Delete or empty destination paths before rerunning; Beam's local copy removes existing file destinations but not non-empty directories.
3. Fix per-pair causes: recreate deleted sources, grant write permission on destinations.
4. Catch BeamIOError and retry only failed pairs, since successful copies are not rolled back.
5. Pre-validate with FileSystems.match and exists checks on sources and destination parents.

Example fix

// before
FileSystems.copy(srcs, dsts)
// after
import os
pairs = [(s, d) for s, d in zip(srcs, dsts) if os.path.exists(s)]
for d in {os.path.dirname(d) for _, d in pairs}:
    os.makedirs(d, exist_ok=True)
FileSystems.copy([p[0] for p in pairs], [p[1] for p in pairs])

import os
pairs = [(s, d) for s, d in zip(srcs, dsts)]
missing = [s for s, _ in pairs if not os.path.exists(s)]
assert not missing, f"missing sources: {missing}"

from apache_beam.io.filesystem import BeamIOError
try:
    FileSystems.copy(srcs, dsts)
except BeamIOError as e:
    failed_pairs = e.exception_details  # retry only these

When it happens

Trigger: Calling LocalFileSystem.copy or FileSystems.copy with a list of local paths where at least one copy fails: missing source, unwritable destination, or a directory tree copy onto a non-empty destination. Any single failing pair triggers this aggregate error even if other pairs succeeded.

Common situations: Batch staging of pipeline outputs where one file was concurrently deleted; mixed jobs where one destination path is a read-only mount; forgetting to clean an output directory before rerunning a pipeline.

Understand the failure class

Background: "failed to write file", "Could not save figure", "Error saving remote file" — file write failed: causes and fixes across languages and libraries — this error's family across 38 libraries.

Related errors

• err (re-raised OSError during copy)
• Rename operation failed
• err (re-raised OSError from os.makedirs)
• List operation failed
• err (re-raised OSError from os.rename)

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/3f77ff961f2803a7. Report an issue: GitHub.