apache/beam · error · IOError

err (re-raised OSError during copy)

Error message

err (re-raised OSError during copy)

What it means

During LocalFileSystem.copy, the helper _copy_path copies one file (shutil.copy2) or directory tree (shutil.copytree), overwriting an existing destination after os.remove. Any OSError raised by these operations is converted to IOError(err) with the OS error attached. This IOError is then captured per (source, destination) pair and aggregated into the copy-level BeamIOError.

Source

Thrown at sdks/python/apache_beam/io/localfilesystem.py:210

        "source_file_names and destination_file_names should "
        "be equal in length")
    assert len(source_file_names) == len(destination_file_names), err_msg

    def _copy_path(source, destination):
      """Recursively copy the file tree from the source to the destination
      """
      try:
        if os.path.exists(destination):
          if os.path.isdir(destination):
            shutil.rmtree(destination)
          else:
            os.remove(destination)
        if os.path.isdir(source):
          shutil.copytree(source, destination)
        else:
          shutil.copy2(source, destination)
      except OSError as err:
        raise IOError(err)

    exceptions = {}
    for source, destination in zip(source_file_names, destination_file_names):
      try:
        _copy_path(source, destination)
      except Exception as e:  # pylint: disable=broad-except
        exceptions[(source, destination)] = e

    if exceptions:
      raise BeamIOError("Copy operation failed", exceptions)

  def rename(self, source_file_names, destination_file_names):
    """Rename the files at the source list to the destination list.
    Source and destination lists should be of the same size.

    Args:
      source_file_names: List of file paths that need to be moved
      destination_file_names: List of destination_file_names for the files

View on GitHub (pinned to 12126d8942)

Solutions

1. Inspect the chained IOError/exception_details for the failing (source, destination) pair to see the exact OS error.
2. Ensure destination directories are empty or removed before copying directory trees.
3. Check write permissions on the destination and existence of each source before calling copy.
4. Catch BeamIOError and use its exception_details keys to skip or retry only the failed pairs.
5. Copy files individually (pair-by-pair) if you need partial-success semantics rather than an aggregate failure.

Example fix

// before
FileSystems.copy([src], [dst])  # all-or-nothing failure, unclear cause
// after
from apache_beam.io.filesystem import BeamIOError
try:
    FileSystems.copy([src], [dst])
except BeamIOError as e:
    for pair, err in e.exception_details.items():
        print('copy failed:', pair, err)

import os
assert os.path.exists(src), f"missing source: {src}"
assert os.access(os.path.dirname(dst) or '.', os.W_OK), f"destination not writable: {dst}"

from apache_beam.io.filesystem import BeamIOError
try:
    fs.copy([src], [dst])
except BeamIOError as e:
    print(e.exception_details)  # (src, dst) -> underlying error

When it happens

Trigger: Copying a source that does not exist (FileNotFoundError), a destination that cannot be removed or written (PermissionError), copying a directory onto an existing non-empty destination via copytree (FileExistsError/directory not empty), or source and destination on filesystems that refuse some operation (e.g. copying with metadata across special mounts).

Common situations: Staging local files over an existing output dir that was not cleaned up; source file deleted by another process between listing and copy; running the pipeline as a user lacking write access to the destination directory.

Understand the failure class

Background: "failed to write file", "Could not save figure", "Error saving remote file" — file write failed: causes and fixes across languages and libraries — this error's family across 38 libraries.

Related errors

• Copy operation failed
• err (re-raised OSError from os.makedirs)
• List operation failed
• err (re-raised OSError from os.rename)
• Rename operation failed

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/21972efc903cfdaa. Report an issue: GitHub.