apache/beam · error · BeamIOError

Copy operation failed

Error message

Copy operation failed

What it means

GcsFileSystem.copy raises BeamIOError('Copy operation failed') when one or more individual GCS copy operations fail. It catches every per-pair exception into an exceptions dict keyed by (source, destination) and aggregates them into a single BeamIOError whose exception_details carry the original errors.

Solutions

1. Inspect BeamIOError.exception_details for the failing (source, destination) pair and fix the underlying cause (missing source, permissions, path format).
2. Verify all source paths exist before copying (filesystem.exists / match).
3. Check destination bucket permissions and that paths match gs://<bucket>/<object>.
4. Retry only the failed pairs from exception_details.

Example fix

// before
fs.copy(sources, dests)
// after
try:
  fs.copy(sources, dests)
except BeamIOError as e:
  for (src, dest), err in e.exception_details.items():
    logging.error('copy %s -> %s failed: %s', src, dest, err)

missing = [s for s in sources if not fs.exists(s)]
if missing:
    raise FileNotFoundError(f'missing sources: {missing}')

def valid_gcs_path(p):
    return isinstance(p, str) and p.startswith('gs://') and p.count('/') >= 3

try:
    fs.copy(sources, dests)
except BeamIOError as e:
    for (src, dst), err in e.exception_details.items():
        logging.error('copy %s -> %s failed: %s', src, dst, err)

When it happens

Trigger: Calling GcsFileSystem.copy(source_file_names, destination_file_names) where at least one source/destination pair fails: source blob missing, permissions denied, invalid gs:// path, or bucket errors during _copy_path.

Common situations: Batch copying many staged files where some were already deleted by a concurrent job; typo'd bucket names; service account lacking storage.objects.create on destination bucket.

Related errors

• Delete operation failed
• Rename operation failed
• Checksum operation failed
• dynamic: wrapped StorageException via…
• empty chunk

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/1f558da44fd4b419. Report an issue: GitHub.

Appendix: source

Thrown at sdks/python/apache_beam/io/gcp/gcsfilesystem.py:225

      """Recursively copy the file tree from the source to the destination
      """
      if not destination.startswith(GCSFileSystem.GCS_PREFIX):
        raise ValueError('Destination %r must be GCS path.' % destination)
      # Use copy_tree if the path ends with / as it is a directory
      if source.endswith('/'):
        self._gcsIO().copytree(source, destination)
      else:
        self._gcsIO().copy(source, destination)

    exceptions = {}
    for source, destination in zip(source_file_names, destination_file_names):
      try:
        _copy_path(source, destination)
      except Exception as e:  # pylint: disable=broad-except
        exceptions[(source, destination)] = e

    if exceptions:
      raise BeamIOError("Copy operation failed", exceptions)

  def rename(self, source_file_names, destination_file_names):
    """Rename the files at the source list to the destination list.
    Source and destination lists should be of the same size.

    Args:
      source_file_names: List of file paths that need to be moved
      destination_file_names: List of destination_file_names for the files

    Raises:
      ``BeamIOError``: if any of the rename operations fail
    """
    err_msg = (
        "source_file_names and destination_file_names should "
        "be equal in length")
    assert len(source_file_names) == len(destination_file_names), err_msg

    gcs_batches = []

View on GitHub (pinned to 12126d8942)