{"record":{"id":"1f558da44fd4b419","repo":"apache/beam","slug":"copy-operation-failed","errorCode":null,"errorMessage":"Copy operation failed","messagePattern":"Copy operation failed","errorType":"exception","errorClass":"BeamIOError","httpStatus":null,"severity":"error","filePath":"sdks/python/apache_beam/io/gcp/gcsfilesystem.py","lineNumber":225,"sourceCode":"      \"\"\"Recursively copy the file tree from the source to the destination\n      \"\"\"\n      if not destination.startswith(GCSFileSystem.GCS_PREFIX):\n        raise ValueError('Destination %r must be GCS path.' % destination)\n      # Use copy_tree if the path ends with / as it is a directory\n      if source.endswith('/'):\n        self._gcsIO().copytree(source, destination)\n      else:\n        self._gcsIO().copy(source, destination)\n\n    exceptions = {}\n    for source, destination in zip(source_file_names, destination_file_names):\n      try:\n        _copy_path(source, destination)\n      except Exception as e:  # pylint: disable=broad-except\n        exceptions[(source, destination)] = e\n\n    if exceptions:\n      raise BeamIOError(\"Copy operation failed\", exceptions)\n\n  def rename(self, source_file_names, destination_file_names):\n    \"\"\"Rename the files at the source list to the destination list.\n    Source and destination lists should be of the same size.\n\n    Args:\n      source_file_names: List of file paths that need to be moved\n      destination_file_names: List of destination_file_names for the files\n\n    Raises:\n      ``BeamIOError``: if any of the rename operations fail\n    \"\"\"\n    err_msg = (\n        \"source_file_names and destination_file_names should \"\n        \"be equal in length\")\n    assert len(source_file_names) == len(destination_file_names), err_msg\n\n    gcs_batches = []","sourceCodeStart":207,"sourceCodeEnd":243,"githubUrl":"https://github.com/apache/beam/blob/12126d8942aaf848030c478b4c6a28c6af861c66/sdks/python/apache_beam/io/gcp/gcsfilesystem.py#L207-L243","documentation":"GcsFileSystem.copy raises BeamIOError('Copy operation failed') when one or more individual GCS copy operations fail. It catches every per-pair exception into an exceptions dict keyed by (source, destination) and aggregates them into a single BeamIOError whose exception_details carry the original errors.","triggerScenarios":"Calling GcsFileSystem.copy(source_file_names, destination_file_names) where at least one source/destination pair fails: source blob missing, permissions denied, invalid gs:// path, or bucket errors during _copy_path.","commonSituations":"Batch copying many staged files where some were already deleted by a concurrent job; typo'd bucket names; service account lacking storage.objects.create on destination bucket.","solutions":["Inspect BeamIOError.exception_details for the failing (source, destination) pair and fix the underlying cause (missing source, permissions, path format).","Verify all source paths exist before copying (filesystem.exists / match).","Check destination bucket permissions and that paths match gs://<bucket>/<object>.","Retry only the failed pairs from exception_details."],"exampleFix":"// before\nfs.copy(sources, dests)\n// after\ntry:\n  fs.copy(sources, dests)\nexcept BeamIOError as e:\n  for (src, dest), err in e.exception_details.items():\n    logging.error('copy %s -> %s failed: %s', src, dest, err)","handlingStrategy":"try-catch","validationCode":"missing = [s for s in sources if not fs.exists(s)]\nif missing:\n    raise FileNotFoundError(f'missing sources: {missing}')","typeGuard":"def valid_gcs_path(p):\n    return isinstance(p, str) and p.startswith('gs://') and p.count('/') >= 3","tryCatchPattern":"try:\n    fs.copy(sources, dests)\nexcept BeamIOError as e:\n    for (src, dst), err in e.exception_details.items():\n        logging.error('copy %s -> %s failed: %s', src, dst, err)","preventionTips":["Check all source paths exist before batch copy","Verify write permission on the destination bucket","Use valid gs://<bucket>/<object> paths","Retry only failed pairs from exception_details"],"tags":["gcs","io","batch-operation","aggregate-error"],"backgroundTag":"file-copy-failed","analyzedSha":"12126d8942aaf848030c478b4c6a28c6af861c66","analyzedAt":"2026-09-13T01:50:10.254Z","contentChangedAt":"2026-09-13T01:50:10.254Z","schemaVersion":2},"datasetVersion":"2026-09-20T03:17:13.778Z"}