{"record":{"id":"3f77ff961f2803a7","repo":"apache/beam","slug":"copy-operation-failed-localfilesystem","errorCode":null,"errorMessage":"Copy operation failed","messagePattern":"Copy operation failed","errorType":"exception","errorClass":"BeamIOError","httpStatus":null,"severity":"error","filePath":"sdks/python/apache_beam/io/localfilesystem.py","lineNumber":220,"sourceCode":"            shutil.rmtree(destination)\n          else:\n            os.remove(destination)\n        if os.path.isdir(source):\n          shutil.copytree(source, destination)\n        else:\n          shutil.copy2(source, destination)\n      except OSError as err:\n        raise IOError(err)\n\n    exceptions = {}\n    for source, destination in zip(source_file_names, destination_file_names):\n      try:\n        _copy_path(source, destination)\n      except Exception as e:  # pylint: disable=broad-except\n        exceptions[(source, destination)] = e\n\n    if exceptions:\n      raise BeamIOError(\"Copy operation failed\", exceptions)\n\n  def rename(self, source_file_names, destination_file_names):\n    \"\"\"Rename the files at the source list to the destination list.\n    Source and destination lists should be of the same size.\n\n    Args:\n      source_file_names: List of file paths that need to be moved\n      destination_file_names: List of destination_file_names for the files\n\n    Raises:\n      ``BeamIOError``: if any of the rename operations fail\n    \"\"\"\n    err_msg = (\n        \"source_file_names and destination_file_names should \"\n        \"be equal in length\")\n    assert len(source_file_names) == len(destination_file_names), err_msg\n\n    def _rename_file(source, destination):","sourceCodeStart":202,"sourceCodeEnd":238,"githubUrl":"https://github.com/apache/beam/blob/12126d8942aaf848030c478b4c6a28c6af861c66/sdks/python/apache_beam/io/localfilesystem.py#L202-L238","documentation":"LocalFileSystem.copy copies each (source, destination) pair via _copy_path and records any exception in a dict keyed by the pair. If any pair failed, it raises BeamIOError('Copy operation failed', exceptions) where exception_details maps (source, destination) tuples to the underlying IOError. This is the batch-level failure signal for local file copies in Beam's FileSystem API.","triggerScenarios":"Calling LocalFileSystem.copy or FileSystems.copy with a list of local paths where at least one copy fails: missing source, unwritable destination, or a directory tree copy onto a non-empty destination. Any single failing pair triggers this aggregate error even if other pairs succeeded.","commonSituations":"Batch staging of pipeline outputs where one file was concurrently deleted; mixed jobs where one destination path is a read-only mount; forgetting to clean an output directory before rerunning a pipeline.","solutions":["Read BeamIOError.exception_details to identify exactly which (source, destination) pairs failed and why.","Delete or empty destination paths before rerunning; Beam's local copy removes existing file destinations but not non-empty directories.","Fix per-pair causes: recreate deleted sources, grant write permission on destinations.","Catch BeamIOError and retry only failed pairs, since successful copies are not rolled back.","Pre-validate with FileSystems.match and exists checks on sources and destination parents."],"exampleFix":"// before\nFileSystems.copy(srcs, dsts)\n// after\nimport os\npairs = [(s, d) for s, d in zip(srcs, dsts) if os.path.exists(s)]\nfor d in {os.path.dirname(d) for _, d in pairs}:\n    os.makedirs(d, exist_ok=True)\nFileSystems.copy([p[0] for p in pairs], [p[1] for p in pairs])","handlingStrategy":"try-catch","validationCode":"import os\npairs = [(s, d) for s, d in zip(srcs, dsts)]\nmissing = [s for s, _ in pairs if not os.path.exists(s)]\nassert not missing, f\"missing sources: {missing}\"","typeGuard":null,"tryCatchPattern":"from apache_beam.io.filesystem import BeamIOError\ntry:\n    FileSystems.copy(srcs, dsts)\nexcept BeamIOError as e:\n    failed_pairs = e.exception_details  # retry only these","preventionTips":["Validate all sources exist before batch copy","Ensure destination directories exist and are writable","Treat copy as non-atomic: design for partial success and retry failed pairs from exception_details"],"tags":["python","apache-beam","filesystem","copy","batch"],"backgroundTag":"file-write-failed","analyzedSha":"12126d8942aaf848030c478b4c6a28c6af861c66","analyzedAt":"2026-09-13T01:50:10.254Z","contentChangedAt":"2026-09-13T01:50:10.254Z","schemaVersion":2},"datasetVersion":"2026-09-14T16:17:12.679Z"}