{"record":{"id":"21972efc903cfdaa","repo":"apache/beam","slug":"err-re-raised-oserror-during-copy","errorCode":null,"errorMessage":"err (re-raised OSError during copy)","messagePattern":"err \\(re-raised OSError during copy\\)","errorType":"exception","errorClass":"IOError","httpStatus":null,"severity":"error","filePath":"sdks/python/apache_beam/io/localfilesystem.py","lineNumber":210,"sourceCode":"        \"source_file_names and destination_file_names should \"\n        \"be equal in length\")\n    assert len(source_file_names) == len(destination_file_names), err_msg\n\n    def _copy_path(source, destination):\n      \"\"\"Recursively copy the file tree from the source to the destination\n      \"\"\"\n      try:\n        if os.path.exists(destination):\n          if os.path.isdir(destination):\n            shutil.rmtree(destination)\n          else:\n            os.remove(destination)\n        if os.path.isdir(source):\n          shutil.copytree(source, destination)\n        else:\n          shutil.copy2(source, destination)\n      except OSError as err:\n        raise IOError(err)\n\n    exceptions = {}\n    for source, destination in zip(source_file_names, destination_file_names):\n      try:\n        _copy_path(source, destination)\n      except Exception as e:  # pylint: disable=broad-except\n        exceptions[(source, destination)] = e\n\n    if exceptions:\n      raise BeamIOError(\"Copy operation failed\", exceptions)\n\n  def rename(self, source_file_names, destination_file_names):\n    \"\"\"Rename the files at the source list to the destination list.\n    Source and destination lists should be of the same size.\n\n    Args:\n      source_file_names: List of file paths that need to be moved\n      destination_file_names: List of destination_file_names for the files","sourceCodeStart":192,"sourceCodeEnd":228,"githubUrl":"https://github.com/apache/beam/blob/12126d8942aaf848030c478b4c6a28c6af861c66/sdks/python/apache_beam/io/localfilesystem.py#L192-L228","documentation":"During LocalFileSystem.copy, the helper _copy_path copies one file (shutil.copy2) or directory tree (shutil.copytree), overwriting an existing destination after os.remove. Any OSError raised by these operations is converted to IOError(err) with the OS error attached. This IOError is then captured per (source, destination) pair and aggregated into the copy-level BeamIOError.","triggerScenarios":"Copying a source that does not exist (FileNotFoundError), a destination that cannot be removed or written (PermissionError), copying a directory onto an existing non-empty destination via copytree (FileExistsError/directory not empty), or source and destination on filesystems that refuse some operation (e.g. copying with metadata across special mounts).","commonSituations":"Staging local files over an existing output dir that was not cleaned up; source file deleted by another process between listing and copy; running the pipeline as a user lacking write access to the destination directory.","solutions":["Inspect the chained IOError/exception_details for the failing (source, destination) pair to see the exact OS error.","Ensure destination directories are empty or removed before copying directory trees.","Check write permissions on the destination and existence of each source before calling copy.","Catch BeamIOError and use its exception_details keys to skip or retry only the failed pairs.","Copy files individually (pair-by-pair) if you need partial-success semantics rather than an aggregate failure."],"exampleFix":"// before\nFileSystems.copy([src], [dst])  # all-or-nothing failure, unclear cause\n// after\nfrom apache_beam.io.filesystem import BeamIOError\ntry:\n    FileSystems.copy([src], [dst])\nexcept BeamIOError as e:\n    for pair, err in e.exception_details.items():\n        print('copy failed:', pair, err)","handlingStrategy":"try-catch","validationCode":"import os\nassert os.path.exists(src), f\"missing source: {src}\"\nassert os.access(os.path.dirname(dst) or '.', os.W_OK), f\"destination not writable: {dst}\"","typeGuard":null,"tryCatchPattern":"from apache_beam.io.filesystem import BeamIOError\ntry:\n    fs.copy([src], [dst])\nexcept BeamIOError as e:\n    print(e.exception_details)  # (src, dst) -> underlying error","preventionTips":["Pre-check source existence and destination writability for every pair","Clean or empty destination directories before directory-tree copies","Copy pair-by-pair if you need partial-success semantics"],"tags":["python","apache-beam","filesystem","copy","shutil"],"backgroundTag":"file-write-failed","analyzedSha":"12126d8942aaf848030c478b4c6a28c6af861c66","analyzedAt":"2026-09-13T01:50:10.254Z","contentChangedAt":"2026-09-13T01:50:10.254Z","schemaVersion":2},"datasetVersion":"2026-09-14T16:17:12.679Z"}