{"record":{"id":"399a46236e7eae89","repo":"apache/beam","slug":"copy-operation-failed-hadoopfilesystem","errorCode":null,"errorMessage":"Copy operation failed","messagePattern":"Copy operation failed","errorType":"exception","errorClass":"BeamIOError","httpStatus":null,"severity":"error","filePath":"sdks/python/apache_beam/io/hadoopfilesystem.py","lineNumber":343,"sourceCode":"        rel_path = posixpath.relpath(path, source)\n        if rel_path == '.':\n          rel_path = ''\n        for file in files:\n          _copy_file(\n              self._join('', path, file),\n              self._join('', destination, rel_path, file))\n\n    exceptions = {}\n    for source, destination in zip(source_file_names, destination_file_names):\n      try:\n        _, rel_source = self._parse_url(source)\n        _, rel_destination = self._parse_url(destination)\n        _copy_path(rel_source, rel_destination)\n      except Exception as e:  # pylint: disable=broad-except\n        exceptions[(source, destination)] = e\n\n    if exceptions:\n      raise BeamIOError('Copy operation failed', exceptions)\n\n  def rename(self, source_file_names, destination_file_names):\n    exceptions = {}\n    for source, destination in zip(source_file_names, destination_file_names):\n      try:\n        _, rel_source = self._parse_url(source)\n        _, rel_destination = self._parse_url(destination)\n        try:\n          self._hdfs_client.rename(rel_source, rel_destination)\n        except hdfs.HdfsError as e:\n          raise BeamIOError(\n              'libhdfs error in renaming %s to %s' % (source, destination), e)\n      except Exception as e:  # pylint: disable=broad-except\n        exceptions[(source, destination)] = e\n\n    if exceptions:\n      raise BeamIOError('Rename operation failed', exceptions)\n","sourceCodeStart":325,"sourceCodeEnd":361,"githubUrl":"https://github.com/apache/beam/blob/12126d8942aaf848030c478b4c6a28c6af861c66/sdks/python/apache_beam/io/hadoopfilesystem.py#L325-L361","documentation":"HadoopFileSystem.copy collects per-pair exceptions into a dict keyed by (source, destination); if any pair failed it raises a single BeamIOError 'Copy operation failed' carrying all failures in exception_payload. It indicates one or more individual file copies failed after the whole batch was attempted.","triggerScenarios":"Any per-file failure during copy: source file does not exist, destination parent directory missing, HDFS permission denied, NameNode connectivity error, or an unparsable source/destination URL inside the loop.","commonSituations":"Copying a batch where one input was deleted between match() and copy(); destination directories never created; permission differences between reading and writing users; transient HDFS outages during large batch copies.","solutions":["Read BeamIOError.exception_payload to find exactly which (source, destination) pairs failed and the underlying causes.","Re-run copy only for the failed pairs after fixing the root cause (missing source, missing parent dir, permissions).","Create destination parent directories (mkdirs) before copying.","Verify all source URLs exist (fs.exists) before starting the batch.","Add retry with backoff around copy for transient HDFS errors."],"exampleFix":null,"handlingStrategy":"try-catch","validationCode":"missing = [s for s in source_file_names if not fs.exists(s)]\nif missing:\n    raise FileNotFoundError(f'Sources missing: {missing}')","typeGuard":null,"tryCatchPattern":"from apache_beam.io.filesystem import BeamIOError\ntry:\n    fs.copy(sources, destinations)\nexcept BeamIOError as e:\n    failed_pairs = e.exception_payload or {}\n    for (src, dst), cause in failed_pairs.items():\n        logger.error('copy failed %s -> %s: %r', src, dst, cause)\n    raise","preventionTips":["Create destination parent directories before batch copy.","Re-check source existence right before copying if files are ephemeral.","Retry failed pairs (from exception_payload) with backoff.","Verify permissions on the destination tree."],"tags":["python","apache-beam","hdfs","batch-operation"],"backgroundTag":"file-write-failed","analyzedSha":"12126d8942aaf848030c478b4c6a28c6af861c66","analyzedAt":"2026-09-13T01:50:10.254Z","contentChangedAt":"2026-09-13T01:50:10.254Z","schemaVersion":2},"datasetVersion":"2026-09-14T16:17:12.679Z"}