apache/beam · error · BeamIOError

Copy operation failed

Error message

Copy operation failed

What it means

HadoopFileSystem.copy collects per-pair exceptions into a dict keyed by (source, destination); if any pair failed it raises a single BeamIOError 'Copy operation failed' carrying all failures in exception_payload. It indicates one or more individual file copies failed after the whole batch was attempted.

Source

Thrown at sdks/python/apache_beam/io/hadoopfilesystem.py:343

        rel_path = posixpath.relpath(path, source)
        if rel_path == '.':
          rel_path = ''
        for file in files:
          _copy_file(
              self._join('', path, file),
              self._join('', destination, rel_path, file))

    exceptions = {}
    for source, destination in zip(source_file_names, destination_file_names):
      try:
        _, rel_source = self._parse_url(source)
        _, rel_destination = self._parse_url(destination)
        _copy_path(rel_source, rel_destination)
      except Exception as e:  # pylint: disable=broad-except
        exceptions[(source, destination)] = e

    if exceptions:
      raise BeamIOError('Copy operation failed', exceptions)

  def rename(self, source_file_names, destination_file_names):
    exceptions = {}
    for source, destination in zip(source_file_names, destination_file_names):
      try:
        _, rel_source = self._parse_url(source)
        _, rel_destination = self._parse_url(destination)
        try:
          self._hdfs_client.rename(rel_source, rel_destination)
        except hdfs.HdfsError as e:
          raise BeamIOError(
              'libhdfs error in renaming %s to %s' % (source, destination), e)
      except Exception as e:  # pylint: disable=broad-except
        exceptions[(source, destination)] = e

    if exceptions:
      raise BeamIOError('Rename operation failed', exceptions)

View on GitHub (pinned to 12126d8942)

Solutions

1. Read BeamIOError.exception_payload to find exactly which (source, destination) pairs failed and the underlying causes.
2. Re-run copy only for the failed pairs after fixing the root cause (missing source, missing parent dir, permissions).
3. Create destination parent directories (mkdirs) before copying.
4. Verify all source URLs exist (fs.exists) before starting the batch.
5. Add retry with backoff around copy for transient HDFS errors.

missing = [s for s in source_file_names if not fs.exists(s)]
if missing:
    raise FileNotFoundError(f'Sources missing: {missing}')

from apache_beam.io.filesystem import BeamIOError
try:
    fs.copy(sources, destinations)
except BeamIOError as e:
    failed_pairs = e.exception_payload or {}
    for (src, dst), cause in failed_pairs.items():
        logger.error('copy failed %s -> %s: %r', src, dst, cause)
    raise

When it happens

Trigger: Any per-file failure during copy: source file does not exist, destination parent directory missing, HDFS permission denied, NameNode connectivity error, or an unparsable source/destination URL inside the loop.

Common situations: Copying a batch where one input was deleted between match() and copy(); destination directories never created; permission differences between reading and writing users; transient HDFS outages during large batch copies.

Understand the failure class

Background: "failed to write file", "Could not save figure", "Error saving remote file" — file write failed: causes and fixes across languages and libraries — this error's family across 38 libraries.

Related errors

• Rename operation failed
• Failed to import hdfs. You can ensure it is installed by ins
• pipeline_options is not set
• hdfs_host is not set
• hdfs_port is not set

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/399a46236e7eae89. Report an issue: GitHub.