apache/beam · error · BeamIOError

Unable to copy unequal number of sources and destinations

Error message

Unable to copy unequal number of sources and destinations

What it means

S3FileSystem.copy() raises BeamIOError('Unable to copy unequal number of sources and destinations') when the source_file_names and destination_file_names lists differ in length. The method performs batched one-to-one copies, so mismatched list sizes indicate a programming error and are rejected up front.

Source

Thrown at sdks/python/apache_beam/io/aws/s3filesystem.py:198

      compression_type: Type of compression to be used for this object

    Returns: file handle with a close function for the user to use
    """
    return self._path_open(path, 'rb', mime_type, compression_type)

  def copy(self, source_file_names, destination_file_names):
    """Recursively copy the file tree from the source to the destination

    Args:
      source_file_names: list of source file objects that needs to be copied
      destination_file_names: list of destination of the new object

    Raises:
      ``BeamIOError``: if any of the copy operations fail
    """
    if not len(source_file_names) == len(destination_file_names):
      message = 'Unable to copy unequal number of sources and destinations'
      raise BeamIOError(message)
    src_dest_pairs = list(zip(source_file_names, destination_file_names))
    return s3io.S3IO(options=self._options).copy_paths(src_dest_pairs)

  def rename(self, source_file_names, destination_file_names):
    """Rename the files at the source list to the destination list.
    Source and destination lists should be of the same size.

    Args:
      source_file_names: List of file paths that need to be moved
      destination_file_names: List of destination_file_names for the files

    Raises:
      ``BeamIOError``: if any of the rename operations fail
    """
    if not len(source_file_names) == len(destination_file_names):
      message = 'Unable to rename unequal number of sources and destinations'
      raise BeamIOError(message)
    src_dest_pairs = list(zip(source_file_names, destination_file_names))

View on GitHub (pinned to 12126d8942)

Solutions

1. Assert len(source_file_names) == len(destination_file_names) before calling copy and log both lengths on mismatch.
2. Build source/destination lists together in a single loop (zip them at construction time) so they can't diverge.
3. If counts legitimately differ, call copy once per pair instead of relying on paired batch copying.
4. Catch BeamIOError and surface which stage produced the mismatched lists.

Example fix

// before
dests = [t.replace('.tmp', '.avro') for t in srcs if not t.endswith('_tmp')]  # filter breaks pairing
fs.copy(srcs, dests)
// after
pairs = [(s, s.replace('.tmp', '.avro')) for s in srcs]
fs.copy([s for s, _ in pairs], [d for _, d in pairs])

def assert_paired(srcs, dests):
    assert len(srcs) == len(dests), f"srcs={len(srcs)} dests={len(dests)}"

try:
    fs.copy(source_file_names, destination_file_names)
except BeamIOError as e:
    if 'unequal number of sources' in str(e):
        logging.error('src=%d dest=%d', len(source_file_names), len(destination_file_names))
    raise

When it happens

Trigger: Calling S3FileSystem().copy(src_list, dest_list) where len(src_list) != len(dest_list) — e.g. renaming a glob of files to fewer/more destinations, filtering sources after building destinations, or concatenating partial lists.

Common situations: Bulk rename/copy helpers mapping matched files to template destinations where pattern substitution dropped or duplicated entries; refactor that filters one list but not the other; committing to S3FileSystem.rename which shares the same length contract.

Understand the failure class

Background: "missing required argument" and "the following required arguments were not provided": what required-argument errors mean and how to fix them — this error's family across 20 libraries.

Related errors

• List operation failed
• Invalid page token.
• Basepath %r must be S3 path.
• Path %r must be S3 path.
• Invalid path: %s

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/668a4401e9ea0cdf. Report an issue: GitHub.