apache/beam · error · BeamIOError

Unable to rename unequal number of sources and destinations

Error message

Unable to rename unequal number of sources and destinations

What it means

Raised by S3FileSystem.rename when the source and destination path lists passed to BeamIO rename have different lengths. Beam's FileSystem.rename performs pairwise renames, so a mismatch means the operation cannot be mapped. The library fails fast with BeamIOError before touching S3.

Source

Thrown at sdks/python/apache_beam/io/aws/s3filesystem.py:215

      message = 'Unable to copy unequal number of sources and destinations'
      raise BeamIOError(message)
    src_dest_pairs = list(zip(source_file_names, destination_file_names))
    return s3io.S3IO(options=self._options).copy_paths(src_dest_pairs)

  def rename(self, source_file_names, destination_file_names):
    """Rename the files at the source list to the destination list.
    Source and destination lists should be of the same size.

    Args:
      source_file_names: List of file paths that need to be moved
      destination_file_names: List of destination_file_names for the files

    Raises:
      ``BeamIOError``: if any of the rename operations fail
    """
    if not len(source_file_names) == len(destination_file_names):
      message = 'Unable to rename unequal number of sources and destinations'
      raise BeamIOError(message)
    src_dest_pairs = list(zip(source_file_names, destination_file_names))
    results = s3io.S3IO(options=self._options).rename_files(src_dest_pairs)
    exceptions = {(src, dest): error
                  for (src, dest, error) in results if error is not None}
    if exceptions:
      raise BeamIOError("Rename operation failed", exceptions)

  def exists(self, path):
    """Check if the provided path exists on the FileSystem.

    Args:
      path: string path that needs to be checked.

    Returns: boolean flag indicating if path exists
    """
    try:
      return s3io.S3IO(options=self._options).exists(path)
    except Exception as e:  # pylint: disable=broad-except

View on GitHub (pinned to 12126d8942)

Solutions

1. Verify len(source_file_names) == len(destination_file_names) before calling rename
2. Generate destinations by mapping over the source list (e.g. [src.replace(a,b) for src in sources]) instead of maintaining a separate list
3. If intentional, call rename per-pair for the subset that matches

Example fix

// before
fs.rename(sorted_sources, destinations)  # lengths differ
// after
assert len(sorted_sources) == len(destinations)
fs.rename(sorted_sources, destinations)

if len(sources) != len(destinations):
    raise ValueError(f'length mismatch: {len(sources)} sources vs {len(destinations)} destinations')

When it happens

Trigger: Calling S3FileSystem.rename(source_file_names, destination_file_names) with len(source_file_names) != len(destination_file_names), e.g. after filtering one list without the other.

Common situations: Bulk rename jobs built by collecting matching files then pairing them with a separately computed destination list; a filter on one list only; off-by-one slicing; paths deduplicated on one side.

Understand the failure class

Background: "Must be a positive integer", "Invalid value", "Unsupported": the invalid-argument-value error family, when a library rejects the value you pass — this error's family across 35 libraries.

Related errors

• Need to set the filepattern of a TFRecordIO.Read transform
• expected %d, but got %d
• Expecting exactly one field, found
• offset is negative:
• offset exceeds the buffer limit. offset: , limit:

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/b65b5a9fec520266. Report an issue: GitHub.