apache/beam · error · BeamIOError

Unable to copy unequal number of sources and destinations.

Error message

Unable to copy unequal number of sources and destinations.

What it means

BlobStorageFileSystem.copy requires source_file_names and destination_file_names to be the same length; otherwise it raises BeamIOError before touching storage. Batch copy/rename operations are defined pairwise, so mismatched lists are a caller bug.

Solutions

1. Assert len(source_file_names) == len(destination_file_names) before calling copy/rename.
2. Filter both lists together (zip then unzip) so they stay aligned.
3. Compute destinations via a transformation applied to sources instead of building them independently.

Example fix

// before
fsys.copy(sources, [rename(s) for s in sources if skip(s) == False])  # length mismatch
// after
pairs = [(s, rename(s)) for s in sources if not skip(s)]
srcs, dests = zip(*pairs)
fsys.copy(list(srcs), list(dests))

if len(source_file_names) != len(destination_file_names):
    raise ValueError('sources and destinations must be equal length')

def is_balanced_pair(srcs, dests):
    return len(srcs) == len(dests)

try:
    fsys.copy(srcs, dests)
except BeamIOError as e:
    logging.error("copy failed: %s", e)
    raise

When it happens

Trigger: Calling copy(srcs, dests) where len(srcs) != len(dests), e.g. building destinations with a filter applied to sources, or renaming a subset of matched files without trimming the destination list.

Common situations: Generating destination names in a loop that skips duplicates while sources are unfiltered; concatenating lists on one side only; off-by-one when appending destinations conditionally.

Understand the failure class

Background: "Must be a positive integer", "Invalid value", "Unsupported": the invalid-argument-value error family, when a library rejects the value you pass — this error's family across 35 libraries.

Related errors

• Basepath %r must be an Azure Blob Storage path.
• Invalid path
• List operation failed
• Path %r must be Azure Blob Storage path.
• delete does not delete containers.

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/70a50b2ed9d16102. Report an issue: GitHub.

Appendix: source

Thrown at sdks/python/apache_beam/io/azure/blobstoragefilesystem.py:194

      compression_type: Type of compression to be used for this object

    Returns: file handle with a close function for the user to use
    """
    return self._path_open(path, 'rb', mime_type, compression_type)

  def copy(self, source_file_names, destination_file_names):
    """Recursively copy the file tree from the source to the destination

    Args:
      source_file_names: list of source file objects that needs to be copied
      destination_file_names: list of destination of the new object

    Raises:
      ``BeamIOError``: if any of the copy operations fail
    """
    if not len(source_file_names) == len(destination_file_names):
      message = 'Unable to copy unequal number of sources and destinations.'
      raise BeamIOError(message)
    src_dest_pairs = list(zip(source_file_names, destination_file_names))
    return self._blobstorageIO().copy_paths(src_dest_pairs)

  def rename(self, source_file_names, destination_file_names):
    """Rename the files at the source list to the destination list.
    Source and destination lists should be of the same size.

    Args:
      source_file_names: List of file paths that need to be moved
      destination_file_names: List of destination_file_names for the files

    Raises:
      ``BeamIOError``: if any of the rename operations fail
    """
    if not len(source_file_names) == len(destination_file_names):
      message = 'Unable to rename unequal number of sources and destinations.'
      raise BeamIOError(message)
    src_dest_pairs = list(zip(source_file_names, destination_file_names))

View on GitHub (pinned to 12126d8942)