apache/beam · error · BeamIOError

source_file_names and destination_file_names should be equal

Error message

source_file_names and destination_file_names should be equal in length: %d != %d

What it means

HadoopFileSystem.copy requires the source and destination iterables to have equal length because copies are performed pairwise (zip). If lengths differ, it raises BeamIOError before doing any work, aborting the whole batch.

Source

Thrown at sdks/python/apache_beam/io/hadoopfilesystem.py:298

      mime_type='application/octet-stream',
      compression_type=CompressionTypes.AUTO):
    stream = io.BufferedReader(
        filesystemio.DownloaderStream(HdfsDownloader(self._hdfs_client, path)),
        buffer_size=_DEFAULT_BUFFER_SIZE)
    return self._add_compression(stream, path, mime_type, compression_type)

  def copy(self, source_file_names, destination_file_names):
    """
    It is an error if any file to copy already exists at the destination.

    Raises ``BeamIOError`` if any error occurred.

    Args:
      source_file_names: iterable of URLs.
      destination_file_names: iterable of URLs.
    """
    if len(source_file_names) != len(destination_file_names):
      raise BeamIOError(
          'source_file_names and destination_file_names should '
          'be equal in length: %d != %d' %
          (len(source_file_names), len(destination_file_names)))

    def _copy_file(source, destination):
      with self._open(source) as f1:
        with self._create(destination) as f2:
          while True:
            buf = f1.read(_COPY_BUFFER_SIZE)
            if not buf:
              break
            f2.write(buf)

    def _copy_path(source, destination):
      """Recursively copy the file tree from the source to the destination."""
      if self._hdfs_client.status(
          source)[_FILE_STATUS_TYPE] != _FILE_STATUS_TYPE_DIRECTORY:
        _copy_file(source, destination)

View on GitHub (pinned to 12126d8942)

Solutions

Ensure both lists are derived from the same source list so lengths always match (build destinations with a 1:1 comprehension).
Assert len(source_file_names) == len(destination_file_names) with a clear message before calling copy.
Regenerate the destination list from the filtered source list rather than maintaining two parallel lists.
If selective copying is needed, copy only matched pairs and handle leftovers separately.

Example fix

# before
fs.copy(src_files, dst_files[:len(src_files)-1])
# after
assert len(src_files) == len(dst_files)
fs.copy(src_files, dst_files)

Defensive patterns

Strategy: validation

Validate before calling

if len(source_file_names) != len(destination_file_names):
    raise ValueError(
        f'copy: mismatched lengths {len(source_file_names)} != {len(destination_file_names)}')

Prevention

Derive destinations from the sources with a 1:1 comprehension instead of parallel lists.
Assert equal lengths in helper wrappers around fs.copy.
Rebuild destination lists after any filtering of sources.
Unit-test rename/copy helpers with mismatched input lists.

When it happens

Trigger: Calling fs.copy(sources, destinations) where the two lists have different lengths — e.g. a glob matched 5 inputs but only 4 destination names were built, or one list was filtered but not the other.

Common situations: Dynamically renaming files with a pattern that produces fewer/more destinations than sources; partial failure earlier in a pipeline leaving lists out of sync; manual construction of destination paths where one transformation dropped an element.

Understand the failure class

Background: "Must be a positive integer", "Invalid value", "Unsupported": the invalid-argument-value error family, when a library rejects the value you pass — this error's family across 35 libraries.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/5a0cc74dbc1d5dac. Report an issue: GitHub.