{"record":{"id":"668a4401e9ea0cdf","repo":"apache/beam","slug":"unable-to-copy-unequal-number-of-sources-and-destinations","errorCode":null,"errorMessage":"Unable to copy unequal number of sources and destinations","messagePattern":"Unable to copy unequal number of sources and destinations","errorType":"validation","errorClass":"BeamIOError","httpStatus":null,"severity":"error","filePath":"sdks/python/apache_beam/io/aws/s3filesystem.py","lineNumber":198,"sourceCode":"      compression_type: Type of compression to be used for this object\n\n    Returns: file handle with a close function for the user to use\n    \"\"\"\n    return self._path_open(path, 'rb', mime_type, compression_type)\n\n  def copy(self, source_file_names, destination_file_names):\n    \"\"\"Recursively copy the file tree from the source to the destination\n\n    Args:\n      source_file_names: list of source file objects that needs to be copied\n      destination_file_names: list of destination of the new object\n\n    Raises:\n      ``BeamIOError``: if any of the copy operations fail\n    \"\"\"\n    if not len(source_file_names) == len(destination_file_names):\n      message = 'Unable to copy unequal number of sources and destinations'\n      raise BeamIOError(message)\n    src_dest_pairs = list(zip(source_file_names, destination_file_names))\n    return s3io.S3IO(options=self._options).copy_paths(src_dest_pairs)\n\n  def rename(self, source_file_names, destination_file_names):\n    \"\"\"Rename the files at the source list to the destination list.\n    Source and destination lists should be of the same size.\n\n    Args:\n      source_file_names: List of file paths that need to be moved\n      destination_file_names: List of destination_file_names for the files\n\n    Raises:\n      ``BeamIOError``: if any of the rename operations fail\n    \"\"\"\n    if not len(source_file_names) == len(destination_file_names):\n      message = 'Unable to rename unequal number of sources and destinations'\n      raise BeamIOError(message)\n    src_dest_pairs = list(zip(source_file_names, destination_file_names))","sourceCodeStart":180,"sourceCodeEnd":216,"githubUrl":"https://github.com/apache/beam/blob/12126d8942aaf848030c478b4c6a28c6af861c66/sdks/python/apache_beam/io/aws/s3filesystem.py#L180-L216","documentation":"S3FileSystem.copy() raises BeamIOError('Unable to copy unequal number of sources and destinations') when the source_file_names and destination_file_names lists differ in length. The method performs batched one-to-one copies, so mismatched list sizes indicate a programming error and are rejected up front.","triggerScenarios":"Calling S3FileSystem().copy(src_list, dest_list) where len(src_list) != len(dest_list) — e.g. renaming a glob of files to fewer/more destinations, filtering sources after building destinations, or concatenating partial lists.","commonSituations":"Bulk rename/copy helpers mapping matched files to template destinations where pattern substitution dropped or duplicated entries; refactor that filters one list but not the other; committing to S3FileSystem.rename which shares the same length contract.","solutions":["Assert len(source_file_names) == len(destination_file_names) before calling copy and log both lengths on mismatch.","Build source/destination lists together in a single loop (zip them at construction time) so they can't diverge.","If counts legitimately differ, call copy once per pair instead of relying on paired batch copying.","Catch BeamIOError and surface which stage produced the mismatched lists."],"exampleFix":"// before\ndests = [t.replace('.tmp', '.avro') for t in srcs if not t.endswith('_tmp')]  # filter breaks pairing\nfs.copy(srcs, dests)\n// after\npairs = [(s, s.replace('.tmp', '.avro')) for s in srcs]\nfs.copy([s for s, _ in pairs], [d for _, d in pairs])","handlingStrategy":"validation","validationCode":"def assert_paired(srcs, dests):\n    assert len(srcs) == len(dests), f\"srcs={len(srcs)} dests={len(dests)}\"","typeGuard":null,"tryCatchPattern":"try:\n    fs.copy(source_file_names, destination_file_names)\nexcept BeamIOError as e:\n    if 'unequal number of sources' in str(e):\n        logging.error('src=%d dest=%d', len(source_file_names), len(destination_file_names))\n    raise","preventionTips":["Build source/destination pairs in a single loop and zip them","Assert equal lengths in tests for bulk copy/rename helpers","Apply any filtering to pairs, not to one list independently"],"tags":["s3","aws","beam-io-error","copy","argument-count"],"backgroundTag":"missing-required-argument","analyzedSha":"12126d8942aaf848030c478b4c6a28c6af861c66","analyzedAt":"2026-09-13T01:50:10.254Z","contentChangedAt":"2026-09-13T01:50:10.254Z","schemaVersion":2},"datasetVersion":"2026-09-14T16:17:12.679Z"}