{"record":{"id":"5a0cc74dbc1d5dac","repo":"apache/beam","slug":"source-file-names-and-destination-file-names-should-be-equal","errorCode":null,"errorMessage":"source_file_names and destination_file_names should be equal in length: %d != %d","messagePattern":"source_file_names and destination_file_names should be equal in length: (.+?) != (.+?)","errorType":"exception","errorClass":"BeamIOError","httpStatus":null,"severity":"error","filePath":"sdks/python/apache_beam/io/hadoopfilesystem.py","lineNumber":298,"sourceCode":"      mime_type='application/octet-stream',\n      compression_type=CompressionTypes.AUTO):\n    stream = io.BufferedReader(\n        filesystemio.DownloaderStream(HdfsDownloader(self._hdfs_client, path)),\n        buffer_size=_DEFAULT_BUFFER_SIZE)\n    return self._add_compression(stream, path, mime_type, compression_type)\n\n  def copy(self, source_file_names, destination_file_names):\n    \"\"\"\n    It is an error if any file to copy already exists at the destination.\n\n    Raises ``BeamIOError`` if any error occurred.\n\n    Args:\n      source_file_names: iterable of URLs.\n      destination_file_names: iterable of URLs.\n    \"\"\"\n    if len(source_file_names) != len(destination_file_names):\n      raise BeamIOError(\n          'source_file_names and destination_file_names should '\n          'be equal in length: %d != %d' %\n          (len(source_file_names), len(destination_file_names)))\n\n    def _copy_file(source, destination):\n      with self._open(source) as f1:\n        with self._create(destination) as f2:\n          while True:\n            buf = f1.read(_COPY_BUFFER_SIZE)\n            if not buf:\n              break\n            f2.write(buf)\n\n    def _copy_path(source, destination):\n      \"\"\"Recursively copy the file tree from the source to the destination.\"\"\"\n      if self._hdfs_client.status(\n          source)[_FILE_STATUS_TYPE] != _FILE_STATUS_TYPE_DIRECTORY:\n        _copy_file(source, destination)","sourceCodeStart":280,"sourceCodeEnd":316,"githubUrl":"https://github.com/apache/beam/blob/12126d8942aaf848030c478b4c6a28c6af861c66/sdks/python/apache_beam/io/hadoopfilesystem.py#L280-L316","documentation":"HadoopFileSystem.copy requires the source and destination iterables to have equal length because copies are performed pairwise (zip). If lengths differ, it raises BeamIOError before doing any work, aborting the whole batch.","triggerScenarios":"Calling fs.copy(sources, destinations) where the two lists have different lengths — e.g. a glob matched 5 inputs but only 4 destination names were built, or one list was filtered but not the other.","commonSituations":"Dynamically renaming files with a pattern that produces fewer/more destinations than sources; partial failure earlier in a pipeline leaving lists out of sync; manual construction of destination paths where one transformation dropped an element.","solutions":["Ensure both lists are derived from the same source list so lengths always match (build destinations with a 1:1 comprehension).","Assert len(source_file_names) == len(destination_file_names) with a clear message before calling copy.","Regenerate the destination list from the filtered source list rather than maintaining two parallel lists.","If selective copying is needed, copy only matched pairs and handle leftovers separately."],"exampleFix":"# before\nfs.copy(src_files, dst_files[:len(src_files)-1])\n# after\nassert len(src_files) == len(dst_files)\nfs.copy(src_files, dst_files)","handlingStrategy":"validation","validationCode":"if len(source_file_names) != len(destination_file_names):\n    raise ValueError(\n        f'copy: mismatched lengths {len(source_file_names)} != {len(destination_file_names)}')","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Derive destinations from the sources with a 1:1 comprehension instead of parallel lists.","Assert equal lengths in helper wrappers around fs.copy.","Rebuild destination lists after any filtering of sources.","Unit-test rename/copy helpers with mismatched input lists."],"tags":["python","apache-beam","hdfs","argument-validation"],"backgroundTag":"invalid-argument-value","analyzedSha":"12126d8942aaf848030c478b4c6a28c6af861c66","analyzedAt":"2026-09-13T01:50:10.254Z","contentChangedAt":"2026-09-13T01:50:10.254Z","schemaVersion":2},"datasetVersion":"2026-09-14T16:17:12.679Z"}