{"record":{"id":"537a7e89dbc6012d","repo":"apache/beam","slug":"rename-operation-failed-hadoopfilesystem","errorCode":null,"errorMessage":"Rename operation failed","messagePattern":"Rename operation failed","errorType":"exception","errorClass":"BeamIOError","httpStatus":null,"severity":"error","filePath":"sdks/python/apache_beam/io/hadoopfilesystem.py","lineNumber":360,"sourceCode":"    if exceptions:\n      raise BeamIOError('Copy operation failed', exceptions)\n\n  def rename(self, source_file_names, destination_file_names):\n    exceptions = {}\n    for source, destination in zip(source_file_names, destination_file_names):\n      try:\n        _, rel_source = self._parse_url(source)\n        _, rel_destination = self._parse_url(destination)\n        try:\n          self._hdfs_client.rename(rel_source, rel_destination)\n        except hdfs.HdfsError as e:\n          raise BeamIOError(\n              'libhdfs error in renaming %s to %s' % (source, destination), e)\n      except Exception as e:  # pylint: disable=broad-except\n        exceptions[(source, destination)] = e\n\n    if exceptions:\n      raise BeamIOError('Rename operation failed', exceptions)\n\n  def exists(self, url: str) -> bool:\n    \"\"\"Checks existence of url in HDFS.\n\n    Args:\n      url: String in the form hdfs://...\n\n    Returns:\n      True if url exists as a file or directory in HDFS.\n    \"\"\"\n    _, path = self._parse_url(url)\n    return self._exists(path)\n\n  def _exists(self, path):\n    \"\"\"Returns True if path exists as a file or directory in HDFS.\n\n    Args:\n      path: String in the form /...","sourceCodeStart":342,"sourceCodeEnd":378,"githubUrl":"https://github.com/apache/beam/blob/12126d8942aaf848030c478b4c6a28c6af861c66/sdks/python/apache_beam/io/hadoopfilesystem.py#L342-L378","documentation":"HadoopFileSystem.rename processes each (source, destination) pair, catches every exception into a dict, and after the loop raises a single BeamIOError 'Rename operation failed' containing all failures in exception_payload if any pair failed. It is the batch-level summary error for renames.","triggerScenarios":"Any rename pair failing: unparsable URL (_parse_url), libhdfs HdfsError (missing source, missing destination parent, permissions, NameNode unreachable). Raised once per batch call regardless of how many pairs failed.","commonSituations":"Renaming a list of staged output files to final names where some sources were already renamed (double rename); a partial prior failure left names inconsistent; HDFS connectivity problems mid-batch.","solutions":["Inspect BeamIOError.exception_payload to see which (source, destination) pairs failed and why.","Make renames idempotent: skip pairs where the source no longer exists but the destination does (already renamed).","Create destination parent directories before renaming.","Retry only failed pairs with backoff for transient HDFS errors.","Validate all URLs parse as hdfs://server/path before the batch."],"exampleFix":null,"handlingStrategy":"try-catch","validationCode":"for s, d in zip(sources, destinations):\n    if not fs.exists(s):\n        raise FileNotFoundError(f'source missing: {s}')\n    parent = d.rsplit('/', 1)[0]\n    if not fs.exists(parent):\n        fs.mkdirs(parent)","typeGuard":null,"tryCatchPattern":"from apache_beam.io.filesystem import BeamIOError\ntry:\n    fs.rename(sources, destinations)\nexcept BeamIOError as e:\n    failed = e.exception_payload or {}\n    retry_pairs = [p for p in failed if not already_moved(p)]\n    if retry_pairs:\n        fs.rename(*zip(*retry_pairs))\n    raise","preventionTips":["Make rename idempotent: skip pairs where destination exists and source is gone.","Avoid renaming the same batch twice after partial failures.","Pre-create destination directories.","Inspect exception_payload to scope retries to failed pairs only."],"tags":["python","apache-beam","hdfs","rename","batch-operation"],"backgroundTag":"api-error-response","analyzedSha":"12126d8942aaf848030c478b4c6a28c6af861c66","analyzedAt":"2026-09-13T01:50:10.254Z","contentChangedAt":"2026-09-13T01:50:10.254Z","schemaVersion":2},"datasetVersion":"2026-09-14T16:17:12.679Z"}