{"record":{"id":"ec73bc1bc12bba06","repo":"apache/beam","slug":"libhdfs-error-in-renaming-s-to-s-source-destination","errorCode":null,"errorMessage":"'libhdfs error in renaming %s to %s' % (source, destination)","messagePattern":"'libhdfs error in renaming (.+?) to (.+?)' % \\(source, destination\\)","errorType":"exception","errorClass":"BeamIOError","httpStatus":null,"severity":"error","filePath":"sdks/python/apache_beam/io/hadoopfilesystem.py","lineNumber":354,"sourceCode":"        _, rel_source = self._parse_url(source)\n        _, rel_destination = self._parse_url(destination)\n        _copy_path(rel_source, rel_destination)\n      except Exception as e:  # pylint: disable=broad-except\n        exceptions[(source, destination)] = e\n\n    if exceptions:\n      raise BeamIOError('Copy operation failed', exceptions)\n\n  def rename(self, source_file_names, destination_file_names):\n    exceptions = {}\n    for source, destination in zip(source_file_names, destination_file_names):\n      try:\n        _, rel_source = self._parse_url(source)\n        _, rel_destination = self._parse_url(destination)\n        try:\n          self._hdfs_client.rename(rel_source, rel_destination)\n        except hdfs.HdfsError as e:\n          raise BeamIOError(\n              'libhdfs error in renaming %s to %s' % (source, destination), e)\n      except Exception as e:  # pylint: disable=broad-except\n        exceptions[(source, destination)] = e\n\n    if exceptions:\n      raise BeamIOError('Rename operation failed', exceptions)\n\n  def exists(self, url: str) -> bool:\n    \"\"\"Checks existence of url in HDFS.\n\n    Args:\n      url: String in the form hdfs://...\n\n    Returns:\n      True if url exists as a file or directory in HDFS.\n    \"\"\"\n    _, path = self._parse_url(url)\n    return self._exists(path)","sourceCodeStart":336,"sourceCodeEnd":372,"githubUrl":"https://github.com/apache/beam/blob/12126d8942aaf848030c478b4c6a28c6af861c66/sdks/python/apache_beam/io/hadoopfilesystem.py#L336-L372","documentation":"During HadoopFileSystem.rename, if the underlying pyarrow.hdfs client raises hdfs.HdfsError for a pair, it is wrapped into BeamIOError with the message 'libhdfs error in renaming <source> to <destination>'. This specific error is then itself caught by the outer broad handler, aggregated, and re-raised as 'Rename operation failed'.","triggerScenarios":"Calling fs.rename(sources, destinations) where libhdfs cannot perform the rename: source path missing, destination parent directory missing, destination exists as a non-empty directory, or HDFS permission/NameNode failure.","commonSituations":"Atomic commit patterns (move temp dir to final location) where the final path already exists; renaming when the destination parent doesn't exist; HDFS permission issues; stale source paths after upstream steps failed.","solutions":["Check that the source exists and the destination's parent directory exists before renaming (mkdirs the parent).","Ensure the destination path does not already exist, or delete it first (hdfs dfs -rm -r).","Inspect the aggregated BeamIOError.exception_payload for the per-pair HdfsError detail.","Verify HDFS permissions for both source and destination trees.","Confirm both URLs parse correctly as hdfs://server/path (invalid URLs also feed this handler via _parse_url)."],"exampleFix":null,"handlingStrategy":"try-catch","validationCode":"if not fs.exists(source):\n    raise FileNotFoundError(f'rename source missing: {source}')\nparent = destination.rsplit('/', 1)[0]\nif not fs.exists(parent):\n    fs.mkdirs(parent)","typeGuard":null,"tryCatchPattern":"from apache_beam.io.filesystem import BeamIOError\ntry:\n    fs.rename([src], [dst])\nexcept BeamIOError as e:\n    for pair, cause in (e.exception_payload or {}).items():\n        logger.error('rename %s -> %s failed: %r', pair[0], pair[1], cause)\n    raise","preventionTips":["Ensure the destination parent directory exists before rename.","Delete or version the destination path if it may already exist.","Check source existence before renaming (avoid double-rename).","Inspect exception_payload for the underlying hdfs.HdfsError."],"tags":["python","apache-beam","hdfs","rename","libhdfs"],"backgroundTag":"api-error-response","analyzedSha":"12126d8942aaf848030c478b4c6a28c6af861c66","analyzedAt":"2026-09-13T01:50:10.254Z","contentChangedAt":"2026-09-13T01:50:10.254Z","schemaVersion":2},"datasetVersion":"2026-09-14T16:17:12.679Z"}