apache/beam · error · BeamIOError

'libhdfs error in renaming %s to %s' % (source, destination)

Error message

'libhdfs error in renaming %s to %s' % (source, destination)

What it means

During HadoopFileSystem.rename, if the underlying pyarrow.hdfs client raises hdfs.HdfsError for a pair, it is wrapped into BeamIOError with the message 'libhdfs error in renaming <source> to <destination>'. This specific error is then itself caught by the outer broad handler, aggregated, and re-raised as 'Rename operation failed'.

Source

Thrown at sdks/python/apache_beam/io/hadoopfilesystem.py:354

        _, rel_source = self._parse_url(source)
        _, rel_destination = self._parse_url(destination)
        _copy_path(rel_source, rel_destination)
      except Exception as e:  # pylint: disable=broad-except
        exceptions[(source, destination)] = e

    if exceptions:
      raise BeamIOError('Copy operation failed', exceptions)

  def rename(self, source_file_names, destination_file_names):
    exceptions = {}
    for source, destination in zip(source_file_names, destination_file_names):
      try:
        _, rel_source = self._parse_url(source)
        _, rel_destination = self._parse_url(destination)
        try:
          self._hdfs_client.rename(rel_source, rel_destination)
        except hdfs.HdfsError as e:
          raise BeamIOError(
              'libhdfs error in renaming %s to %s' % (source, destination), e)
      except Exception as e:  # pylint: disable=broad-except
        exceptions[(source, destination)] = e

    if exceptions:
      raise BeamIOError('Rename operation failed', exceptions)

  def exists(self, url: str) -> bool:
    """Checks existence of url in HDFS.

    Args:
      url: String in the form hdfs://...

    Returns:
      True if url exists as a file or directory in HDFS.
    """
    _, path = self._parse_url(url)
    return self._exists(path)

View on GitHub (pinned to 12126d8942)

Solutions

1. Check that the source exists and the destination's parent directory exists before renaming (mkdirs the parent).
2. Ensure the destination path does not already exist, or delete it first (hdfs dfs -rm -r).
3. Inspect the aggregated BeamIOError.exception_payload for the per-pair HdfsError detail.
4. Verify HDFS permissions for both source and destination trees.
5. Confirm both URLs parse correctly as hdfs://server/path (invalid URLs also feed this handler via _parse_url).

if not fs.exists(source):
    raise FileNotFoundError(f'rename source missing: {source}')
parent = destination.rsplit('/', 1)[0]
if not fs.exists(parent):
    fs.mkdirs(parent)

from apache_beam.io.filesystem import BeamIOError
try:
    fs.rename([src], [dst])
except BeamIOError as e:
    for pair, cause in (e.exception_payload or {}).items():
        logger.error('rename %s -> %s failed: %r', pair[0], pair[1], cause)
    raise

When it happens

Trigger: Calling fs.rename(sources, destinations) where libhdfs cannot perform the rename: source path missing, destination parent directory missing, destination exists as a non-empty directory, or HDFS permission/NameNode failure.

Common situations: Atomic commit patterns (move temp dir to final location) where the final path already exists; renaming when the destination parent doesn't exist; HDFS permission issues; stale source paths after upstream steps failed.

Related errors

• Rename operation failed
• Encountered exceptions in finalize_write: %s
• Failed to import hdfs. You can ensure it is installed by ins
• pipeline_options is not set
• hdfs_host is not set

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/ec73bc1bc12bba06. Report an issue: GitHub.