apache/beam · error · BeamIOError

Rename operation failed

Error message

Rename operation failed

What it means

HadoopFileSystem.rename processes each (source, destination) pair, catches every exception into a dict, and after the loop raises a single BeamIOError 'Rename operation failed' containing all failures in exception_payload if any pair failed. It is the batch-level summary error for renames.

Source

Thrown at sdks/python/apache_beam/io/hadoopfilesystem.py:360

    if exceptions:
      raise BeamIOError('Copy operation failed', exceptions)

  def rename(self, source_file_names, destination_file_names):
    exceptions = {}
    for source, destination in zip(source_file_names, destination_file_names):
      try:
        _, rel_source = self._parse_url(source)
        _, rel_destination = self._parse_url(destination)
        try:
          self._hdfs_client.rename(rel_source, rel_destination)
        except hdfs.HdfsError as e:
          raise BeamIOError(
              'libhdfs error in renaming %s to %s' % (source, destination), e)
      except Exception as e:  # pylint: disable=broad-except
        exceptions[(source, destination)] = e

    if exceptions:
      raise BeamIOError('Rename operation failed', exceptions)

  def exists(self, url: str) -> bool:
    """Checks existence of url in HDFS.

    Args:
      url: String in the form hdfs://...

    Returns:
      True if url exists as a file or directory in HDFS.
    """
    _, path = self._parse_url(url)
    return self._exists(path)

  def _exists(self, path):
    """Returns True if path exists as a file or directory in HDFS.

    Args:
      path: String in the form /...

View on GitHub (pinned to 12126d8942)

Solutions

1. Inspect BeamIOError.exception_payload to see which (source, destination) pairs failed and why.
2. Make renames idempotent: skip pairs where the source no longer exists but the destination does (already renamed).
3. Create destination parent directories before renaming.
4. Retry only failed pairs with backoff for transient HDFS errors.
5. Validate all URLs parse as hdfs://server/path before the batch.

for s, d in zip(sources, destinations):
    if not fs.exists(s):
        raise FileNotFoundError(f'source missing: {s}')
    parent = d.rsplit('/', 1)[0]
    if not fs.exists(parent):
        fs.mkdirs(parent)

from apache_beam.io.filesystem import BeamIOError
try:
    fs.rename(sources, destinations)
except BeamIOError as e:
    failed = e.exception_payload or {}
    retry_pairs = [p for p in failed if not already_moved(p)]
    if retry_pairs:
        fs.rename(*zip(*retry_pairs))
    raise

When it happens

Trigger: Any rename pair failing: unparsable URL (_parse_url), libhdfs HdfsError (missing source, missing destination parent, permissions, NameNode unreachable). Raised once per batch call regardless of how many pairs failed.

Common situations: Renaming a list of staged output files to final names where some sources were already renamed (double rename); a partial prior failure left names inconsistent; HDFS connectivity problems mid-batch.

Related errors

• Copy operation failed
• 'libhdfs error in renaming %s to %s' % (source, destination)
• Encountered exceptions in finalize_write: %s
• Failed to import hdfs. You can ensure it is installed by ins
• pipeline_options is not set

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/537a7e89dbc6012d. Report an issue: GitHub.