apache/beam · error · BeamIOError

Rename operation failed

Error message

Rename operation failed

What it means

LocalFileSystem.rename renames each (source, destination) pair via _rename_file, collecting per-pair exceptions into a dict. If any pair failed, it raises BeamIOError('Rename operation failed', exceptions) whose exception_details maps each (source, destination) tuple to the underlying error. This is the batch-level failure signal for local renames in Beam's FileSystem API.

Source

Thrown at sdks/python/apache_beam/io/localfilesystem.py:253

        "be equal in length")
    assert len(source_file_names) == len(destination_file_names), err_msg

    def _rename_file(source, destination):
      """Rename a single file object"""
      try:
        os.rename(source, destination)
      except OSError as err:
        raise IOError(err)

    exceptions = {}
    for source, destination in zip(source_file_names, destination_file_names):
      try:
        _rename_file(source, destination)
      except Exception as e:  # pylint: disable=broad-except
        exceptions[(source, destination)] = e

    if exceptions:
      raise BeamIOError("Rename operation failed", exceptions)

  def exists(self, path):
    """Check if the provided path exists on the FileSystem.

    Args:
      path: string path that needs to be checked.

    Returns: boolean flag indicating if path exists
    """
    return os.path.exists(path)

  def size(self, path):
    """Get size of path on the FileSystem.

    Args:
      path: string path in question.

    Returns: int size of path according to the FileSystem.

View on GitHub (pinned to 12126d8942)

Solutions

1. Inspect BeamIOError.exception_details to find which pairs failed; note that successful renames are not rolled back.
2. Fix the specific cause: correct missing source paths, or replace rename with copy+delete for cross-device moves.
3. Make renames idempotent: skip pairs whose destination already exists from a previous partial run.
4. Check write permission on every destination directory before renaming.
5. Catch BeamIOError and retry only the failed pairs from exception_details.

Example fix

// before
FileSystems.rename(srcs, dsts)  # partial failures lost in one exception
// after
from apache_beam.io.filesystem import BeamIOError
try:
    FileSystems.rename(srcs, dsts)
except BeamIOError as e:
    failed = list(e.exception_details.keys())
    # retry only failed pairs after fixing their cause

import os
assert all(os.path.exists(s) for s in srcs), "some rename sources are missing"

from apache_beam.io.filesystem import BeamIOError
try:
    FileSystems.rename(srcs, dsts)
except BeamIOError as e:
    failed = list(e.exception_details.keys())  # successful ones are NOT rolled back

When it happens

Trigger: Calling LocalFileSystem.rename or FileSystems.rename where at least one os.rename fails: source missing, cross-device move, or permission denied on the destination directory. One failing pair raises the aggregate error even though other pairs may have been renamed already.

Common situations: Batch commit of output files where one source was already renamed/removed by a prior run or concurrent worker; moving files from tmpfs to disk (EXDEV); destination directory not writable.

Understand the failure class

Background: "failed to write file", "Could not save figure", "Error saving remote file" — file write failed: causes and fixes across languages and libraries — this error's family across 38 libraries.

Related errors

• Copy operation failed
• err (re-raised OSError from os.rename)
• Encountered exceptions in finalize_write: %s
• 'libhdfs error in renaming %s to %s' % (source, destination)
• Rename operation failed

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/c1c4f6872c0df07f. Report an issue: GitHub.