apache/beam · error · IOError

err (re-raised OSError from os.rename)

Error message

err (re-raised OSError from os.rename)

What it means

Inside LocalFileSystem.rename, the helper _rename_file performs os.rename for a single path pair and converts any OSError into IOError(err). Typical causes are a missing source file (FileNotFoundError) or cross-filesystem rename attempts (Invalid cross-device link, EXDEV), since os.rename cannot move across mount points. The per-file IOError is then aggregated by rename into a BeamIOError.

Source

Thrown at sdks/python/apache_beam/io/localfilesystem.py:243

    Args:
      source_file_names: List of file paths that need to be moved
      destination_file_names: List of destination_file_names for the files

    Raises:
      ``BeamIOError``: if any of the rename operations fail
    """
    err_msg = (
        "source_file_names and destination_file_names should "
        "be equal in length")
    assert len(source_file_names) == len(destination_file_names), err_msg

    def _rename_file(source, destination):
      """Rename a single file object"""
      try:
        os.rename(source, destination)
      except OSError as err:
        raise IOError(err)

    exceptions = {}
    for source, destination in zip(source_file_names, destination_file_names):
      try:
        _rename_file(source, destination)
      except Exception as e:  # pylint: disable=broad-except
        exceptions[(source, destination)] = e

    if exceptions:
      raise BeamIOError("Rename operation failed", exceptions)

  def exists(self, path):
    """Check if the provided path exists on the FileSystem.

    Args:
      path: string path that needs to be checked.

    Returns: boolean flag indicating if path exists

View on GitHub (pinned to 12126d8942)

Solutions

  1. Check the underlying error in the chained IOError: ENOENT means fix the source path, EXDEV means copy+delete instead of rename.
  2. Use shutil.move semantics (copy then remove) when source and destination are on different filesystems.
  3. Verify the source exists (FileSystems.exists / os.path.exists) before renaming and that the destination parent exists and is writable.
  4. Guard against concurrent workers renaming the same file with unique per-worker temp names.
  5. Catch the eventual BeamIOError from rename and inspect exception_details for the failing pair.

Example fix

// before
FileSystems.rename(['/tmp/stage/out'], ['/data/out'])  # EXDEV if mounts differ
// after
import shutil
if os.path.exists('/tmp/stage/out'):
    shutil.move('/tmp/stage/out', '/data/out')  # falls back to copy across devices
Defensive patterns

Strategy: validation

Validate before calling

import os
if os.path.exists(src) and os.path.exists(os.path.dirname(dst)):
    os.rename(src, dst)

Try / catch

try:
    FileSystems.rename([src], [dst])
except Exception as e:
    # EXDEV / missing source -> fall back to copy+delete
    import shutil
    shutil.move(src, dst)

Prevention

When it happens

Trigger: Renaming a file that does not exist; renaming between different filesystems/mount points (e.g. /tmp to /home on separate devices) which raises 'Invalid cross-device link'; renaming onto a destination in a directory without write permission; renaming a directory into a subdirectory of itself.

Common situations: Moving staging output from a tmpfs temp dir to a persistent volume; two workers renaming the same temp file concurrently; mis-typed paths where the source was never created.

Understand the failure class

Background: "File not found" and ENOENT errors: why libraries can't find a file that should exist — this error's family across 50 libraries.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/6739ed6b66844318. Report an issue: GitHub.