apache/beam · error · BeamIOError

Rename operation failed.

Error message

Rename operation failed.

What it means

BlobStorageFileSystem.rename() collects per-pair errors returned by the underlying BlobStorageIO.rename_files() call and raises BeamIOError('Rename operation failed.', exceptions) if any pair failed. The exceptions dict maps each failed (source, destination) pair to its underlying error, so callers can inspect which specific blobs failed to rename.

Source

Thrown at sdks/python/apache_beam/io/azure/blobstoragefilesystem.py:218

    Source and destination lists should be of the same size.

    Args:
      source_file_names: List of file paths that need to be moved
      destination_file_names: List of destination_file_names for the files

    Raises:
      ``BeamIOError``: if any of the rename operations fail
    """
    if not len(source_file_names) == len(destination_file_names):
      message = 'Unable to rename unequal number of sources and destinations.'
      raise BeamIOError(message)
    src_dest_pairs = list(zip(source_file_names, destination_file_names))
    results = self._blobstorageIO().rename_files(src_dest_pairs)
    # Retrieve exceptions.
    exceptions = {(src, dest): error
                  for (src, dest, error) in results if error is not None}
    if exceptions:
      raise BeamIOError("Rename operation failed.", exceptions)

  def exists(self, path):
    """Check if the provided path exists on the FileSystem.

    Args:
      path: string path that needs to be checked.

    Returns: boolean flag indicating if path exists
    """
    try:
      return self._blobstorageIO().exists(path)
    except Exception as e:  # pylint: disable=broad-except
      raise BeamIOError("Exists operation failed", {path: e})

  def size(self, path):
    """Get size in bytes of a file on the FileSystem.

    Args:

View on GitHub (pinned to 12126d8942)

Solutions

1. Inspect the BeamIOError.exception_details dict to see which (src, dest) pairs failed and why
2. Verify each source blob exists (fs.exists) and each destination container exists before renaming
3. Check credentials/SAS permissions allow read on source and write (delete/create) on destination
4. Retry the failed subset after transient errors, since already-succeeded pairs do not need re-running

Example fix

// before
fs.rename(srcs, dests)
// after
try:
    fs.rename(srcs, dests)
except BeamIOError as e:
    failed = [pair for pair in e.exception_details]
    retry_pairs = [(s, d) for (s, d) in failed if fs.exists(s)]
    if retry_pairs:
        fs.rename([p[0] for p in retry_pairs], [p[1] for p in retry_pairs])

missing = [s for s in source_file_names if not fs.exists(s)]
if missing:
    raise FileNotFoundError(f"sources missing before rename: {missing}")

try:
    fs.rename(srcs, dests)
except BeamIOError as e:
    failed = e.exception_details  # dict {(src, dest): error}
    for pair, err in failed.items():
        logging.error("rename failed for %s: %s", pair, err)
    # optionally retry the failed subset

When it happens

Trigger: Calling rename() where at least one (src, dest) pair fails at the Azure level: source blob does not exist, destination container does not exist or is not writable, or the storage account is unreachable/unauthorized. The pre-flight length check passed, so the failure is remote.

Common situations: Renaming files discovered earlier that were deleted by another process in the meantime; destination container name typo or missing container; SAS token / credentials lacking write permission on the destination container; transient Azure storage throttling or network errors during bulk renames.

Understand the failure class

Background: "failed to write file", "Could not save figure", "Error saving remote file" — file write failed: causes and fixes across languages and libraries — this error's family across 38 libraries.

Related errors

• Delete operation failed
• Unable to rename unequal number of sources and destinations.
• Exists operation failed
• Size operation failed
• Last updated operation failed

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/5f5525f4598ee139. Report an issue: GitHub.