apache/beam · error · BeamIOError

Rename operation failed

Error message

Rename operation failed

What it means

Raised by S3FileSystem.rename when at least one of the pairwise S3 rename (copy+delete) operations failed server-side or client-side. BeamIOError carries a dict of (src,dest) -> exception details so callers can see which pairs failed.

Source

Thrown at sdks/python/apache_beam/io/aws/s3filesystem.py:221

    """Rename the files at the source list to the destination list.
    Source and destination lists should be of the same size.

    Args:
      source_file_names: List of file paths that need to be moved
      destination_file_names: List of destination_file_names for the files

    Raises:
      ``BeamIOError``: if any of the rename operations fail
    """
    if not len(source_file_names) == len(destination_file_names):
      message = 'Unable to rename unequal number of sources and destinations'
      raise BeamIOError(message)
    src_dest_pairs = list(zip(source_file_names, destination_file_names))
    results = s3io.S3IO(options=self._options).rename_files(src_dest_pairs)
    exceptions = {(src, dest): error
                  for (src, dest, error) in results if error is not None}
    if exceptions:
      raise BeamIOError("Rename operation failed", exceptions)

  def exists(self, path):
    """Check if the provided path exists on the FileSystem.

    Args:
      path: string path that needs to be checked.

    Returns: boolean flag indicating if path exists
    """
    try:
      return s3io.S3IO(options=self._options).exists(path)
    except Exception as e:  # pylint: disable=broad-except
      raise BeamIOError("exists() operation failed", {path: e})

  def size(self, path):
    """Get size of path on the FileSystem.

    Args:

View on GitHub (pinned to 12126d8942)

Solutions

1. Inspect BeamIOError.exception_details to identify failed (src,dest) pairs and the underlying cause
2. Verify the source objects exist (fs.exists) and the destination bucket/permissions allow copy and delete
3. Retry the failed subset after addressing the cause
4. Check SSE configuration and bucket region compatibility for copy operations

Example fix

// before
fs.rename(sources, dests)  # may raise with partial failures
// after
from apache_beam.io.filesystem import BeamIOError
try:
    fs.rename(sources, dests)
except BeamIOError as e:
    for (src, dest), err in (e.exception_details or {}).items():
        print(f'rename {src} -> {dest} failed: {err}')

try:
    fs.rename(srcs, dsts)
except BeamIOError as e:
    failed = (e.exception_details or {}).items()
    for (src, dst), err in failed:
        handle_failure(src, dst, err)

When it happens

Trigger: Calling rename() where S3 copy/delete fails for any pair: source object missing, destination bucket not writable, SSE/region mismatch making copy fail, or throttling during bulk rename.

Common situations: Renaming files that another process deleted concurrently; renaming across buckets with different permissions; large batch renames hitting S3 rate limits; typo in destination bucket name.

Related errors

• Basepath %r must be S3 path.
• Path %r must be S3 path.
• Invalid path: %s
• List operation failed
• exists() operation failed

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/f7e21a537ac351ed. Report an issue: GitHub.