apache/beam · error · BeamIOError

Rename operation failed

Error message

Rename operation failed

What it means

GcsFileSystem.rename raises BeamIOError('Rename operation failed') when any copy or subsequent delete step of the rename fails. Rename is implemented as copy-then-delete; exceptions from either phase are aggregated per (source, destination) into one BeamIOError.

Solutions

1. Read exception_details to see whether the copy or the delete phase failed and for which pair.
2. Ensure source files exist and are readable, and destination is writable before renaming.
3. Avoid concurrent mutations of the same objects during rename.
4. Re-run rename for the failed pairs after fixing the underlying GCS error.

Example fix

// before
fs.rename(srcs, dsts)
// after
try:
  fs.rename(srcs, dsts)
except BeamIOError as e:
  failed = [src for src, _ in e.exception_details]
  # re-copy/rename only the failed sources

missing = [s for s in sources if not fs.exists(s)]
if missing:
    raise FileNotFoundError(f'missing sources: {missing}')

def valid_gcs_path(p):
    return isinstance(p, str) and p.startswith('gs://') and p.count('/') >= 3

try:
    fs.rename(srcs, dsts)
except BeamIOError as e:
    for (src, dst), err in e.exception_details.items():
        logging.error('rename %s -> %s failed: %s', src, dst, err)

When it happens

Trigger: Calling GcsFileSystem.rename(source_file_names, destination_file_names) where a copy fails (missing source, permissions) or the follow-up delete of the source fails, and then any exception is registered in the exceptions dict.

Common situations: Renaming files concurrently modified by another pipeline stage; source deleted between copy and delete; destination bucket quota or IAM issues.

Related errors

• Copy operation failed
• Delete operation failed
• Checksum operation failed
• dynamic: wrapped StorageException via…
• empty chunk

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/c5c68845d8afed15. Report an issue: GitHub.

Appendix: source

Thrown at sdks/python/apache_beam/io/gcp/gcsfilesystem.py:272

    exceptions = {}
    for batch in gcs_batches:
      copy_statuses = self._gcsIO().copy_batch(batch)
      copy_succeeded = {}
      delete_targets = []
      for src, dest, exception in copy_statuses:
        if exception:
          exceptions[(src, dest)] = exception
        else:
          copy_succeeded[src] = dest
          delete_targets.append(src)
      delete_statuses = self._gcsIO().delete_batch(delete_targets)
      for src, exception in delete_statuses:
        if exception:
          dest = copy_succeeded[src]
          exceptions[(src, dest)] = exception

    if exceptions:
      raise BeamIOError("Rename operation failed", exceptions)

  def exists(self, path):
    """Check if the provided path exists on the FileSystem.

    Args:
      path: string path that needs to be checked.

    Returns: boolean flag indicating if path exists
    """
    return self._gcsIO().exists(path)

  def size(self, path):
    """Get size of path on the FileSystem.

    Args:
      path: string path in question.

    Returns: int size of path according to the FileSystem.

View on GitHub (pinned to 12126d8942)