apache/beam · error · ValueError

Cannot rename a directory

Error message

Cannot rename a directory

What it means

S3IO.rename_files refuses to rename any source or destination path that ends with '/', i.e. a directory/prefix. S3 has no atomic directory rename; only object keys can be copied-and-deleted, so prefix renames are rejected up front.

Solutions

1. Rename individual object keys without trailing slashes.
2. To move a prefix, list all keys under it and copy/delete each key individually.
3. Strip trailing '/' from paths before building src_dest_pairs.
4. Consider using S3 CopyObject in bulk via the AWS CLI / SDK for prefix moves outside Beam.

Example fix

// before
io.rename_files([('s3://bucket/data/', 's3://bucket/data2/')])  # ValueError
// after
keys = io.list_files('s3://bucket/data/')
io.rename_files([(k, k.replace('data/', 'data2/', 1)) for k in keys])

if any(s.endswith('/') or d.endswith('/') for s, d in pairs):
    raise ValueError('S3 rename does not support directories/prefixes')

def is_object_key(path):
    return not path.endswith('/')

try:
    io.rename_files(pairs)
except ValueError:
    # expand prefix into per-object pairs
    pairs = expand_prefix_pairs(pairs)
    io.rename_files(pairs)

When it happens

Trigger: Calling rename_files([('s3://bucket/dir/', 's3://bucket/dir2/')]) or any pair where either key keeps a trailing slash.

Common situations: Users thinking of S3 prefixes as folders and trying to 'move a folder'; code generated from POSIX semantics (mv dir1 dir2); renaming results of list_prefix that retain trailing slashes.

Understand the failure class

Background: UnsupportedOperationException and "is not supported" errors: when a library deliberately refuses a call — this error's family across 30 libraries.

Related errors

• AWS dependencies are not installed, and no alternative…
• Basepath %r must be S3 path.
• ENOENT
• Invalid file open mode
• Invalid path

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/8d0f7663cd64ec4b. Report an issue: GitHub.

Appendix: source

Thrown at sdks/python/apache_beam/io/aws/s3io.py:528

        time.mktime(updated.timetuple()) - time.timezone +
        updated.microsecond / 1000000.0)

  def rename_files(self, src_dest_pairs):
    """Renames the given S3 objects from src to dest.

    Args:
      src_dest_pairs: list of (src, dest) tuples of s3://<bucket>/<name> file
                      paths to rename from src to dest
    Returns: List of tuples of (src, dest, exception) in the same order as the
            src_dest_pairs argument, where exception is None if the operation
            succeeded or the relevant exception if the operation failed.
    """
    if not src_dest_pairs: return []

    # TODO: Throw value error if path has directory
    for src, dest in src_dest_pairs:
      if src.endswith('/') or dest.endswith('/'):
        raise ValueError('Cannot rename a directory')

    copy_results = self.copy_paths(src_dest_pairs)
    paths_to_delete = [src for (src, _, err) in copy_results if err is None]
    delete_results = self.delete_files(paths_to_delete)

    delete_results_dict = {src: err for (src, err) in delete_results}
    rename_results = []
    for src, dest, err in copy_results:
      if err is not None: rename_results.append((src, dest, err))
      elif delete_results_dict[src] is not None:
        rename_results.append((src, dest, delete_results_dict[src]))
      else:
        rename_results.append((src, dest, None))

    return rename_results


class S3Downloader(Downloader):

View on GitHub (pinned to 12126d8942)