apache/flink · error · UnsupportedOperationException

NativeS3FileSystem does not support renaming directories: {}

Error message

NativeS3FileSystem does not support renaming directories: {}

What it means

NativeS3FileSystem.rename only supports single-object renames (copy + delete). If getFileStatus(src).isDir(), it throws UnsupportedOperationException, because renaming a directory prefix in S3 requires copying every object under the prefix, which this implementation deliberately does not do (cost/consistency reasons).

Source

Thrown at flink-filesystems/flink-s3-fs-native/src/main/java/org/apache/flink/fs/s3native/NativeS3FileSystem.java:474

    }

    /**
     * Renames a single file from {@code src} to {@code dst}.
     *
     * <p><b>Directory rename is not supported.</b>
     *
     * @throws UnsupportedOperationException if {@code src} is a directory
     */
    @Override
    public boolean rename(Path src, Path dst) throws IOException {
        checkNotClosed();
        final String srcKey = NativeS3ObjectOperations.extractKey(src);
        final String dstKey = NativeS3ObjectOperations.extractKey(dst);
        final S3Client s3Client = clientProvider.getS3Client();

        final FileStatus srcStatus = getFileStatus(src);
        if (srcStatus.isDir()) {
            throw new UnsupportedOperationException(
                    "NativeS3FileSystem does not support renaming directories: " + src);
        }

        try {
            final CopyObjectRequest copyRequest =
                    CopyObjectRequest.builder()
                            .sourceBucket(bucketName)
                            .sourceKey(srcKey)
                            .destinationBucket(bucketName)
                            .destinationKey(dstKey)
                            .build();
            s3Client.copyObject(copyRequest);
            final DeleteObjectRequest deleteRequest =
                    DeleteObjectRequest.builder().bucket(bucketName).key(srcKey).build();
            s3Client.deleteObject(deleteRequest);
            return true;
        } catch (S3Exception e) {
            throw new IOException("Failed to rename " + src + " to " + dst, e);

View on GitHub (pinned to 2f3c205e92)

Solutions

1. Rename objects individually: list all keys under the prefix, copyObject each to the new prefix, then delete the sources (wrap in a loop with your own error handling).
2. Write directly to the final destination keys instead of moving directories afterwards.
3. Use a committer designed for S3 (e.g. S3 committers via magic/staged commits) instead of rename-based commit protocols.

Example fix

// before
boolean ok = s3Fs.rename(srcDir, dstDir); // UnsupportedOperationException

// after
// rename objects one by one under the prefix
for (FileStatus st : s3Fs.listStatus(srcDir)) {
    Path dst = new Path(dstDir, st.getPath().getName());
    if (!s3Fs.rename(st.getPath(), dst)) {
        throw new IOException("Failed to move " + st.getPath());
    }
}

// guard: only rename files, never directories
FileStatus st = s3Fs.getFileStatus(src);
if (st.isDir()) {
    throw new UnsupportedOperationException("Directory rename not supported; move children individually");
}

static boolean isFileRenameSupported(FileSystem fs, Path src) throws IOException {
    return !fs.getFileStatus(src).isDir(); // NativeS3FileSystem renames only objects
}

try {
    s3Fs.rename(src, dst);
} catch (UnsupportedOperationException e) {
    // directory rename: fall back to per-object copy+delete under the prefix
    for (FileStatus child : s3Fs.listStatus(src)) {
        s3Fs.rename(child.getPath(), new Path(dst, child.getPath().getName()));
    }
}

When it happens

Trigger: Calling rename() where src resolves to a directory (objects exist under src/ prefix or a directory marker exists) — e.g. commit-time directory moves, moving date-partitioned output trees, or filesystem tests that assume POSIX rename.

Common situations: Porting HDFS-style jobs that atomically move output directories on commit; using S3 with frameworks that rely on directory rename for transactions (the classic S3A consistency limitation); moving partition folders after writing.

Related errors

• Directory not empty and recursive = false
• Only S3 to local copies are currently supported: {} -> {}
• File not found: {}
• Failed to rename {} to {}
• Recoverable writer not available

AI-assisted analysis of apache/flink@2f3c205e92 (2026-08-14). Data as JSON: /api/errors/fd3e8d0d4a0bbd85. Report an issue: GitHub.