apache/flink · error · IOException

Failed to rename {} to {}

Error message

Failed to rename {} to {}

What it means

Rename is implemented as copyObject followed by deleteObject on the same bucket. Any AWS SDK S3Exception from either call is wrapped in IOException('Failed to rename <src> to <dst>') with the cause preserved. Common causes are access denied on the source key, missing s3:PutObject at the destination, KMS/SSE permission issues on copy, or the source vanishing mid-rename.

Source

Thrown at flink-filesystems/flink-s3-fs-native/src/main/java/org/apache/flink/fs/s3native/NativeS3FileSystem.java:492

            throw new UnsupportedOperationException(
                    "NativeS3FileSystem does not support renaming directories: " + src);
        }

        try {
            final CopyObjectRequest copyRequest =
                    CopyObjectRequest.builder()
                            .sourceBucket(bucketName)
                            .sourceKey(srcKey)
                            .destinationBucket(bucketName)
                            .destinationKey(dstKey)
                            .build();
            s3Client.copyObject(copyRequest);
            final DeleteObjectRequest deleteRequest =
                    DeleteObjectRequest.builder().bucket(bucketName).key(srcKey).build();
            s3Client.deleteObject(deleteRequest);
            return true;
        } catch (S3Exception e) {
            throw new IOException("Failed to rename " + src + " to " + dst, e);
        }
    }

    @Override
    public boolean isDistributedFS() {
        return true;
    }

    @Nullable
    @Override
    public String getEntropyInjectionKey() {
        return entropyInjectionKey;
    }

    @Override
    public String generateEntropy() {
        return StringUtils.generateRandomAlphanumericString(
                ThreadLocalRandom.current(), entropyLength);

View on GitHub (pinned to 2f3c205e92)

Solutions

1. Inspect the wrapped S3Exception status code and grant the needed s3:GetObject, s3:PutObject, s3:DeleteObject (and kms:Decrypt/GenerateDataKey for SSE-KMS copy) permissions.
2. Handle the race: catch IOException, re-check exists(src), and treat a missing source as success or skip.
3. For transient 5xx causes, retry rename with backoff — copyObject+deleteObject is safe to retry if the copy is idempotent for your use case.

try {
    s3Fs.rename(src, dst);
} catch (IOException e) {
    if (e.getCause() instanceof S3Exception) {
        S3Exception s3e = (S3Exception) e.getCause();
        if (s3e.statusCode() >= 500) {
            // transient: retry copyObject+deleteObject with backoff
        } else if (!s3Fs.exists(src)) {
            // source vanished mid-rename: treat as moved/skip
        } else {
            // permissions/SSE issue surfaced by s3e: fix IAM/KMS before retrying
        }
    } else {
        throw e;
    }
}

When it happens

Trigger: rename(src, dst) on an existing object when copyObject or deleteObject fails — 403 (missing s3:GetObject/s3:PutObject/s3:DeleteObject), SSE-KMS key not usable, source deleted concurrently, or transient S3 errors.

Common situations: IAM roles without copy/delete permissions; cross-account buckets; KMS key policy not granting decrypt to the writer; racing cleanup jobs deleting the source between status check and copy.

Related errors

• Failed to delete: {}
• Unable to recover. Rename operation failed
• Committing file by rename failed: {} to {}
• NativeS3FileSystem does not support renaming directories: {}
• Failed to start multipart upload for key: {}

AI-assisted analysis of apache/flink@2f3c205e92 (2026-08-14). Data as JSON: /api/errors/607501be043434fa. Report an issue: GitHub.