{"record":{"id":"8786df2ea755ea2a","repo":"apache/dolphinscheduler","slug":"s3-does-not-support-copying-directories","errorCode":null,"errorMessage":"S3 does not support copying directories.","messagePattern":"S3 does not support copying directories\\.","errorType":"exception","errorClass":"UnsupportedOperationException","httpStatus":null,"severity":"error","filePath":"dolphinscheduler-storage-plugin/dolphinscheduler-storage-s3/src/main/java/org/apache/dolphinscheduler/plugin/storage/s3/S3StorageOperator.java","lineNumber":165,"sourceCode":"            return;\n        }\n        if (recursive) {\n            List<StorageEntity> storageEntities = listStorageEntityRecursively(absolutePath);\n            for (StorageEntity storageEntity : storageEntities) {\n                s3Client.deleteObject(bucketName, transformAbsolutePathToS3Key(storageEntity.getFullName()));\n            }\n        }\n        s3Client.deleteObject(bucketName, absolutePath);\n    }\n\n    @Override\n    public void copy(String srcPath, String dstPath, boolean deleteSource, boolean overwrite) {\n        srcPath = transformAbsolutePathToS3Key(srcPath);\n        dstPath = transformAbsolutePathToS3Key(dstPath);\n\n        ResourceMetadata resourceMetaData = getResourceMetaData(srcPath);\n        if (resourceMetaData.isDirectory()) {\n            throw new UnsupportedOperationException(\"S3 does not support copying directories.\");\n        }\n        s3Client.copyObject(bucketName, srcPath, bucketName, dstPath);\n        if (deleteSource) {\n            s3Client.deleteObject(bucketName, srcPath);\n        }\n    }\n\n    @SneakyThrows\n    @Override\n    public void upload(String srcFile, String dstPath, boolean deleteSource, boolean overwrite) {\n        dstPath = transformAbsolutePathToS3Key(dstPath);\n\n        if (s3Client.doesObjectExist(bucketName, dstPath)) {\n            if (overwrite) {\n                s3Client.deleteObject(bucketName, dstPath);\n            } else {\n                throw new FileAlreadyExistsException(\"The file \" + dstPath + \" already exists in the bucket \"\n                        + bucketName + \" and overwrite is not allowed.\");","sourceCodeStart":147,"sourceCodeEnd":183,"githubUrl":"https://github.com/apache/dolphinscheduler/blob/02eac45a1b6676e639fcbfb4be2243de5771b05d/dolphinscheduler-storage-plugin/dolphinscheduler-storage-s3/src/main/java/org/apache/dolphinscheduler/plugin/storage/s3/S3StorageOperator.java#L147-L183","documentation":"S3StorageOperator.copy throws UnsupportedOperationException when the source path resolves to a directory. The S3 plugin only implements single-object CopyObject; recursive directory copy is not supported and is rejected up front.","triggerScenarios":"Calling copy(srcPath, dstPath, deleteSource, overwrite) where getResourceMetaData(srcPath).isDirectory() is true — the source key is a directory marker or has children.","commonSituations":"Trying to move/rename a whole resource folder via the storage API (e.g. renaming a user's resource directory) or porting code that worked against local/HDFS storage operators.","solutions":["Copy individual files: enumerate directory entries (listStorageEntity) and call copy per file, then delete the source files","Perform a server-side bulk operation outside the plugin (S3 Batch Operations) for large folders","Use a storage operator that supports directory copy (HDFS/local) if folder semantics are required"],"exampleFix":"// before\nstorageOperator.copy(\"/resources/folder\", \"/archive/folder\", true, false);\n// after\nfor (StorageEntity entity : storageOperator.listStorageEntity(\"/resources/folder\")) {\n    storageOperator.copy(entity.getPath(),\n        entity.getPath().replaceFirst(\"^/resources\", \"/archive\"), true, false);\n}","handlingStrategy":"try-catch","validationCode":"ResourceMetadata meta = s3StorageOperator.getResourceMetaData(srcPath);\nif (meta.isDirectory()) {\n    throw new IllegalArgumentException(\"Use per-file copy for directories: \" + srcPath);\n}\ns3StorageOperator.copy(srcPath, dstPath, deleteSource, overwrite);","typeGuard":null,"tryCatchPattern":"try {\n    s3StorageOperator.copy(src, dst, deleteSource, overwrite);\n} catch (UnsupportedOperationException e) {\n    log.warn(\"Directory copy unsupported by S3 plugin; falling back to per-file copy\");\n    copyDirectoryFileByFile(src, dst, deleteSource, overwrite);\n}","preventionTips":["Check getResourceMetaData(srcPath).isDirectory() before copying","Implement a helper that lists and copies files individually for folders","Remember S3 copy semantics differ from HDFS/local operators when abstracting over StorageOperator"],"tags":["storage","s3","unsupported-operation","directories"],"backgroundTag":"unsupported-operation","analyzedSha":"02eac45a1b6676e639fcbfb4be2243de5771b05d","analyzedAt":"2026-09-06T17:43:00.555Z","contentChangedAt":"2026-09-06T17:43:00.555Z","schemaVersion":2},"datasetVersion":"2026-09-14T00:17:10.932Z"}