{"record":{"id":"d7021996d14d366e","repo":"apache/flink","slug":"only-s3-to-local-copies-are-currently-supported","errorCode":null,"errorMessage":"Only S3 to local copies are currently supported: {} -> {}","messagePattern":"Only S3 to local copies are currently supported: (.+?) -> (.+?)","errorType":"exception","errorClass":"UnsupportedOperationException","httpStatus":null,"severity":"error","filePath":"flink-filesystems/flink-s3-fs-native/src/main/java/org/apache/flink/fs/s3native/NativeS3BulkCopyHelper.java","lineNumber":192,"sourceCode":"                                        LOG.error(\n                                                \"Uncaught exception in S3 bulk-copy worker {}\",\n                                                thread.getName(),\n                                                error)));\n        BulkCopyCancellation cancellation = new BulkCopyCancellation(downloadPool);\n        ICloseableRegistry registry =\n                closeableRegistry == null ? ICloseableRegistry.NO_OP : closeableRegistry;\n        List<CompletableFuture<Void>> copyFutures = new ArrayList<>();\n        int batchNumber = 0;\n\n        try (Closeable ignored = registry.registerCloseableTemporarily(cancellation)) {\n            for (int i = 0; i < requests.size(); i++) {\n                PathsCopyingFileSystem.CopyRequest request = requests.get(i);\n                String sourceUri = request.getSource().toUri().toString();\n                if (isSupportedS3Scheme(request.getSource())\n                        && isSupportedLocalScheme(request.getDestination())) {\n                    copyFutures.add(copyS3ToLocal(request, downloadPool, cancellation));\n                } else {\n                    throw new UnsupportedOperationException(\n                            \"Only S3 to local copies are currently supported: \"\n                                    + sourceUri\n                                    + \" -> \"\n                                    + request.getDestination());\n                }\n\n                if (copyFutures.size() >= maxConcurrentCopies || i == requests.size() - 1) {\n                    batchNumber++;\n                    LOG.debug(\n                            \"Waiting for batch {}/{} ({} files)\",\n                            batchNumber,\n                            totalBatches,\n                            copyFutures.size());\n                    waitForCopies(copyFutures);\n                    copyFutures.clear();\n                }\n            }\n","sourceCodeStart":174,"sourceCodeEnd":210,"githubUrl":"https://github.com/apache/flink/blob/2f3c205e9266cb30240eb7f4fdab15cad629a70f/flink-filesystems/flink-s3-fs-native/src/main/java/org/apache/flink/fs/s3native/NativeS3BulkCopyHelper.java#L174-L210","documentation":"NativeS3BulkCopyHelper.copyFiles only supports copying FROM an s3:// or s3a:// source TO a local (file: or schemeless) destination. Any request whose source is not S3 or whose destination is not local throws UnsupportedOperationException with both URIs in the message.","triggerScenarios":"Calling PathsCopyingFileSystem.copyFiles (via NativeS3FileSystem.copyFiles) with requests like local->S3, S3->S3, or S3->HDFS. isSupportedS3Scheme accepts only s3/s3a schemes and isSupportedLocalScheme accepts only null or file schemes.","commonSituations":"Using bulk copy as a generic distributed-copy utility (e.g. staging files back to S3 or between buckets), or a pipeline whose recovery/download direction was inverted in configuration.","solutions":["Restructure the copy so sources are s3:// or s3a:// URIs and destinations are local file:// paths.","For S3->S3 or local->S3 copies, fall back to the regular FileSystem copy APIs or the S3 TransferManager outside this helper.","Filter requests before calling copyFiles and route unsupported direction pairs to a different mechanism."],"exampleFix":"// before\ncopyRequests.add(new CopyRequest(localPath, s3Path)); // throws\n\n// after\n// only S3 -> local is supported by bulk copy\ncopyRequests.add(new CopyRequest(s3Path, localPath));\n// handle local -> S3 with the standard FileSystem API instead:\ntry (FSDataInputStream in = localFs.open(localPath);\n     FSDataOutputStream out = s3Fs.create(s3Path)) {\n    in.transferTo(out);\n}","handlingStrategy":"validation","validationCode":"static boolean isBulkCopySupported(Path src, Path dst) {\n    String s = src.toUri().getScheme();\n    String d = dst.toUri().getScheme();\n    return (\"s3\".equalsIgnoreCase(s) || \"s3a\".equalsIgnoreCase(s))\n            && (d == null || \"file\".equalsIgnoreCase(d));\n}\n\n// use before building the request list:\nList<CopyRequest> supported = requests.stream()\n        .filter(r -> isBulkCopySupported(r.getSource(), r.getDestination()))\n        .collect(Collectors.toList());","typeGuard":null,"tryCatchPattern":"try {\n    s3Fs.copyFiles(requests, registry);\n} catch (UnsupportedOperationException e) {\n    // direction not supported by bulk copy; fall back to per-file copy via standard API\n}","preventionTips":["Remember bulk copy only moves S3 -> local; never assume it is a generic copy utility.","Validate request direction pairs before calling copyFiles and route others elsewhere.","Log rejected URIs explicitly to catch inverted source/destination wiring early."],"tags":["s3","bulk-copy","unsupported-operation","api-misuse"],"backgroundTag":null,"analyzedSha":"2f3c205e9266cb30240eb7f4fdab15cad629a70f","analyzedAt":"2026-08-14T08:48:24.518Z","schemaVersion":2},"datasetVersion":"2026-08-14T10:17:34.591Z"}