apache/flink · error · UnsupportedOperationException

Only S3 to local copies are currently supported: {} -> {}

Error message

Only S3 to local copies are currently supported: {} -> {}

What it means

NativeS3BulkCopyHelper.copyFiles only supports copying FROM an s3:// or s3a:// source TO a local (file: or schemeless) destination. Any request whose source is not S3 or whose destination is not local throws UnsupportedOperationException with both URIs in the message.

Source

Thrown at flink-filesystems/flink-s3-fs-native/src/main/java/org/apache/flink/fs/s3native/NativeS3BulkCopyHelper.java:192

                                        LOG.error(
                                                "Uncaught exception in S3 bulk-copy worker {}",
                                                thread.getName(),
                                                error)));
        BulkCopyCancellation cancellation = new BulkCopyCancellation(downloadPool);
        ICloseableRegistry registry =
                closeableRegistry == null ? ICloseableRegistry.NO_OP : closeableRegistry;
        List<CompletableFuture<Void>> copyFutures = new ArrayList<>();
        int batchNumber = 0;

        try (Closeable ignored = registry.registerCloseableTemporarily(cancellation)) {
            for (int i = 0; i < requests.size(); i++) {
                PathsCopyingFileSystem.CopyRequest request = requests.get(i);
                String sourceUri = request.getSource().toUri().toString();
                if (isSupportedS3Scheme(request.getSource())
                        && isSupportedLocalScheme(request.getDestination())) {
                    copyFutures.add(copyS3ToLocal(request, downloadPool, cancellation));
                } else {
                    throw new UnsupportedOperationException(
                            "Only S3 to local copies are currently supported: "
                                    + sourceUri
                                    + " -> "
                                    + request.getDestination());
                }

                if (copyFutures.size() >= maxConcurrentCopies || i == requests.size() - 1) {
                    batchNumber++;
                    LOG.debug(
                            "Waiting for batch {}/{} ({} files)",
                            batchNumber,
                            totalBatches,
                            copyFutures.size());
                    waitForCopies(copyFutures);
                    copyFutures.clear();
                }
            }

View on GitHub (pinned to 2f3c205e92)

Solutions

1. Restructure the copy so sources are s3:// or s3a:// URIs and destinations are local file:// paths.
2. For S3->S3 or local->S3 copies, fall back to the regular FileSystem copy APIs or the S3 TransferManager outside this helper.
3. Filter requests before calling copyFiles and route unsupported direction pairs to a different mechanism.

Example fix

// before
copyRequests.add(new CopyRequest(localPath, s3Path)); // throws

// after
// only S3 -> local is supported by bulk copy
copyRequests.add(new CopyRequest(s3Path, localPath));
// handle local -> S3 with the standard FileSystem API instead:
try (FSDataInputStream in = localFs.open(localPath);
     FSDataOutputStream out = s3Fs.create(s3Path)) {
    in.transferTo(out);
}

static boolean isBulkCopySupported(Path src, Path dst) {
    String s = src.toUri().getScheme();
    String d = dst.toUri().getScheme();
    return ("s3".equalsIgnoreCase(s) || "s3a".equalsIgnoreCase(s))
            && (d == null || "file".equalsIgnoreCase(d));
}

// use before building the request list:
List<CopyRequest> supported = requests.stream()
        .filter(r -> isBulkCopySupported(r.getSource(), r.getDestination()))
        .collect(Collectors.toList());

try {
    s3Fs.copyFiles(requests, registry);
} catch (UnsupportedOperationException e) {
    // direction not supported by bulk copy; fall back to per-file copy via standard API
}

When it happens

Trigger: Calling PathsCopyingFileSystem.copyFiles (via NativeS3FileSystem.copyFiles) with requests like local->S3, S3->S3, or S3->HDFS. isSupportedS3Scheme accepts only s3/s3a schemes and isSupportedLocalScheme accepts only null or file schemes.

Common situations: Using bulk copy as a generic distributed-copy utility (e.g. staging files back to S3 or between buckets), or a pipeline whose recovery/download direction was inverted in configuration.

Related errors

• PostVersionedIOReadableWritable cannot read from a DataInput
• SimpleVersionedSerializerWrapper is not meant to be used as
• S3 File System cannot recover recoverable for other file sys
• Bulk copy interrupted
• S3 connection pool exhausted during bulk copy. The configure

AI-assisted analysis of apache/flink@2f3c205e92 (2026-08-14). Data as JSON: /api/errors/d7021996d14d366e. Report an issue: GitHub.