apache/flink · warning · IOException

Bulk copy interrupted

Error message

Bulk copy interrupted

What it means

NativeS3BulkCopyHelper wraps InterruptedException from waiting on the batch of copy futures as an IOException('Bulk copy interrupted') after restoring the thread's interrupt flag. It means the copying thread was interrupted (e.g. task cancellation) rather than an S3 failure.

Source

Thrown at flink-filesystems/flink-s3-fs-native/src/main/java/org/apache/flink/fs/s3native/NativeS3BulkCopyHelper.java:370

            // Fail fast: complete as soon as either all downloads finish successfully or the first
            // one fails, rather than waiting for every in-flight download to run to completion. The
            // outer copyFiles handler aborts the remaining streams and shuts the pool down on the
            // resulting exception.
            CompletableFuture<Void> allDone =
                    CompletableFuture.allOf(futures.toArray(new CompletableFuture[0]));
            CompletableFuture<Void> firstFailure = new CompletableFuture<>();
            for (CompletableFuture<Void> future : futures) {
                future.whenComplete(
                        (ignored, error) -> {
                            if (error != null) {
                                firstFailure.completeExceptionally(error);
                            }
                        });
            }
            CompletableFuture.anyOf(allDone, firstFailure).get();
        } catch (InterruptedException e) {
            Thread.currentThread().interrupt();
            throw new IOException("Bulk copy interrupted", e);
        } catch (ExecutionException e) {
            Throwable cause = e.getCause();
            ExceptionUtils.rethrowIfFatalError(cause);
            if (isConnectionPoolExhausted(cause)) {
                throw new IOException(
                        String.format(
                                "S3 connection pool exhausted during bulk copy. "
                                        + "The configured connection pool size (%d) could not serve "
                                        + "the concurrent download requests (%d). "
                                        + "Consider reducing '%s' or increasing '%s'.",
                                maxConnections,
                                maxConcurrentCopies,
                                NativeS3FileSystemFactory.BULK_COPY_MAX_CONCURRENT.key(),
                                NativeS3FileSystemFactory.MAX_CONNECTIONS.key()),
                        cause);
            }
            throw new IOException("Bulk copy failed", cause);
        }

View on GitHub (pinned to 2f3c205e92)

Solutions

1. Treat this as a cancellation signal: stop submitting further batches and let the closeableRegistry/cancellation mechanism abort in-flight downloads.
2. If it appears without an intentional cancel, find who interrupts the thread (task cancellation, executor shutdownNow) and sequence shutdown after copies complete.
3. Do not retry blindly; re-run the bulk copy only after the interruption source is resolved and clean up partially written local files first.

try {
    s3Fs.copyFiles(requests, registry);
} catch (IOException e) {
    if (e.getCause() instanceof InterruptedException) {
        Thread.currentThread().interrupt();
        // cancellation path: stop cleanly, do not retry
        return;
    }
    throw e;
}

When it happens

Trigger: The thread executing copyFiles is interrupted while blocked in CompletableFuture.anyOf(...).get() — typically job/task cancellation, operator close during a bulk download, or an explicit thread interrupt from shutdown logic.

Common situations: Cancelling a Flink job or stopping a task while a bulk copy is in flight; test harnesses that interrupt worker threads; shutdown hooks racing with active downloads.

Related errors

• Interrupted while uploading object for key: {}
• interrupted while acquiring lock
• Only S3 to local copies are currently supported: {} -> {}
• S3 connection pool exhausted during bulk copy. The configure
• Bulk copy failed

AI-assisted analysis of apache/flink@2f3c205e92 (2026-08-14). Data as JSON: /api/errors/d15f4017d73872fb. Report an issue: GitHub.