apache/flink · error · IOException

S3 connection pool exhausted during bulk copy. The configure

Error message

S3 connection pool exhausted during bulk copy. The configured connection pool size (%d) could not serve the concurrent download requests (%d). Consider reducing '%s' or increasing '%s'.

What it means

During bulk copy, the async HTTP client's connection pool (size = MAX_CONNECTIONS) cannot serve the configured number of concurrent downloads (BULK_COPY_MAX_CONCURRENT), and requests fail with connection-pool-exhaustion errors. The exception names both config keys and values so the mismatch is directly actionable.

Source

Thrown at flink-filesystems/flink-s3-fs-native/src/main/java/org/apache/flink/fs/s3native/NativeS3BulkCopyHelper.java:375

                    CompletableFuture.allOf(futures.toArray(new CompletableFuture[0]));
            CompletableFuture<Void> firstFailure = new CompletableFuture<>();
            for (CompletableFuture<Void> future : futures) {
                future.whenComplete(
                        (ignored, error) -> {
                            if (error != null) {
                                firstFailure.completeExceptionally(error);
                            }
                        });
            }
            CompletableFuture.anyOf(allDone, firstFailure).get();
        } catch (InterruptedException e) {
            Thread.currentThread().interrupt();
            throw new IOException("Bulk copy interrupted", e);
        } catch (ExecutionException e) {
            Throwable cause = e.getCause();
            ExceptionUtils.rethrowIfFatalError(cause);
            if (isConnectionPoolExhausted(cause)) {
                throw new IOException(
                        String.format(
                                "S3 connection pool exhausted during bulk copy. "
                                        + "The configured connection pool size (%d) could not serve "
                                        + "the concurrent download requests (%d). "
                                        + "Consider reducing '%s' or increasing '%s'.",
                                maxConnections,
                                maxConcurrentCopies,
                                NativeS3FileSystemFactory.BULK_COPY_MAX_CONCURRENT.key(),
                                NativeS3FileSystemFactory.MAX_CONNECTIONS.key()),
                        cause);
            }
            throw new IOException("Bulk copy failed", cause);
        }
    }

    static boolean isSupportedS3Scheme(org.apache.flink.core.fs.Path path) {
        String scheme = path.toUri().getScheme();
        return "s3".equalsIgnoreCase(scheme) || "s3a".equalsIgnoreCase(scheme);

View on GitHub (pinned to 2f3c205e92)

Solutions

Increase the connection pool size (NativeS3FileSystemFactory.MAX_CONNECTIONS, e.g. s3.connection-pool.max-connections) so it is >= the bulk-copy concurrency.
Alternatively reduce s3.bulk-copy.max-concurrent to at or below the current pool size.
Re-run the bulk copy; partially downloaded local files should be cleaned or overwritten first.

Example fix

# before
s3.bulk-copy.max-concurrent: 64
s3.connection-pool.max-connections: 50 # default

# after
s3.bulk-copy.max-concurrent: 64
s3.connection-pool.max-connections: 128

Defensive patterns

Strategy: validation

Validate before calling

// before enabling bulk copy, assert pool can serve concurrency
if (bulkCopyMaxConcurrent > maxConnections) {
    throw new IllegalConfigurationException(
        "s3.bulk-copy.max-concurrent (" + bulkCopyMaxConcurrent
        + ") must be <= connection pool size (" + maxConnections + ")");
}

Try / catch

try {
    s3Fs.copyFiles(requests, registry);
} catch (IOException e) {
    if (e.getMessage() != null && e.getMessage().contains("connection pool exhausted")) {
        // adjust config: raise max-connections or lower bulk-copy.max-concurrent, then retry
    } else {
        throw e;
    }
}

Prevention

Always review s3.bulk-copy.max-concurrent together with the SDK connection pool size.
Keep pool size comfortably above copy concurrency to absorb slow S3 responses.
Encode the pool >= concurrency invariant in config validation at deploy time.

When it happens

Trigger: Configuring s3.bulk-copy.max-concurrent greater than the AWS SDK client's max connections (s3.connection-pool.max-connections or equivalent), so concurrent download futures block/fail waiting for pooled connections.

Common situations: Tuning bulk-copy parallelism up for throughput without raising the HTTP client pool size; using defaults where pool size is smaller than copy concurrency; slow S3 responses holding connections longer, effectively shrinking the pool.

Related errors

AI-assisted analysis of apache/flink@2f3c205e92 (2026-08-14). Data as JSON: /api/errors/c6dfb84b97e58096. Report an issue: GitHub.