{"record":{"id":"c6dfb84b97e58096","repo":"apache/flink","slug":"s3-connection-pool-exhausted-during-bulk-copy-the","errorCode":null,"errorMessage":"S3 connection pool exhausted during bulk copy. The configured connection pool size (%d) could not serve the concurrent download requests (%d). Consider reducing '%s' or increasing '%s'.","messagePattern":"S3 connection pool exhausted during bulk copy\\. The configured connection pool size \\((.+?)\\) could not serve the concurrent download requests \\((.+?)\\)\\. Consider reducing '(.+?)' or increasing '(.+?)'\\.","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"error","filePath":"flink-filesystems/flink-s3-fs-native/src/main/java/org/apache/flink/fs/s3native/NativeS3BulkCopyHelper.java","lineNumber":375,"sourceCode":"                    CompletableFuture.allOf(futures.toArray(new CompletableFuture[0]));\n            CompletableFuture<Void> firstFailure = new CompletableFuture<>();\n            for (CompletableFuture<Void> future : futures) {\n                future.whenComplete(\n                        (ignored, error) -> {\n                            if (error != null) {\n                                firstFailure.completeExceptionally(error);\n                            }\n                        });\n            }\n            CompletableFuture.anyOf(allDone, firstFailure).get();\n        } catch (InterruptedException e) {\n            Thread.currentThread().interrupt();\n            throw new IOException(\"Bulk copy interrupted\", e);\n        } catch (ExecutionException e) {\n            Throwable cause = e.getCause();\n            ExceptionUtils.rethrowIfFatalError(cause);\n            if (isConnectionPoolExhausted(cause)) {\n                throw new IOException(\n                        String.format(\n                                \"S3 connection pool exhausted during bulk copy. \"\n                                        + \"The configured connection pool size (%d) could not serve \"\n                                        + \"the concurrent download requests (%d). \"\n                                        + \"Consider reducing '%s' or increasing '%s'.\",\n                                maxConnections,\n                                maxConcurrentCopies,\n                                NativeS3FileSystemFactory.BULK_COPY_MAX_CONCURRENT.key(),\n                                NativeS3FileSystemFactory.MAX_CONNECTIONS.key()),\n                        cause);\n            }\n            throw new IOException(\"Bulk copy failed\", cause);\n        }\n    }\n\n    static boolean isSupportedS3Scheme(org.apache.flink.core.fs.Path path) {\n        String scheme = path.toUri().getScheme();\n        return \"s3\".equalsIgnoreCase(scheme) || \"s3a\".equalsIgnoreCase(scheme);","sourceCodeStart":357,"sourceCodeEnd":393,"githubUrl":"https://github.com/apache/flink/blob/2f3c205e9266cb30240eb7f4fdab15cad629a70f/flink-filesystems/flink-s3-fs-native/src/main/java/org/apache/flink/fs/s3native/NativeS3BulkCopyHelper.java#L357-L393","documentation":"During bulk copy, the async HTTP client's connection pool (size = MAX_CONNECTIONS) cannot serve the configured number of concurrent downloads (BULK_COPY_MAX_CONCURRENT), and requests fail with connection-pool-exhaustion errors. The exception names both config keys and values so the mismatch is directly actionable.","triggerScenarios":"Configuring s3.bulk-copy.max-concurrent greater than the AWS SDK client's max connections (s3.connection-pool.max-connections or equivalent), so concurrent download futures block/fail waiting for pooled connections.","commonSituations":"Tuning bulk-copy parallelism up for throughput without raising the HTTP client pool size; using defaults where pool size is smaller than copy concurrency; slow S3 responses holding connections longer, effectively shrinking the pool.","solutions":["Increase the connection pool size (NativeS3FileSystemFactory.MAX_CONNECTIONS, e.g. s3.connection-pool.max-connections) so it is >= the bulk-copy concurrency.","Alternatively reduce s3.bulk-copy.max-concurrent to at or below the current pool size.","Re-run the bulk copy; partially downloaded local files should be cleaned or overwritten first."],"exampleFix":"# before\ns3.bulk-copy.max-concurrent: 64\ns3.connection-pool.max-connections: 50 # default\n\n# after\ns3.bulk-copy.max-concurrent: 64\ns3.connection-pool.max-connections: 128","handlingStrategy":"validation","validationCode":"// before enabling bulk copy, assert pool can serve concurrency\nif (bulkCopyMaxConcurrent > maxConnections) {\n    throw new IllegalConfigurationException(\n        \"s3.bulk-copy.max-concurrent (\" + bulkCopyMaxConcurrent\n        + \") must be <= connection pool size (\" + maxConnections + \")\");\n}","typeGuard":null,"tryCatchPattern":"try {\n    s3Fs.copyFiles(requests, registry);\n} catch (IOException e) {\n    if (e.getMessage() != null && e.getMessage().contains(\"connection pool exhausted\")) {\n        // adjust config: raise max-connections or lower bulk-copy.max-concurrent, then retry\n    } else {\n        throw e;\n    }\n}","preventionTips":["Always review s3.bulk-copy.max-concurrent together with the SDK connection pool size.","Keep pool size comfortably above copy concurrency to absorb slow S3 responses.","Encode the pool >= concurrency invariant in config validation at deploy time."],"tags":["s3","bulk-copy","connection-pool","configuration","throughput"],"backgroundTag":null,"analyzedSha":"2f3c205e9266cb30240eb7f4fdab15cad629a70f","analyzedAt":"2026-08-14T08:48:24.518Z","schemaVersion":2},"datasetVersion":"2026-08-14T10:17:34.591Z"}