{"record":{"id":"b0a3b5a18a9f7ba7","repo":"apache/flink","slug":"crt-transport-requested-s3-crt-enabled-true-but","errorCode":null,"errorMessage":"CRT transport requested (s3.crt.enabled=true) but the aws-crt JAR is not on the classpath. Place it in the Flink plugin directory (e.g. $FLINK_HOME/plugins/s3-fs-native/) alongside flink-s3-fs-native.jar. Run tools/download-crt-jars.sh to download the matching version. See the module README for setup details.","messagePattern":"CRT transport requested \\(s3\\.crt\\.enabled=true\\) but the aws-crt JAR is not on the classpath\\. Place it in the Flink plugin directory \\(e\\.g\\. \\$FLINK_HOME/plugins/s3-fs-native/\\) alongside flink-s3-fs-native\\.jar\\. Run tools/download-crt-jars\\.sh to download the matching version\\. See the module README for setup details\\.","errorType":"exception","errorClass":"IllegalStateException","httpStatus":null,"severity":"critical","filePath":"flink-filesystems/flink-s3-fs-native/src/main/java/org/apache/flink/fs/s3native/S3ClientProvider.java","lineNumber":774,"sourceCode":"                        crtHttpBuilder.readBufferSizeInBytes(crtReadBufferSizeInBytes);\n                    }\n                    clientBuilder.httpClientBuilder(crtHttpBuilder);\n                } else {\n                    clientBuilder.httpClientBuilder(\n                            ApacheHttpClient.builder()\n                                    .maxConnections(maxConnections)\n                                    .connectionTimeout(connectionTimeout)\n                                    .socketTimeout(socketTimeout)\n                                    .tcpKeepAlive(true)\n                                    .connectionMaxIdleTime(connectionMaxIdleTime));\n                }\n                if (endpointUri != null) {\n                    clientBuilder.endpointOverride(endpointUri);\n                }\n                return clientBuilder.build();\n            } catch (LinkageError e) {\n                if (useCrt) {\n                    throw new IllegalStateException(crtMissingJarsMessage(), e);\n                }\n                throw e;\n            } catch (IllegalStateException e) {\n                if (useCrt && isCrtClasspathFailure(e)) {\n                    throw new IllegalStateException(crtMissingJarsMessage(), e);\n                }\n                throw e;\n            }\n        }\n\n        /**\n         * Builds the asynchronous {@link S3AsyncClient}, choosing between the Netty transport and\n         * the AWS CRT transport based on {@link #useCrt}.\n         *\n         * <p>Note: when CRT is enabled the {@code s3.chunked-encoding.enabled} option is silently\n         * ignored. The CRT runtime manages wire encoding internally and {@link\n         * software.amazon.awssdk.services.s3.S3CrtAsyncClientBuilder} exposes no equivalent setter.\n         */","sourceCodeStart":756,"sourceCodeEnd":792,"githubUrl":"https://github.com/apache/flink/blob/2f3c205e9266cb30240eb7f4fdab15cad629a70f/flink-filesystems/flink-s3-fs-native/src/main/java/org/apache/flink/fs/s3native/S3ClientProvider.java#L756-L792","documentation":"Thrown while building the synchronous S3Client when s3.crt.enabled=true but the AWS CRT (Common Runtime) classes cannot be loaded. The builder catches LinkageError (NoClassDefFoundError/ClassNotFoundException shapes) raised by clientBuilder.build() when the CRT HTTP client factory is referenced, and rethrows it as an IllegalStateException with actionable guidance. The aws-crt-client and its native libraries are optional dependencies shipped as separate JARs, not bundled into flink-s3-fs-native.","triggerScenarios":"Setting fs.s3.crt.enabled (s3.crt.enabled) to true in flink-conf.yaml without placing aws-crt JARs in the plugin directory; putting the CRT JAR in lib/ instead of plugins/s3-fs-native/ so the isolated plugin classloader cannot see it; a CRT JAR version whose native .so fails to load, surfacing as LinkageError from the sync client builder path in S3ClientProvider.Builder (around buildSyncClient).","commonSituations":"Enabling CRT for throughput on large S3 writes after reading performance docs but skipping JAR deployment; upgrading the AWS SDK version in the Flink build so the previously downloaded CRT JAR no longer matches; running on an OS/arch (e.g. Alpine musl) where the CRT native library is unavailable.","solutions":["Run tools/download-crt-jars.sh (repo: flink-filesystems/flink-s3-fs-native) to fetch the CRT JARs matching this build's AWS SDK version, and copy them into $FLINK_HOME/plugins/s3-fs-native/ next to flink-s3-fs-native.jar.","Verify the JARs are in the plugin directory (not lib/): ls $FLINK_HOME/plugins/s3-fs-native/ should show aws-crt-client-*.jar and, if needed for your SDK version, http-client-* CRT artifacts.","Ensure the CRT version matches the aws-sdk version compiled into flink-s3-fs-native (check the SDK bom in the module pom); mismatched versions cause LinkageError even when the JAR is present.","If you cannot deploy the JARs, remove s3.crt.enabled=true to fall back to the Netty/NIO async transport.","Confirm the runtime OS/arch has CRT native support (glibc Linux x86_64/arm64); on unsupported platforms use Netty instead."],"exampleFix":"# before (flink-conf.yaml)\ns3.crt.enabled: true\n# plugins dir contains only flink-s3-fs-native.jar\n\n# after\nbash tools/download-crt-jars.sh\ncp download/aws-crt-client-*.jar $FLINK_HOME/plugins/s3-fs-native/\n# flink-conf.yaml unchanged; restart cluster","handlingStrategy":"validation","validationCode":"// Pre-flight: verify CRT classes load BEFORE building the provider with s3.crt.enabled=true\nstatic boolean crtAvailable() {\n    try {\n        Class.forName(\"software.amazon.awssdk.http.crt.AwsCrtHttpClient\");\n        return true;\n    } catch (ClassNotFoundException | LinkageError e) {\n        return false;\n    }\n}\n// boolean useCrt = config.getBoolean(\"s3.crt.enabled\") && crtAvailable();","typeGuard":null,"tryCatchPattern":"catch (IllegalStateException e) { if (e.getMessage()!=null && e.getMessage().contains(\"aws-crt JAR is not on the classpath\")) { deployCrtJars(); /* or disable crt and rebuild with Netty */ } else throw e; }","preventionTips":["Bake tools/download-crt-jars.sh output into your docker image / AMI so every node ships the matched CRT JARs in plugins/s3-fs-native/.","Add a startup smoke check that lists $FLINK_HOME/plugins/s3-fs-native/ and fails fast when s3.crt.enabled=true but aws-crt-*.jar is absent.","Keep CRT JAR version pinned to the SDK version of the flink-s3-fs-native build; re-run the download script after every Flink upgrade."],"tags":["s3","crt","classpath","plugins","deployment","flink"],"backgroundTag":null,"analyzedSha":"2f3c205e9266cb30240eb7f4fdab15cad629a70f","analyzedAt":"2026-08-14T08:48:24.518Z","schemaVersion":2},"datasetVersion":"2026-08-14T10:17:34.591Z"}