apache/flink · critical · IllegalStateException

CRT transport requested (s3.crt.enabled=true) but the aws-cr

Error message

CRT transport requested (s3.crt.enabled=true) but the aws-crt JAR is not on the classpath. Place it in the Flink plugin directory (e.g. $FLINK_HOME/plugins/s3-fs-native/) alongside flink-s3-fs-native.jar. Run tools/download-crt-jars.sh to download the matching version. See the module README for setup details.

What it means

Thrown while building the synchronous S3Client when s3.crt.enabled=true but the AWS CRT (Common Runtime) classes cannot be loaded. The builder catches LinkageError (NoClassDefFoundError/ClassNotFoundException shapes) raised by clientBuilder.build() when the CRT HTTP client factory is referenced, and rethrows it as an IllegalStateException with actionable guidance. The aws-crt-client and its native libraries are optional dependencies shipped as separate JARs, not bundled into flink-s3-fs-native.

Source

Thrown at flink-filesystems/flink-s3-fs-native/src/main/java/org/apache/flink/fs/s3native/S3ClientProvider.java:774

                        crtHttpBuilder.readBufferSizeInBytes(crtReadBufferSizeInBytes);
                    }
                    clientBuilder.httpClientBuilder(crtHttpBuilder);
                } else {
                    clientBuilder.httpClientBuilder(
                            ApacheHttpClient.builder()
                                    .maxConnections(maxConnections)
                                    .connectionTimeout(connectionTimeout)
                                    .socketTimeout(socketTimeout)
                                    .tcpKeepAlive(true)
                                    .connectionMaxIdleTime(connectionMaxIdleTime));
                }
                if (endpointUri != null) {
                    clientBuilder.endpointOverride(endpointUri);
                }
                return clientBuilder.build();
            } catch (LinkageError e) {
                if (useCrt) {
                    throw new IllegalStateException(crtMissingJarsMessage(), e);
                }
                throw e;
            } catch (IllegalStateException e) {
                if (useCrt && isCrtClasspathFailure(e)) {
                    throw new IllegalStateException(crtMissingJarsMessage(), e);
                }
                throw e;
            }
        }

        /**
         * Builds the asynchronous {@link S3AsyncClient}, choosing between the Netty transport and
         * the AWS CRT transport based on {@link #useCrt}.
         *
         * <p>Note: when CRT is enabled the {@code s3.chunked-encoding.enabled} option is silently
         * ignored. The CRT runtime manages wire encoding internally and {@link
         * software.amazon.awssdk.services.s3.S3CrtAsyncClientBuilder} exposes no equivalent setter.
         */

View on GitHub (pinned to 2f3c205e92)

Solutions

1. Run tools/download-crt-jars.sh (repo: flink-filesystems/flink-s3-fs-native) to fetch the CRT JARs matching this build's AWS SDK version, and copy them into $FLINK_HOME/plugins/s3-fs-native/ next to flink-s3-fs-native.jar.
2. Verify the JARs are in the plugin directory (not lib/): ls $FLINK_HOME/plugins/s3-fs-native/ should show aws-crt-client-*.jar and, if needed for your SDK version, http-client-* CRT artifacts.
3. Ensure the CRT version matches the aws-sdk version compiled into flink-s3-fs-native (check the SDK bom in the module pom); mismatched versions cause LinkageError even when the JAR is present.
4. If you cannot deploy the JARs, remove s3.crt.enabled=true to fall back to the Netty/NIO async transport.
5. Confirm the runtime OS/arch has CRT native support (glibc Linux x86_64/arm64); on unsupported platforms use Netty instead.

Example fix

# before (flink-conf.yaml)
s3.crt.enabled: true
# plugins dir contains only flink-s3-fs-native.jar

# after
bash tools/download-crt-jars.sh
cp download/aws-crt-client-*.jar $FLINK_HOME/plugins/s3-fs-native/
# flink-conf.yaml unchanged; restart cluster

// Pre-flight: verify CRT classes load BEFORE building the provider with s3.crt.enabled=true
static boolean crtAvailable() {
    try {
        Class.forName("software.amazon.awssdk.http.crt.AwsCrtHttpClient");
        return true;
    } catch (ClassNotFoundException | LinkageError e) {
        return false;
    }
}
// boolean useCrt = config.getBoolean("s3.crt.enabled") && crtAvailable();

catch (IllegalStateException e) { if (e.getMessage()!=null && e.getMessage().contains("aws-crt JAR is not on the classpath")) { deployCrtJars(); /* or disable crt and rebuild with Netty */ } else throw e; }

When it happens

Trigger: Setting fs.s3.crt.enabled (s3.crt.enabled) to true in flink-conf.yaml without placing aws-crt JARs in the plugin directory; putting the CRT JAR in lib/ instead of plugins/s3-fs-native/ so the isolated plugin classloader cannot see it; a CRT JAR version whose native .so fails to load, surfacing as LinkageError from the sync client builder path in S3ClientProvider.Builder (around buildSyncClient).

Common situations: Enabling CRT for throughput on large S3 writes after reading performance docs but skipping JAR deployment; upgrading the AWS SDK version in the Flink build so the previously downloaded CRT JAR no longer matches; running on an OS/arch (e.g. Alpine musl) where the CRT native library is unavailable.

Related errors

• Multiple compatible client factories found for: {}.
• No ClusterClientFactory found. If you were targeting a Yarn
• Input opening request timed out. Opener was {} alive. Stack
• Output path could not be initialized.
• Class '%s' not found. Perhaps you forgot to add the module '

AI-assisted analysis of apache/flink@2f3c205e92 (2026-08-14). Data as JSON: /api/errors/b0a3b5a18a9f7ba7. Report an issue: GitHub.