apache/flink · error · UnsupportedFileSystemSchemeException

Hadoop File System abstraction does not support scheme '${sc

Error message

Hadoop File System abstraction does not support scheme '${scheme}'. Either no file system implementation exists for that scheme, or the relevant classes are missing from the classpath.

What it means

HadoopFsFactory cannot resolve a FileSystem class for the requested scheme: org.apache.hadoop.fs.FileSystem.getFileSystemClass(scheme, config) threw, so Flink raises UnsupportedFileSystemSchemeException. This means either no Hadoop filesystem implementation is registered for that scheme or its classes are not on the classpath.

Source

Thrown at flink-filesystems/flink-hadoop-fs/src/main/java/org/apache/flink/runtime/fs/hdfs/HadoopFsFactory.java:100

                hadoopConfig = this.hadoopConfig;
            } else if (flinkConfig != null) {
                hadoopConfig = HadoopUtils.getHadoopConfiguration(flinkConfig);
                this.hadoopConfig = hadoopConfig;
            } else {
                LOG.warn(
                        "Hadoop configuration has not been explicitly initialized prior to loading a Hadoop file system."
                                + " Using configuration from the classpath.");

                hadoopConfig = new org.apache.hadoop.conf.Configuration();
            }

            // -- (2) get the Hadoop file system class for that scheme

            final Class<? extends org.apache.hadoop.fs.FileSystem> fsClass;
            try {
                fsClass = org.apache.hadoop.fs.FileSystem.getFileSystemClass(scheme, hadoopConfig);
            } catch (IOException e) {
                throw new UnsupportedFileSystemSchemeException(
                        "Hadoop File System abstraction does not support scheme '"
                                + scheme
                                + "'. "
                                + "Either no file system implementation exists for that scheme, "
                                + "or the relevant classes are missing from the classpath.",
                        e);
            }

            // -- (3) instantiate the Hadoop file system

            LOG.debug(
                    "Instantiating for file system scheme {} Hadoop File System {}",
                    scheme,
                    fsClass.getName());

            final org.apache.hadoop.fs.FileSystem hadoopFs = fsClass.newInstance();

            // -- (4) create the proper URI to initialize the file system

View on GitHub (pinned to 2f3c205e92)

Solutions

1. Add the Hadoop filesystem connector jar for that scheme to Flink's lib/ directory on ALL nodes (e.g. hadoop-aws for s3a, hadoop-azure for abfs) and restart
2. Prefer Flink's native filesystem plugins (flink-s3-fs-hadoop, flink-oss-fs-hadoop, etc.) placed in lib/ or opt/ instead of raw Hadoop jars
3. Set fs.<scheme>.impl (and fs.<scheme>.impl.kerberos...) in flink-conf.yaml / core-site.xml if the implementation class exists but is not auto-registered
4. Verify the jar is not shaded with unrelocated META-INF/services entries that break the Hadoop FileSystem service loader

Example fix

# before: scheme 's3a' unsupported
# after: copy plugin and restart
 cp opt/flink-s3-fs-hadoop* lib/ && ./bin/start-cluster.sh

// pre-flight: can Hadoop resolve the scheme?
try {
    org.apache.hadoop.fs.FileSystem.getFileSystemClass(
            uri.getScheme(), new org.apache.hadoop.conf.Configuration());
} catch (IOException e) {
    throw new IllegalStateException("Add the connector jar for scheme " + uri.getScheme(), e);
}

try {
    fs = FileSystem.get(uri);
} catch (UnsupportedFileSystemSchemeException e) {
    // missing connector jar: fail fast with setup instructions, not a retry
}

When it happens

Trigger: Calling FileSystem.get(uri) for a scheme (e.g. 'hdfs', 's3a', 'abfs') whose Hadoop connector jar is absent, or whose service-provider entry in core-site.xml (fs.<scheme>.impl) is missing, so getFileSystemClass fails.

Common situations: Using the bare flink-shaded-hadoop jar which bundles only HDFS; putting an S3/OBS/Azure filesystem path in state.checkpoints.dir or output paths without adding the corresponding Hadoop connector jar to Flink's lib/ directory; shaded-class relocation issues where the service loader file was not relocated.

Related errors

• Bad syntax for classpath: {}
• Multiple compatible client factories found for: {}.
• No ClusterClientFactory found. If you were targeting a Yarn
• Could not load class for serialization config
• Could not instantiate class '%s' of type '%s'. Please make s

AI-assisted analysis of apache/flink@2f3c205e92 (2026-08-14). Data as JSON: /api/errors/7fc78a8a6312eb89. Report an issue: GitHub.