apache/seatunnel · critical · CheckpointStorageException

Failed to get file system

Error message

Failed to get file system

What it means

HdfsStorage.initStorage obtains a Hadoop FileSystem handle from the configured storage type/config. If FileSystem.get(hadoopConf) throws an IOException — bad HDFS URI, missing Hadoop configuration, unreachable NameNode, or missing filesystem implementation on the classpath (wrapped as IOException in some paths) — the plugin fails fast with this CheckpointStorageException.

Source

Thrown at seatunnel-engine/seatunnel-engine-storage/checkpoint-storage-plugins/checkpoint-storage-hdfs/src/main/java/org/apache/seatunnel/engine/checkpoint/storage/hdfs/HdfsStorage.java:71

    public FileSystem fs;
    private static final String STORAGE_TMP_SUFFIX = "tmp";
    private static final String STORAGE_TYPE_KEY = "storage.type";

    public HdfsStorage(Map<String, String> configuration) throws CheckpointStorageException {
        this.initStorage(configuration);
    }

    @Override
    public void initStorage(Map<String, String> configuration) throws CheckpointStorageException {
        if (StringUtils.isNotBlank(configuration.get(STORAGE_NAME_SPACE))) {
            setStorageNameSpace(configuration.get(STORAGE_NAME_SPACE));
            configuration.remove(STORAGE_NAME_SPACE);
        }
        Configuration hadoopConf = getConfiguration(configuration);
        try {
            fs = FileSystem.get(hadoopConf);
        } catch (IOException e) {
            throw new CheckpointStorageException("Failed to get file system", e);
        }
    }

    private Configuration getConfiguration(Map<String, String> config)
            throws CheckpointStorageException {
        String storageType =
                config.getOrDefault(STORAGE_TYPE_KEY, FileConfiguration.LOCAL.toString());
        config.remove(STORAGE_TYPE_KEY);
        AbstractConfiguration configuration =
                FileConfiguration.valueOf(storageType.toUpperCase()).getConfiguration();
        return configuration.buildConfiguration(config);
    }

    @Override
    public String storeCheckPoint(PipelineState state) throws CheckpointStorageException {
        byte[] datas;
        try {
            datas = serializeCheckPointData(state);

View on GitHub (pinned to cf67b549a7)

Solutions

1. Verify hdfs dfs -ls / works from the same node/user with the same configs
2. Check the storage config: fs.defaultFS / storage.type values and that Hadoop conf XMLs are on the classpath (HADOOP_CONF_DIR)
3. Confirm the NameNode is reachable (host, port, DNS, firewall)
4. Ensure required Hadoop/HDFS dependency jars are in the SeaTunnel connector lib directory
5. If HA nameservice is used, include all dfs.nameservices settings

Example fix

// before
HdfsStorage hdfs = new HdfsStorage(Map.of("storage.type", "hdfs")); // no fs config
// after
HdfsStorage hdfs = new HdfsStorage(Map.of(
    "storage.type", "hdfs",
    "fs.defaultFS", "hdfs://namenode:8020"));

// before constructing HdfsStorage, verify the filesystem is reachable
Configuration conf = new Configuration();
conf.set("fs.defaultFS", config.get("fs.defaultFS"));
FileSystem fs = FileSystem.get(conf);
if (!fs.exists(new Path(baseDir))) fs.mkdirs(new Path(baseDir));

try {
  HdfsStorage storage = new HdfsStorage(config);
} catch (CheckpointStorageException e) {
  if (e.getMessage().equals("Failed to get file system")) {
    log.error("HDFS unavailable or misconfigured; check fs.defaultFS, NameNode, and hadoop conf classpath", e);
  } else throw e;
}

When it happens

Trigger: Calling initStorage (directly or via the HdfsStorage constructor) with a configuration whose fs.defaultFS/storage.type points to an unreachable or misconfigured filesystem: wrong hdfs host/port, NameNode down, kerberos misconfig, or missing hdfs client jars.

Common situations: checkpoint-storage-hdfs plugin configured with 'storage.type=hdfs' but Hadoop conf (core-site.xml/hdfs-site.xml) not on the classpath, HDFS cluster down or network firewalled, typo in the fs.defaultFS URL, or running on a node without the required Hadoop native libs.

Related errors

• Checking catalog path %s exists exception.
• Failed to serialize checkpoint data, state: ${state}
• Failed to write checkpoint data, file: ${tmpFilePath}, state
• Failed to rename tmp file to final file
• No checkpoint found for job, job id is: ${jobId}

AI-assisted analysis of apache/seatunnel@cf67b549a7 (2026-09-10). Data as JSON: /api/errors/d6738bd49837f87a. Report an issue: GitHub.