{"record":{"id":"d6738bd49837f87a","repo":"apache/seatunnel","slug":"failed-to-get-file-system","errorCode":null,"errorMessage":"Failed to get file system","messagePattern":"Failed to get file system","errorType":"exception","errorClass":"CheckpointStorageException","httpStatus":null,"severity":"critical","filePath":"seatunnel-engine/seatunnel-engine-storage/checkpoint-storage-plugins/checkpoint-storage-hdfs/src/main/java/org/apache/seatunnel/engine/checkpoint/storage/hdfs/HdfsStorage.java","lineNumber":71,"sourceCode":"    public FileSystem fs;\n    private static final String STORAGE_TMP_SUFFIX = \"tmp\";\n    private static final String STORAGE_TYPE_KEY = \"storage.type\";\n\n    public HdfsStorage(Map<String, String> configuration) throws CheckpointStorageException {\n        this.initStorage(configuration);\n    }\n\n    @Override\n    public void initStorage(Map<String, String> configuration) throws CheckpointStorageException {\n        if (StringUtils.isNotBlank(configuration.get(STORAGE_NAME_SPACE))) {\n            setStorageNameSpace(configuration.get(STORAGE_NAME_SPACE));\n            configuration.remove(STORAGE_NAME_SPACE);\n        }\n        Configuration hadoopConf = getConfiguration(configuration);\n        try {\n            fs = FileSystem.get(hadoopConf);\n        } catch (IOException e) {\n            throw new CheckpointStorageException(\"Failed to get file system\", e);\n        }\n    }\n\n    private Configuration getConfiguration(Map<String, String> config)\n            throws CheckpointStorageException {\n        String storageType =\n                config.getOrDefault(STORAGE_TYPE_KEY, FileConfiguration.LOCAL.toString());\n        config.remove(STORAGE_TYPE_KEY);\n        AbstractConfiguration configuration =\n                FileConfiguration.valueOf(storageType.toUpperCase()).getConfiguration();\n        return configuration.buildConfiguration(config);\n    }\n\n    @Override\n    public String storeCheckPoint(PipelineState state) throws CheckpointStorageException {\n        byte[] datas;\n        try {\n            datas = serializeCheckPointData(state);","sourceCodeStart":53,"sourceCodeEnd":89,"githubUrl":"https://github.com/apache/seatunnel/blob/cf67b549a7a6c35fa0beb12d83c62892427ea919/seatunnel-engine/seatunnel-engine-storage/checkpoint-storage-plugins/checkpoint-storage-hdfs/src/main/java/org/apache/seatunnel/engine/checkpoint/storage/hdfs/HdfsStorage.java#L53-L89","documentation":"HdfsStorage.initStorage obtains a Hadoop FileSystem handle from the configured storage type/config. If FileSystem.get(hadoopConf) throws an IOException — bad HDFS URI, missing Hadoop configuration, unreachable NameNode, or missing filesystem implementation on the classpath (wrapped as IOException in some paths) — the plugin fails fast with this CheckpointStorageException.","triggerScenarios":"Calling initStorage (directly or via the HdfsStorage constructor) with a configuration whose fs.defaultFS/storage.type points to an unreachable or misconfigured filesystem: wrong hdfs host/port, NameNode down, kerberos misconfig, or missing hdfs client jars.","commonSituations":"checkpoint-storage-hdfs plugin configured with 'storage.type=hdfs' but Hadoop conf (core-site.xml/hdfs-site.xml) not on the classpath, HDFS cluster down or network firewalled, typo in the fs.defaultFS URL, or running on a node without the required Hadoop native libs.","solutions":["Verify hdfs dfs -ls / works from the same node/user with the same configs","Check the storage config: fs.defaultFS / storage.type values and that Hadoop conf XMLs are on the classpath (HADOOP_CONF_DIR)","Confirm the NameNode is reachable (host, port, DNS, firewall)","Ensure required Hadoop/HDFS dependency jars are in the SeaTunnel connector lib directory","If HA nameservice is used, include all dfs.nameservices settings"],"exampleFix":"// before\nHdfsStorage hdfs = new HdfsStorage(Map.of(\"storage.type\", \"hdfs\")); // no fs config\n// after\nHdfsStorage hdfs = new HdfsStorage(Map.of(\n    \"storage.type\", \"hdfs\",\n    \"fs.defaultFS\", \"hdfs://namenode:8020\"));","handlingStrategy":"validation","validationCode":"// before constructing HdfsStorage, verify the filesystem is reachable\nConfiguration conf = new Configuration();\nconf.set(\"fs.defaultFS\", config.get(\"fs.defaultFS\"));\nFileSystem fs = FileSystem.get(conf);\nif (!fs.exists(new Path(baseDir))) fs.mkdirs(new Path(baseDir));","typeGuard":null,"tryCatchPattern":"try {\n  HdfsStorage storage = new HdfsStorage(config);\n} catch (CheckpointStorageException e) {\n  if (e.getMessage().equals(\"Failed to get file system\")) {\n    log.error(\"HDFS unavailable or misconfigured; check fs.defaultFS, NameNode, and hadoop conf classpath\", e);\n  } else throw e;\n}","preventionTips":["Smoke-test HDFS access with hdfs dfs -ls from the same node/user before starting jobs","Keep HADOOP_CONF_DIR or core-site.xml/hdfs-site.xml on the SeaTunnel classpath","Pin connector Hadoop dependency versions and verify jars in the plugin directory","Monitor NameNode health and network reachability from worker nodes"],"tags":["hdfs","checkpoint-storage","filesystem"],"backgroundTag":"module-init-failed","analyzedSha":"cf67b549a7a6c35fa0beb12d83c62892427ea919","analyzedAt":"2026-09-10T21:44:55.265Z","contentChangedAt":"2026-09-10T21:44:55.265Z","schemaVersion":2},"datasetVersion":"2026-09-14T11:17:12.474Z"}