apache/seatunnel · error · CheckpointStorageException

Failed to list files from names${path}

Error message

Failed to list files from names${path}

What it means

HdfsStorage.getFileNames lists files under a directory on the HDFS filesystem; if the underlying FileSystem.listStatus (or equivalent) call throws IOException, it wraps it in CheckpointStorageException with the path included. This is an I/O-level failure to enumerate the directory, not a 'no files' condition.

Source

Thrown at seatunnel-engine/seatunnel-engine-storage/checkpoint-storage-plugins/checkpoint-storage-hdfs/src/main/java/org/apache/seatunnel/engine/checkpoint/storage/hdfs/HdfsStorage.java:350

                });
    }

    public List<String> getFileNames(String path) throws CheckpointStorageException {
        try {
            Path parentPath = new Path(path);
            if (!fs.exists(parentPath)) {
                log.info("Path " + path + " is not a directory");
                return new ArrayList<>();
            }
            FileStatus[] fileStatus =
                    fs.listStatus(parentPath, path1 -> path1.getName().endsWith(FILE_FORMAT));
            List<String> fileNames = new ArrayList<>();
            for (FileStatus status : fileStatus) {
                fileNames.add(status.getPath().getName());
            }
            return fileNames;
        } catch (IOException e) {
            throw new CheckpointStorageException("Failed to list files from names" + path, e);
        }
    }

    /**
     * Get checkpoint name
     *
     * @param fileName file name
     * @return checkpoint data
     */
    private PipelineState readPipelineState(String fileName, String jobId)
            throws CheckpointStorageException {
        fileName =
                getStorageParentDirectory() + jobId + DEFAULT_CHECKPOINT_FILE_PATH_SPLIT + fileName;
        try (FSDataInputStream in = fs.open(new Path(fileName));
                ByteArrayOutputStream stream = new ByteArrayOutputStream()) {
            IOUtils.copyBytes(in, stream, 1024);
            byte[] bytes = stream.toByteArray();
            return deserializeCheckPointData(bytes);

View on GitHub (pinned to cf67b549a7)

Solutions

1. Check HDFS connectivity and that the Namenode is up (hdfs dfs -ls <path> works from the same host).
2. Verify the storage parent directory path and fs configuration (fs.defaultFS, core-site.xml/hdfs-site.xml on classpath).
3. Check HDFS permissions for the running user on the job directory.
4. Retry with backoff if the cluster was temporarily unavailable.

Example fix

// before
List<String> names = storage.getAllCheckpoints();
// after
Retryer<String> retryer = Retryer.ofBackoff(3, Duration.ofSeconds(2));
List<String> names = retryer.call(() -> storage.getAllCheckpoints());

// Pre-check HDFS reachability before listing
org.apache.hadoop.fs.FileSystem fs =
    org.apache.hadoop.fs.FileSystem.get(conf);
boolean reachable = fs.exists(new org.apache.hadoop.fs.Path(storageParentDir));

boolean jobDirListable(String jobId) {
    try { storage.getCheckpointsByJobIdAndPipelineId(jobId, pipelineId); return true; }
    catch (CheckpointStorageException e) { return false; }
}

int attempts = 0;
while (true) {
    try { return storage.getAllCheckpoints(); }
    catch (CheckpointStorageException e) {
        if (e.getMessage().startsWith("Failed to list files") && ++attempts < 3) {
            Thread.sleep(1000L * attempts); continue;
        }
        throw e;
    }
}

When it happens

Trigger: Calling any API that lists job directories (getCheckpoint, deleteCheckpoint, getAllCheckpoints) when HDFS is unreachable, the directory does not exist on that filesystem, or listing fails due to permissions/Namenode errors.

Common situations: Wrong fs.defaultFS / storage path configuration; HDFS Namenode down or in safe mode; Kerberos/auth issues causing access failures; network partition between client and HDFS.

Understand the failure class

Background: "failed to read file", EACCES, ENOENT and "could not read <path>" errors: when a program can't read a file from disk — this error's family across 49 libraries.

Related errors

• Failed to get file system
• load all data error
• load all keys error parent path is ${businessRootPath}
• FILE_LIST_GET_FAILED
• Failed during target_bulk_listing for path=%s

AI-assisted analysis of apache/seatunnel@cf67b549a7 (2026-09-10). Data as JSON: /api/errors/8aefea0686874cd4. Report an issue: GitHub.