apache/seatunnel · warning

Failed to delete checkpoint for job {}

Error message

Failed to delete checkpoint for job {}

What it means

HdfsStorage.deleteCheckpoint removes the job's checkpoint directory recursively on HDFS. On IOException the deletion failure is logged at WARN and swallowed — the method does not throw — so orphaned checkpoint data can remain in HDFS and consume storage. Job cleanup continues elsewhere.

Source

Thrown at seatunnel-engine/seatunnel-engine-storage/checkpoint-storage-plugins/checkpoint-storage-hdfs/src/main/java/org/apache/seatunnel/engine/checkpoint/storage/hdfs/HdfsStorage.java:242

                    String filePipelineId = getPipelineIdByFileName(file);
                    if (pipelineId.equals(filePipelineId)) {
                        try {
                            pipelineStates.add(readPipelineState(file, jobId));
                        } catch (Exception e) {
                            log.error("Failed to read checkpoint data from file " + file, e);
                        }
                    }
                });
        return pipelineStates;
    }

    @Override
    public void deleteCheckpoint(String jobId) {
        String jobPath = getStorageParentDirectory() + jobId;
        try {
            fs.delete(new Path(jobPath), true);
        } catch (IOException e) {
            log.warn("Failed to delete checkpoint for job {}", jobId, e);
        }
    }

    @Override
    public PipelineState getCheckpoint(String jobId, String pipelineId, String checkpointId)
            throws CheckpointStorageException {
        String path = getStorageParentDirectory() + jobId;
        List<String> fileNames = getFileNames(path);
        if (fileNames.isEmpty()) {
            log.info("No checkpoint found for this job,  the job id is: " + jobId);
            return null;
        }
        for (String fileName : fileNames) {
            if (pipelineId.equals(getPipelineIdByFileName(fileName))
                    && checkpointId.equals(getCheckpointIdByFileName(fileName))) {
                try {
                    return readPipelineState(fileName, jobId);
                } catch (Exception e) {

View on GitHub (pinned to cf67b549a7)

Solutions

1. Check HDFS cluster health (NameNode reachable, not in safe mode) at cleanup time.
2. Verify the SeaTunnel process user has delete permission on the checkpoint storage parent directory.
3. Fix/verify HDFS configuration (fs.defaultFS, Kerberos tokens) used by checkpoint-storage-hdfs.
4. Manually delete the orphaned jobPath (getStorageParentDirectory()+jobId) from HDFS once the cluster is healthy.

Example fix

// before: user lacks delete permission on job path
hdfs dfs -chmod 755 /tmp/seatunnel/checkpoints
// after: grant write/delete to seatunnel user
hdfs dfs -chmod -R 770 /tmp/seatunnel/checkpoints && hdfs dfs -chown -R seatunnel /tmp/seatunnel/checkpoints

// pre-check HDFS availability and permissions
try (FileSystem fs = FileSystem.get(conf)) {
    Path root = new Path(checkpointRoot);
    if (!fs.exists(root) || !fs.getFileStatus(root).getPermission().toString().contains("wx")) {
        log.warn("checkpoint root missing or not deletable: {}", root);
    }
}

try {
    hdfsStorage.deleteCheckpoint(jobId);
} catch (Throwable t) {
    // engine swallows IOException; schedule manual HDFS cleanup
    scheduleManualHdfsCleanup("/tmp/seatunnel/checkpoint-storage/" + jobId);
}

When it happens

Trigger: fs.delete(new Path(jobPath), true) throws IOException: HDFS NameNode unreachable, permission denied for the submitting user on the checkpoint parent dir, or filesystem/HA configuration issues.

Common situations: HDFS in safe mode or NameNode outage during job cleanup; the SeaTunnel user lacking write/delete permission on the storage root; Kerberos/credentials expired; wrong fs.defaultFS config so the path cannot be resolved.

Related errors

• Failed to read checkpoint data, file name is ${fileName},job
• Failed to delete checkpoint directory
• Unable to delete directory " + localFileDir
• FILE_LIST_GET_FAILED
• Failed during target_bulk_listing for path=%s

AI-assisted analysis of apache/seatunnel@cf67b549a7 (2026-09-10). Data as JSON: /api/errors/4342c65287d5b772. Report an issue: GitHub.