apache/seatunnel · warning

Failed to delete checkpoint directory

Error message

Failed to delete checkpoint directory 

What it means

LocalFileStorage.deleteCheckpoint deletes the job's checkpoint directory recursively via FileUtils.deleteDirectory. On IOException the failure is only logged at WARN and swallowed, leaving orphaned checkpoint files on local disk. Storage may grow over time and job cleanup is only partially done.

Source

Thrown at seatunnel-engine/seatunnel-engine-storage/checkpoint-storage-plugins/checkpoint-storage-local-file/src/main/java/org/apache/seatunnel/engine/checkpoint/storage/localfile/LocalFileStorage.java:281

                        } catch (IOException e) {
                            log.error(
                                    "Failed to read checkpoint data from file "
                                            + file.getAbsolutePath(),
                                    e);
                        }
                    }
                });
        return pipelineStates;
    }

    @Override
    public void deleteCheckpoint(String jobId) {
        String jobPath = getStorageParentDirectory() + jobId;
        File file = new File(jobPath);
        try {
            FileUtils.deleteDirectory(file);
        } catch (IOException e) {
            log.warn("Failed to delete checkpoint directory " + jobPath, e);
        }
    }

    @Override
    public PipelineState getCheckpoint(String jobId, String pipelineId, String checkpointId)
            throws CheckpointStorageException {
        String parentPath = getStorageParentDirectory() + jobId;
        Collection<File> fileList = new ArrayList<>();
        try {
            fileList = FileUtils.listFiles(new File(parentPath), FILE_EXTENSIONS, false);
        } catch (Exception e) {
            if (!(e.getCause() instanceof NoSuchFileException)) {
                throw new CheckpointStorageException(ExceptionUtils.getMessage(e));
            }
        }
        if (fileList.isEmpty()) {
            log.info("No checkpoint found for this job,  the job id is: " + jobId);
            return null;

View on GitHub (pinned to cf67b549a7)

Solutions

1. Ensure the job is fully terminated before cleanup and only one cleanup path exists (avoid deleting checkpoints of a still-running job).
2. Check permissions/ownership of the local checkpoint storage directory for the SeaTunnel process user.
3. Move checkpoint storage off tmpfs/tmpwatch-managed locations or exclude it from OS cleanup.
4. Manually remove the orphaned jobPath directory once nothing is using it.

Example fix

// before: checkpoints under /tmp removed by tmpwatch mid-delete
storage-dir: /tmp/seatunnel/checkpoint-storage
// after
storage-dir: /var/lib/seatunnel/checkpoint-storage

// pre-check local checkpoint dir is deletable
File dir = new File(storageRoot, jobId);
if (dir.exists() && !dir.canWrite()) {
    log.warn("cannot delete checkpoint dir {}", dir);
}

boolean isDeletableDir(File d) {
    return d.isDirectory() && d.canWrite();
}

try {
    localFileStorage.deleteCheckpoint(jobId);
} catch (Throwable t) {
    // engine swallows IOException; orphaned dir may remain
    scheduleManualCleanup(new File(storageRoot, jobId));
}

When it happens

Trigger: FileUtils.deleteDirectory(new File(getStorageParentDirectory()+jobId)) throws IOException — file locked/in use, permission denied, or the directory disappears mid-delete (concurrent cleanup).

Common situations: Checkpoint dir under /tmp cleaned by OS tmpwatch/systemd-tmpfiles during delete; disk permission issues after running SeaTunnel as different users; a running job on the same node still writing checkpoints while another process deletes them.

Related errors

• Failed to delete checkpoint for job {}
• Unable to delete directory " + localFileDir
• Error closing MQTT source client
• Failed to read checkpoint data, file name is ${fileName},job
• Failed to serialize checkpoint data

AI-assisted analysis of apache/seatunnel@cf67b549a7 (2026-09-10). Data as JSON: /api/errors/4c486233e9056f74. Report an issue: GitHub.