{"record":{"id":"4342c65287d5b772","repo":"apache/seatunnel","slug":"failed-to-delete-checkpoint-for-job","errorCode":null,"errorMessage":"Failed to delete checkpoint for job {}","messagePattern":"Failed to delete checkpoint for job (.+?)","errorType":"console","errorClass":null,"httpStatus":null,"severity":"warning","filePath":"seatunnel-engine/seatunnel-engine-storage/checkpoint-storage-plugins/checkpoint-storage-hdfs/src/main/java/org/apache/seatunnel/engine/checkpoint/storage/hdfs/HdfsStorage.java","lineNumber":242,"sourceCode":"                    String filePipelineId = getPipelineIdByFileName(file);\n                    if (pipelineId.equals(filePipelineId)) {\n                        try {\n                            pipelineStates.add(readPipelineState(file, jobId));\n                        } catch (Exception e) {\n                            log.error(\"Failed to read checkpoint data from file \" + file, e);\n                        }\n                    }\n                });\n        return pipelineStates;\n    }\n\n    @Override\n    public void deleteCheckpoint(String jobId) {\n        String jobPath = getStorageParentDirectory() + jobId;\n        try {\n            fs.delete(new Path(jobPath), true);\n        } catch (IOException e) {\n            log.warn(\"Failed to delete checkpoint for job {}\", jobId, e);\n        }\n    }\n\n    @Override\n    public PipelineState getCheckpoint(String jobId, String pipelineId, String checkpointId)\n            throws CheckpointStorageException {\n        String path = getStorageParentDirectory() + jobId;\n        List<String> fileNames = getFileNames(path);\n        if (fileNames.isEmpty()) {\n            log.info(\"No checkpoint found for this job,  the job id is: \" + jobId);\n            return null;\n        }\n        for (String fileName : fileNames) {\n            if (pipelineId.equals(getPipelineIdByFileName(fileName))\n                    && checkpointId.equals(getCheckpointIdByFileName(fileName))) {\n                try {\n                    return readPipelineState(fileName, jobId);\n                } catch (Exception e) {","sourceCodeStart":224,"sourceCodeEnd":260,"githubUrl":"https://github.com/apache/seatunnel/blob/cf67b549a7a6c35fa0beb12d83c62892427ea919/seatunnel-engine/seatunnel-engine-storage/checkpoint-storage-plugins/checkpoint-storage-hdfs/src/main/java/org/apache/seatunnel/engine/checkpoint/storage/hdfs/HdfsStorage.java#L224-L260","documentation":"HdfsStorage.deleteCheckpoint removes the job's checkpoint directory recursively on HDFS. On IOException the deletion failure is logged at WARN and swallowed — the method does not throw — so orphaned checkpoint data can remain in HDFS and consume storage. Job cleanup continues elsewhere.","triggerScenarios":"fs.delete(new Path(jobPath), true) throws IOException: HDFS NameNode unreachable, permission denied for the submitting user on the checkpoint parent dir, or filesystem/HA configuration issues.","commonSituations":"HDFS in safe mode or NameNode outage during job cleanup; the SeaTunnel user lacking write/delete permission on the storage root; Kerberos/credentials expired; wrong fs.defaultFS config so the path cannot be resolved.","solutions":["Check HDFS cluster health (NameNode reachable, not in safe mode) at cleanup time.","Verify the SeaTunnel process user has delete permission on the checkpoint storage parent directory.","Fix/verify HDFS configuration (fs.defaultFS, Kerberos tokens) used by checkpoint-storage-hdfs.","Manually delete the orphaned jobPath (getStorageParentDirectory()+jobId) from HDFS once the cluster is healthy."],"exampleFix":"// before: user lacks delete permission on job path\nhdfs dfs -chmod 755 /tmp/seatunnel/checkpoints\n// after: grant write/delete to seatunnel user\nhdfs dfs -chmod -R 770 /tmp/seatunnel/checkpoints && hdfs dfs -chown -R seatunnel /tmp/seatunnel/checkpoints","handlingStrategy":"retry","validationCode":"// pre-check HDFS availability and permissions\ntry (FileSystem fs = FileSystem.get(conf)) {\n    Path root = new Path(checkpointRoot);\n    if (!fs.exists(root) || !fs.getFileStatus(root).getPermission().toString().contains(\"wx\")) {\n        log.warn(\"checkpoint root missing or not deletable: {}\", root);\n    }\n}","typeGuard":null,"tryCatchPattern":"try {\n    hdfsStorage.deleteCheckpoint(jobId);\n} catch (Throwable t) {\n    // engine swallows IOException; schedule manual HDFS cleanup\n    scheduleManualHdfsCleanup(\"/tmp/seatunnel/checkpoint-storage/\" + jobId);\n}","preventionTips":["Monitor NameNode health and safe-mode status during job cleanup windows.","Grant the SeaTunnel user delete permission on the checkpoint storage root.","Keep Kerberos/HDFS credentials valid for the lifetime of the cluster.","Periodically scan the checkpoint root for orphaned jobId directories."],"tags":["hdfs","checkpoint","cleanup","io"],"backgroundTag":"file-delete-failed","analyzedSha":"cf67b549a7a6c35fa0beb12d83c62892427ea919","analyzedAt":"2026-09-10T21:44:55.265Z","contentChangedAt":"2026-09-10T21:44:55.265Z","schemaVersion":2},"datasetVersion":"2026-09-14T11:17:12.474Z"}