apache/seatunnel · critical · CheckpointStorageException
Failed to write checkpoint data, file: ${tmpFilePath}, state
Error message
Failed to write checkpoint data, file: ${tmpFilePath}, state: ${state} What it means
After serializing, storeCheckPoint writes the bytes to a temporary HDFS file (final name + tmp suffix) via fs.create/FSDataOutputStream. An IOException during create or write is wrapped in a CheckpointStorageException identifying the temp file path and the state, so the developer can see exactly which file failed to write.
Source
Thrown at seatunnel-engine/seatunnel-engine-storage/checkpoint-storage-plugins/checkpoint-storage-hdfs/src/main/java/org/apache/seatunnel/engine/checkpoint/storage/hdfs/HdfsStorage.java:111
}
Path filePath =
new Path(
getStorageParentDirectory()
+ state.getJobId()
+ "/"
+ getCheckPointName(state));
Path tmpFilePath =
new Path(
getStorageParentDirectory()
+ state.getJobId()
+ "/"
+ getCheckPointName(state)
+ STORAGE_TMP_SUFFIX);
try (FSDataOutputStream out = fs.create(tmpFilePath, false)) {
out.write(datas);
} catch (IOException e) {
throw new CheckpointStorageException(
String.format(
"Failed to write checkpoint data, file: %s, state: %s",
tmpFilePath, state),
e);
}
try {
boolean success = fs.rename(tmpFilePath, filePath);
if (!success) {
throw new CheckpointStorageException("Failed to rename tmp file to final file");
}
} catch (IOException e) {
throw new CheckpointStorageException("Failed to rename tmp file to final file");
} finally {
try {
// clean up tmp file, if still lying around
if (fs.exists(tmpFilePath)) {
fs.delete(tmpFilePath, false);View on GitHub (pinned to cf67b549a7)
Solutions
- Check HDFS health: hdfs dfsadmin -report and dfsadmin -safemode get; clear safe mode or wait
- Free HDFS space or expand the cluster if disk/inode usage is near capacity
- Verify write permissions for the SeaTunnel user on the checkpoint parent directory
- Look for leftover .tmp files and stale writers; avoid running duplicate jobs with the same jobId
- Inspect the nested IOException cause for the specific DataNode/NameNode error
Defensive patterns
Strategy: retry
Validate before calling
// pre-flight: writable target dir and adequate space if (!fs.exists(parentDir)) fs.mkdirs(parentDir); // check quota/space via hdfs dfsadmin -report / dfs quota commands
Try / catch
try {
storage.storeCheckPoint(state);
} catch (CheckpointStorageException e) {
if (e.getMessage().startsWith("Failed to write checkpoint data")) {
log.error("HDFS write failed for " + tmpFilePath + ", retrying after health check", e);
// check safemode/disk space, then retry with backoff
} else throw e;
} Prevention
- Monitor HDFS disk/inode usage and directory quotas
- Ensure the SeaTunnel user has write permission on the checkpoint directory
- Avoid duplicate jobs sharing the same jobId and checkpoint paths
- Keep the cluster out of safe mode; alert on DataNode loss
When it happens
Trigger: storeCheckPoint (also via modifyResumeTokenInCheckpoint) when fs.create(tmpFilePath, false) or out.write(datas) throws IOException: HDFS disk full, DataNode unavailable, file already exists with create(false), lease recovery conflicts, or NameNode in safe mode.
Common situations: HDFS cluster at capacity (disk or inodes), flaky network to DataNodes, concurrent writers colliding on the same tmp file name for the same job/pipeline, HDFS in safe mode after restart, permission problems on the checkpoint directory.
Understand the failure class
Background: "failed to write file", "Could not save figure", "Error saving remote file" — file write failed: causes and fixes across languages and libraries — this error's family across 38 libraries.
Related errors
- Failed to get file system
- Failed to serialize checkpoint data, state: ${state}
- Failed to rename tmp file to final file
- No checkpoint found for job, job id is: ${jobId}
- FILE_LIST_GET_FAILED
AI-assisted analysis of apache/seatunnel@cf67b549a7 (2026-09-10).
Data as JSON: /api/errors/9d07933ef30cc0fa.
Report an issue: GitHub.