apache/flink · error · IOException

Cannot clean commit: Staging file does not exist.

Error message

Cannot clean commit: Staging file does not exist.

What it means

HadoopFsCommitter.commit() (the clean, non-recovery path) fetches the FileStatus of the staging/temp file recorded in the HadoopFsRecoverable before renaming it to the final location. If fs.getFileStatus(src) throws any IOException, it is rethrown wrapped as 'Staging file does not exist' - i.e. the temp file named by the recoverable is gone from HDFS.

Source

Thrown at flink-filesystems/flink-hadoop-fs/src/main/java/org/apache/flink/runtime/fs/hdfs/HadoopRecoverableFsDataOutputStream.java:236

        private final FileSystem fs;
        private final HadoopFsRecoverable recoverable;

        HadoopFsCommitter(FileSystem fs, HadoopFsRecoverable recoverable) {
            this.fs = checkNotNull(fs);
            this.recoverable = checkNotNull(recoverable);
        }

        @Override
        public void commit() throws IOException {
            final Path src = recoverable.tempFile();
            final Path dest = recoverable.targetFile();
            final long expectedLength = recoverable.offset();

            final FileStatus srcStatus;
            try {
                srcStatus = fs.getFileStatus(src);
            } catch (IOException e) {
                throw new IOException("Cannot clean commit: Staging file does not exist.");
            }

            if (srcStatus.getLen() != expectedLength) {
                // something was done to this file since the committer was created.
                // this is not the "clean" case
                throw new IOException("Cannot clean commit: File has trailing junk data.");
            }

            try {
                fs.rename(src, dest);
            } catch (IOException e) {
                throw new IOException(
                        "Committing file by rename failed: " + src + " to " + dest, e);
            }
        }

        @Override
        public void commitAfterRecovery() throws IOException {

View on GitHub (pinned to 2f3c205e92)

Solutions

Verify the temp file path from the recoverable actually exists on HDFS (hdfs dfs -ls on the in-progress directory) before resuming
Start a fresh pipeline (new output path or allow part re-creation) when the in-progress file is permanently lost
Ensure checkpoints/savepoints are recent enough that staging files still exist, and that no retention policy deletes them
Rule out permission/NN connectivity issues: the same message wraps non-FileNotFoundException IOErrors from getFileStatus

Defensive patterns

Strategy: validation

Validate before calling

// Before committing, verify the staging file exists
org.apache.hadoop.fs.Path temp = recoverable.tempFile();
if (!fs.exists(temp)) {
    throw new IOException("Staging file missing, cannot commit: " + temp);
}

Try / catch

catch (IOException e) {
    if (e.getMessage() != null && e.getMessage().contains("Staging file does not exist")) {
        // recoverable is stale: drop this pending file or restart part, then continue
    } else { throw e; }
}

Prevention

Keep in-progress files under the checkpoint data dir so retention follows checkpoints
Never manually delete the sink's in-progress directory
Resume from the latest checkpoint, not ancient savepoints
Ensure HDFS access/permissions for the job user before resume

When it happens

Trigger: Calling commit() on a Committer recovered from a checkpoint/savepoint after the in-progress temp file was deleted or never written; using a recoverable from a different cluster/nameservice; HDFS permission or connectivity problems on getFileStatus also surface as this message because the catch is broad.

Common situations: Resuming from an old savepoint whose .inprogress files were cleaned up by retention jobs or manually; HDFS trash/route differences after cluster migration; sink part files removed by an external process; two jobs committing from the same savepoint.

Related errors

AI-assisted analysis of apache/flink@2f3c205e92 (2026-08-14). Data as JSON: /api/errors/2474a28f6d5ebc55. Report an issue: GitHub.