{"record":{"id":"2474a28f6d5ebc55","repo":"apache/flink","slug":"cannot-clean-commit-staging-file-does-not-exist-2474a2","errorCode":null,"errorMessage":"Cannot clean commit: Staging file does not exist.","messagePattern":"Cannot clean commit: Staging file does not exist\\.","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"error","filePath":"flink-filesystems/flink-hadoop-fs/src/main/java/org/apache/flink/runtime/fs/hdfs/HadoopRecoverableFsDataOutputStream.java","lineNumber":236,"sourceCode":"        private final FileSystem fs;\n        private final HadoopFsRecoverable recoverable;\n\n        HadoopFsCommitter(FileSystem fs, HadoopFsRecoverable recoverable) {\n            this.fs = checkNotNull(fs);\n            this.recoverable = checkNotNull(recoverable);\n        }\n\n        @Override\n        public void commit() throws IOException {\n            final Path src = recoverable.tempFile();\n            final Path dest = recoverable.targetFile();\n            final long expectedLength = recoverable.offset();\n\n            final FileStatus srcStatus;\n            try {\n                srcStatus = fs.getFileStatus(src);\n            } catch (IOException e) {\n                throw new IOException(\"Cannot clean commit: Staging file does not exist.\");\n            }\n\n            if (srcStatus.getLen() != expectedLength) {\n                // something was done to this file since the committer was created.\n                // this is not the \"clean\" case\n                throw new IOException(\"Cannot clean commit: File has trailing junk data.\");\n            }\n\n            try {\n                fs.rename(src, dest);\n            } catch (IOException e) {\n                throw new IOException(\n                        \"Committing file by rename failed: \" + src + \" to \" + dest, e);\n            }\n        }\n\n        @Override\n        public void commitAfterRecovery() throws IOException {","sourceCodeStart":218,"sourceCodeEnd":254,"githubUrl":"https://github.com/apache/flink/blob/2f3c205e9266cb30240eb7f4fdab15cad629a70f/flink-filesystems/flink-hadoop-fs/src/main/java/org/apache/flink/runtime/fs/hdfs/HadoopRecoverableFsDataOutputStream.java#L218-L254","documentation":"HadoopFsCommitter.commit() (the clean, non-recovery path) fetches the FileStatus of the staging/temp file recorded in the HadoopFsRecoverable before renaming it to the final location. If fs.getFileStatus(src) throws any IOException, it is rethrown wrapped as 'Staging file does not exist' - i.e. the temp file named by the recoverable is gone from HDFS.","triggerScenarios":"Calling commit() on a Committer recovered from a checkpoint/savepoint after the in-progress temp file was deleted or never written; using a recoverable from a different cluster/nameservice; HDFS permission or connectivity problems on getFileStatus also surface as this message because the catch is broad.","commonSituations":"Resuming from an old savepoint whose .inprogress files were cleaned up by retention jobs or manually; HDFS trash/route differences after cluster migration; sink part files removed by an external process; two jobs committing from the same savepoint.","solutions":["Verify the temp file path from the recoverable actually exists on HDFS (hdfs dfs -ls on the in-progress directory) before resuming","Start a fresh pipeline (new output path or allow part re-creation) when the in-progress file is permanently lost","Ensure checkpoints/savepoints are recent enough that staging files still exist, and that no retention policy deletes them","Rule out permission/NN connectivity issues: the same message wraps non-FileNotFoundException IOErrors from getFileStatus"],"exampleFix":null,"handlingStrategy":"validation","validationCode":"// Before committing, verify the staging file exists\norg.apache.hadoop.fs.Path temp = recoverable.tempFile();\nif (!fs.exists(temp)) {\n    throw new IOException(\"Staging file missing, cannot commit: \" + temp);\n}","typeGuard":null,"tryCatchPattern":"catch (IOException e) {\n    if (e.getMessage() != null && e.getMessage().contains(\"Staging file does not exist\")) {\n        // recoverable is stale: drop this pending file or restart part, then continue\n    } else { throw e; }\n}","preventionTips":["Keep in-progress files under the checkpoint data dir so retention follows checkpoints","Never manually delete the sink's in-progress directory","Resume from the latest checkpoint, not ancient savepoints","Ensure HDFS access/permissions for the job user before resume"],"tags":["hdfs","file-sink","commit","staging-file","recovery"],"backgroundTag":null,"analyzedSha":"2f3c205e9266cb30240eb7f4fdab15cad629a70f","analyzedAt":"2026-08-14T08:48:24.518Z","schemaVersion":2},"datasetVersion":"2026-08-14T10:17:34.591Z"}