{"record":{"id":"0a4b8f59c71db74a","repo":"apache/flink","slug":"cannot-clean-commit-file-has-trailing-junk-data","errorCode":null,"errorMessage":"Cannot clean commit: File has trailing junk data.","messagePattern":"Cannot clean commit: File has trailing junk data\\.","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"error","filePath":"flink-core/src/main/java/org/apache/flink/core/fs/local/LocalRecoverableFsDataOutputStream.java","lineNumber":156,"sourceCode":"\n    static class LocalCommitter implements Committer {\n\n        private final LocalRecoverable recoverable;\n\n        LocalCommitter(LocalRecoverable recoverable) {\n            this.recoverable = checkNotNull(recoverable);\n        }\n\n        @Override\n        public void commit() throws IOException {\n            final File src = recoverable.tempFile();\n            final File dest = recoverable.targetFile();\n\n            // sanity check\n            if (src.length() != recoverable.offset()) {\n                // something was done to this file since the committer was created.\n                // this is not the \"clean\" case\n                throw new IOException(\"Cannot clean commit: File has trailing junk data.\");\n            }\n\n            // rather than fall into default recovery, handle errors explicitly\n            // in order to improve error messages\n            try {\n                Files.move(src.toPath(), dest.toPath(), StandardCopyOption.ATOMIC_MOVE);\n            } catch (UnsupportedOperationException | AtomicMoveNotSupportedException e) {\n                if (!src.renameTo(dest)) {\n                    throw new IOException(\n                            \"Committing file failed, could not rename \" + src + \" -> \" + dest);\n                }\n            } catch (FileAlreadyExistsException e) {\n                throw new IOException(\n                        \"Committing file failed. Target file already exists: \" + dest);\n            }\n        }\n\n        @Override","sourceCodeStart":138,"sourceCodeEnd":174,"githubUrl":"https://github.com/apache/flink/blob/2f3c205e9266cb30240eb7f4fdab15cad629a70f/flink-core/src/main/java/org/apache/flink/core/fs/local/LocalRecoverableFsDataOutputStream.java#L138-L174","documentation":"Thrown by LocalRecoverableFsDataOutputStream.commit() when the temp file's current length differs from the offset recorded in the recoverable. The 'clean commit' path requires the file to be exactly at the persisted offset; any extra bytes mean 'junk data' was appended after persist(), so the file is not in the expected state for an atomic rename.","triggerScenarios":"Calling commit() on a LocalRecoverableFsDataOutputStream after data was written to the temp file beyond the persisted offset (e.g. writer kept writing between persist and commit).","commonSituations":"Race between persist() and continued writes; double-commit; a writer that did not stop writing after creating the committer; recovery scenarios where commit() is called instead of commitAfterRecovery() despite extra data.","solutions":["Stop writing to the stream immediately after persist() and before commit().","Use commitAfterRecovery() when trailing data may exist; it truncates junk before moving.","Ensure exactly one commit per recoverable and no post-persist writes.","Audit the writer lifecycle so persist->commit is atomic with respect to writes."],"exampleFix":"// before\nRecoverableWriter.CommitRecoverable rec = writer.persistForRecoverySafely();\nout.write(extra); // appends junk\ncommitter.commit(); // throws\n\n// after\nRecoverableWriter.CommitRecoverable rec = writer.persistForRecoverySafely();\n// no writes after persist\ncommitter.commit();","handlingStrategy":"validation","validationCode":"void safeCommit(LocalRecoverable r) throws IOException {\n    if (r.tempFile().length() != r.offset())\n        throw new IOException(\"temp file changed after persist; use commitAfterRecovery()\");\n}","typeGuard":null,"tryCatchPattern":"try {\n    committer.commit();\n} catch (IOException e) {\n    if (e.getMessage().contains(\"trailing junk data\")) {\n        committer.commitAfterRecovery(); // truncates then moves\n    } else throw e;\n}","preventionTips":["Cease all writes immediately after persist().","Use commitAfterRecovery() whenever post-persist writes are possible.","Enforce a single commit per recoverable."],"tags":["filesystem","local-fs","recoverable-writer","commit","race-condition"],"backgroundTag":null,"analyzedSha":"2f3c205e9266cb30240eb7f4fdab15cad629a70f","analyzedAt":"2026-08-14T08:48:24.518Z","schemaVersion":2},"datasetVersion":"2026-08-14T10:17:34.591Z"}