What causes "Cannot clean commit: File has trailing junk data." in apache/flink?

Ensure only one job instance processes a given in-progress file at a time (proper job restart, not parallel duplicates)

apache/flink · error · IOException

Cannot clean commit: File has trailing junk data.

Error message

Cannot clean commit: File has trailing junk data.

What it means

During a clean commit, HadoopFsCommitter compares the staging file's current length against recoverable.offset() (the byte position recorded when persist() snapshot the state). A mismatch means bytes were appended to (or truncated from) the temp file after the recoverable was created, so the 'clean' rename-only commit is unsafe and is refused.

Source

Thrown at flink-filesystems/flink-hadoop-fs/src/main/java/org/apache/flink/runtime/fs/hdfs/HadoopRecoverableFsDataOutputStream.java:242

        }

        @Override
        public void commit() throws IOException {
            final Path src = recoverable.tempFile();
            final Path dest = recoverable.targetFile();
            final long expectedLength = recoverable.offset();

            final FileStatus srcStatus;
            try {
                srcStatus = fs.getFileStatus(src);
            } catch (IOException e) {
                throw new IOException("Cannot clean commit: Staging file does not exist.");
            }

            if (srcStatus.getLen() != expectedLength) {
                // something was done to this file since the committer was created.
                // this is not the "clean" case
                throw new IOException("Cannot clean commit: File has trailing junk data.");
            }

            try {
                fs.rename(src, dest);
            } catch (IOException e) {
                throw new IOException(
                        "Committing file by rename failed: " + src + " to " + dest, e);
            }
        }

        @Override
        public void commitAfterRecovery() throws IOException {
            final Path src = recoverable.tempFile();
            final Path dest = recoverable.targetFile();
            final long expectedLength = recoverable.offset();

            FileStatus srcStatus = null;
            try {

View on GitHub (pinned to 2f3c205e92)

Solutions

Ensure only one job instance processes a given in-progress file at a time (proper job restart, not parallel duplicates)
Resume from the latest checkpoint/savepoint so the recoverable offset matches the file tail, or let commitAfterRecovery handle the truncation path instead of clean commit
If unavoidable, discard the mismatched in-progress file and restart the part

Defensive patterns

Strategy: validation

Validate before calling

// Pre-check the clean-commit invariant: file length must equal recoverable offset
FileStatus st = fs.getFileStatus(recoverable.tempFile());
if (st.getLen() != recoverable.offset()) {
    // route to the recovery-style commit (truncate then rename) instead of clean commit
}

Try / catch

catch (IOException e) {
    if (e.getMessage() != null && e.getMessage().contains("trailing junk data")) {
        // fall back to truncate-then-rename semantics or discard the part
    } else { throw e; }
}

Prevention

One active writer per output path/UID at a time
Prefer rolling policies that roll on checkpoint (offset always matches tail)
Avoid replaying old committer state alongside a running job

When it happens

Trigger: persist() is called for a checkpoint, writing continues into the same temp file (buffered data flushed later), and then commit() is invoked with the older recoverable; a commit retried concurrently with an active writer; duplicate consumption of the same checkpointed committer state.

Common situations: StreamingFileSink recovery where the operator resumes writing and then commits an older pending checkpoint; at-least-once replay causing the same part file to be appended by two attempts; taking a savepoint while a checkpoint commit is still pending.

Related errors

AI-assisted analysis of apache/flink@2f3c205e92 (2026-08-14). Data as JSON: /api/errors/62b0426fb19b5e0f. Report an issue: GitHub.