apache/flink · error · IOException

Committing file failed. Target file already exists: {}

Error message

Committing file failed. Target file already exists: {}

What it means

Thrown by LocalRecoverableFsDataOutputStream.commit() when the atomic move raises FileAlreadyExistsException — i.e. the target file already exists at commit time. This is a distinct, more specific failure than a generic rename error.

Source

Thrown at flink-core/src/main/java/org/apache/flink/core/fs/local/LocalRecoverableFsDataOutputStream.java:169

            // sanity check
            if (src.length() != recoverable.offset()) {
                // something was done to this file since the committer was created.
                // this is not the "clean" case
                throw new IOException("Cannot clean commit: File has trailing junk data.");
            }

            // rather than fall into default recovery, handle errors explicitly
            // in order to improve error messages
            try {
                Files.move(src.toPath(), dest.toPath(), StandardCopyOption.ATOMIC_MOVE);
            } catch (UnsupportedOperationException | AtomicMoveNotSupportedException e) {
                if (!src.renameTo(dest)) {
                    throw new IOException(
                            "Committing file failed, could not rename " + src + " -> " + dest);
                }
            } catch (FileAlreadyExistsException e) {
                throw new IOException(
                        "Committing file failed. Target file already exists: " + dest);
            }
        }

        @Override
        public void commitAfterRecovery() throws IOException {
            final File src = recoverable.tempFile();
            final File dest = recoverable.targetFile();
            final long expectedLength = recoverable.offset();

            if (src.exists()) {
                if (src.length() > expectedLength) {
                    // can happen if we co from persist to recovering for commit directly
                    // truncate the trailing junk away
                    try (FileOutputStream fos = new FileOutputStream(src, true)) {
                        fos.getChannel().truncate(expectedLength);
                    }
                } else if (src.length() < expectedLength) {

View on GitHub (pinned to 2f3c205e92)

Solutions

1. Ensure each target is committed exactly once; deduplicate or guard commit calls.
2. Delete or overwrite the existing target before commit if re-commit is intended.
3. In recovery, prefer commitAfterRecovery() which handles existing-target cases.

void ensureTargetAbsent(LocalRecoverable r) throws IOException {
    if (r.targetFile().exists())
        throw new FileAlreadyExistsException("target exists; remove or use unique name: " + r.targetFile());
}

try {
    committer.commit();
} catch (IOException e) {
    if (e.getMessage().contains("Target file already exists")) {
        // deduplicate: treat as already-committed, or pick unique target
    } else throw e;
}

When it happens

Trigger: Calling commit() when the target file already exists on disk, so Files.move(src, dest, ATOMIC_MOVE) refuses to overwrite.

Common situations: Duplicate commit; a previous successful commit left the target; concurrent writers committing to the same target; a prior run's output not cleaned.

Related errors

• Cannot clean commit: File has trailing junk data.
• Committing file failed, could not rename {} -> {}
• File Not Found: {}
• Missing data in tmp file: {}
• Cannot sync state to system like S3. Use persist() to create

AI-assisted analysis of apache/flink@2f3c205e92 (2026-08-14). Data as JSON: /api/errors/6bc602408c098a6f. Report an issue: GitHub.