apache/flink · error · IOException

In progress file(%s) not exists.

Error message

In progress file(%s) not exists.

What it means

Thrown by HadoopRenameFileCommitter.commit() when the pending (temp) file no longer exists on the target FileSystem at commit time. The committer writes to a hidden temp path next to the target and renames it on commit; if the temp file vanished before the rename, the pre-commit existence assertion fails. This typically indicates the commit is being retried after the file was already moved, deleted by an external process, or lost due to eventual-consistency effects on object stores.

Source

Thrown at flink-formats/flink-hadoop-bulk/src/main/java/org/apache/flink/formats/hadoop/bulk/committer/HadoopRenameFileCommitter.java:90

        // Do nothing.
    }

    @Override
    public void commit() throws IOException {
        rename(true);
    }

    @Override
    public void commitAfterRecovery() throws IOException {
        rename(false);
    }

    private void rename(boolean assertFileExists) throws IOException {
        FileSystem fileSystem = FileSystem.get(targetFilePath.toUri(), configuration);

        if (!fileSystem.exists(tempFilePath)) {
            if (assertFileExists) {
                throw new IOException(
                        String.format("In progress file(%s) not exists.", tempFilePath));
            } else {
                // By pass the re-commit if source file not exists.
                // TODO: in the future we may also need to check if the target file exists.
                return;
            }
        }

        try {
            // If file exists, it will be overwritten.
            fileSystem.rename(tempFilePath, targetFilePath);
        } catch (IOException e) {
            throw new IOException(
                    String.format(
                            "Could not commit file from %s to %s", tempFilePath, targetFilePath),
                    e);
        }
    }

View on GitHub (pinned to 2f3c205e92)

Solutions

1. Verify only one sink/subtask owns the target path and the part-file path is unique (partIndex / bucket assigner)
2. Check no external cleaner deletes hidden in-progress files (temp name starts with '.' next to the target)
3. On failure-recovery paths make sure the framework calls commitAfterRecovery() (which bypasses a missing source file) rather than a fresh commit()
4. Inspect FileSystem logs around the failure to see who renamed or deleted the temp file
5. For eventually-consistent object stores, verify exists() is not reading a stale listing before concluding the file is lost

Example fix

// before: assuming the pending file always survives until commit
committer.commit();

// after: tolerate already-committed state on retry
if (fs.exists(committer.getTempFilePath())) {
    committer.commit();
} else {
    // file was already moved or deleted; skip or use commitAfterRecovery()
    committer.commitAfterRecovery();
}

FileSystem fs = FileSystem.get(targetPath.toUri(), conf);
if (!fs.exists(tempPath)) {
    // already committed or cleaned; skip fresh commit, use recovery semantics
}

try {
    committer.commit();
} catch (IOException e) {
    if (e.getMessage() != null && e.getMessage().contains("not exists")) {
        // treat as already-committed; log and continue idempotently
    } else {
        throw e;
    }
}

When it happens

Trigger: Calling StreamingFileSink/FileSink commit (rename(true)) when tempFilePath does not exist: FileSink pending-file cleanup ran early, another committer instance already renamed the file, an external job/ cleanser deleted in-progress dot-files, or S3-style stores where exists() is stale.

Common situations: Recovery after a task failure where commit() (not commitAfterRecovery()) is replayed; two sink subtasks writing the same target path; lifecycle policies on HDFS/S3 deleting hidden in-progress files; misconfigured part-file prefix/suffix colliding with cleanup rules.

Related errors

• Could not commit file from %s to %s
• Could not create writer state serializer.
• Could not create committable serializer.
• An IO error occurred while accessing the staging FileSystem.
• Error opening the Input Split {} [{},{}]: {}

AI-assisted analysis of apache/flink@2f3c205e92 (2026-08-14). Data as JSON: /api/errors/998a9677b0e8a95e. Report an issue: GitHub.