apache/flink · error · IOException

Could not commit file from %s to %s

Error message

Could not commit file from %s to %s

What it means

Wrapped IOException thrown when HadoopRenameFileCommitter's FileSystem.rename(temp, target) fails during commit or commitAfterRecovery. The rename is the atomic commit step of the hadoop-bulk file sink; any underlying FileSystem error (permissions, missing parent, unsupported overwrite semantics, connection loss) surfaces here with the original cause attached.

Source

Thrown at flink-formats/flink-hadoop-bulk/src/main/java/org/apache/flink/formats/hadoop/bulk/committer/HadoopRenameFileCommitter.java:103

    private void rename(boolean assertFileExists) throws IOException {
        FileSystem fileSystem = FileSystem.get(targetFilePath.toUri(), configuration);

        if (!fileSystem.exists(tempFilePath)) {
            if (assertFileExists) {
                throw new IOException(
                        String.format("In progress file(%s) not exists.", tempFilePath));
            } else {
                // By pass the re-commit if source file not exists.
                // TODO: in the future we may also need to check if the target file exists.
                return;
            }
        }

        try {
            // If file exists, it will be overwritten.
            fileSystem.rename(tempFilePath, targetFilePath);
        } catch (IOException e) {
            throw new IOException(
                    String.format(
                            "Could not commit file from %s to %s", tempFilePath, targetFilePath),
                    e);
        }
    }

    private Path generateTempFilePath() throws IOException {
        checkArgument(targetFilePath.isAbsolute(), "Target file must be absolute");

        FileSystem fileSystem = FileSystem.get(targetFilePath.toUri(), configuration);

        Path parent = targetFilePath.getParent();
        String name = targetFilePath.getName();

        while (true) {
            Path candidate =
                    new Path(parent, "." + name + ".inprogress." + UUID.randomUUID().toString());
            if (!fileSystem.exists(candidate)) {

View on GitHub (pinned to 2f3c205e92)

Solutions

1. Read the attached cause: AccessControlException -> fix dir permissions; FileNotFoundException -> create the target parent directory
2. Verify the target file system supports rename-overwrite if the target may already exist
3. Retry the job: transient HDFS/network failures during rename are often resolved by restart from the last checkpoint
4. Ensure the checkpoint interval aligns with commit so partially-copied targets are retried, not duplicated
5. For object stores, prefer a store-native sink connector (e.g., S3 Sink) instead of rename-based commit

Example fix

# before: bucket output dir not created for dynamic partitions
WITH ('partitioned' ...)

# after: pre-create or auto-create partition dirs
hdfs dfs -mkdir -p /warehouse/db/table/dt=2026-08-14
# and configure the file system with create-directory-on-commit / bucket assignor that mkdirs

FileSystem fs = FileSystem.get(targetPath.toUri(), conf);
Path parent = targetPath.getParent();
if (!fs.exists(parent)) { fs.mkdirs(parent); }
// also verify write permission: fs.create(parent.suffix(".probe")).close(); fs.delete(...)

catch (IOException e) {
    // inspect cause: AccessControlException / FileNotFoundException / RPC timeout
    // transient FS errors -> restart job; commit retries from checkpoint are safe (rename is the atomic step)
}

When it happens

Trigger: FileSystem.rename(tempFilePath, targetFilePath) throws: target parent directory missing, permission denied on target dir, target file exists on a store that does not support rename-overwrite, NameNode/DataNode unavailability, or S3 rename (copy) failing mid-way.

Common situations: HDFS permissions/quotas misconfigured; bucket path not created for new partitions; using an object store where rename is non-atomic and expensive; transient NN/RPC timeouts during checkpoint-aligned commits.

Related errors

• In progress file(%s) not exists.
• Committing file failed, could not rename {} -> {}
• An I/O error occurred while creating temporary file to extra
• Failed to create parent(s) for given base dir: %s
• An IO error occurred while accessing the staging FileSystem.

AI-assisted analysis of apache/flink@2f3c205e92 (2026-08-14). Data as JSON: /api/errors/493284c898ad1994. Report an issue: GitHub.