{"record":{"id":"493284c898ad1994","repo":"apache/flink","slug":"could-not-commit-file-from-s-to-s","errorCode":null,"errorMessage":"Could not commit file from %s to %s","messagePattern":"Could not commit file from (.+?) to (.+?)","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"error","filePath":"flink-formats/flink-hadoop-bulk/src/main/java/org/apache/flink/formats/hadoop/bulk/committer/HadoopRenameFileCommitter.java","lineNumber":103,"sourceCode":"    private void rename(boolean assertFileExists) throws IOException {\n        FileSystem fileSystem = FileSystem.get(targetFilePath.toUri(), configuration);\n\n        if (!fileSystem.exists(tempFilePath)) {\n            if (assertFileExists) {\n                throw new IOException(\n                        String.format(\"In progress file(%s) not exists.\", tempFilePath));\n            } else {\n                // By pass the re-commit if source file not exists.\n                // TODO: in the future we may also need to check if the target file exists.\n                return;\n            }\n        }\n\n        try {\n            // If file exists, it will be overwritten.\n            fileSystem.rename(tempFilePath, targetFilePath);\n        } catch (IOException e) {\n            throw new IOException(\n                    String.format(\n                            \"Could not commit file from %s to %s\", tempFilePath, targetFilePath),\n                    e);\n        }\n    }\n\n    private Path generateTempFilePath() throws IOException {\n        checkArgument(targetFilePath.isAbsolute(), \"Target file must be absolute\");\n\n        FileSystem fileSystem = FileSystem.get(targetFilePath.toUri(), configuration);\n\n        Path parent = targetFilePath.getParent();\n        String name = targetFilePath.getName();\n\n        while (true) {\n            Path candidate =\n                    new Path(parent, \".\" + name + \".inprogress.\" + UUID.randomUUID().toString());\n            if (!fileSystem.exists(candidate)) {","sourceCodeStart":85,"sourceCodeEnd":121,"githubUrl":"https://github.com/apache/flink/blob/2f3c205e9266cb30240eb7f4fdab15cad629a70f/flink-formats/flink-hadoop-bulk/src/main/java/org/apache/flink/formats/hadoop/bulk/committer/HadoopRenameFileCommitter.java#L85-L121","documentation":"Wrapped IOException thrown when HadoopRenameFileCommitter's FileSystem.rename(temp, target) fails during commit or commitAfterRecovery. The rename is the atomic commit step of the hadoop-bulk file sink; any underlying FileSystem error (permissions, missing parent, unsupported overwrite semantics, connection loss) surfaces here with the original cause attached.","triggerScenarios":"FileSystem.rename(tempFilePath, targetFilePath) throws: target parent directory missing, permission denied on target dir, target file exists on a store that does not support rename-overwrite, NameNode/DataNode unavailability, or S3 rename (copy) failing mid-way.","commonSituations":"HDFS permissions/quotas misconfigured; bucket path not created for new partitions; using an object store where rename is non-atomic and expensive; transient NN/RPC timeouts during checkpoint-aligned commits.","solutions":["Read the attached cause: AccessControlException -> fix dir permissions; FileNotFoundException -> create the target parent directory","Verify the target file system supports rename-overwrite if the target may already exist","Retry the job: transient HDFS/network failures during rename are often resolved by restart from the last checkpoint","Ensure the checkpoint interval aligns with commit so partially-copied targets are retried, not duplicated","For object stores, prefer a store-native sink connector (e.g., S3 Sink) instead of rename-based commit"],"exampleFix":"# before: bucket output dir not created for dynamic partitions\nWITH ('partitioned' ...)\n\n# after: pre-create or auto-create partition dirs\nhdfs dfs -mkdir -p /warehouse/db/table/dt=2026-08-14\n# and configure the file system with create-directory-on-commit / bucket assignor that mkdirs","handlingStrategy":"retry","validationCode":"FileSystem fs = FileSystem.get(targetPath.toUri(), conf);\nPath parent = targetPath.getParent();\nif (!fs.exists(parent)) { fs.mkdirs(parent); }\n// also verify write permission: fs.create(parent.suffix(\".probe\")).close(); fs.delete(...)","typeGuard":null,"tryCatchPattern":"catch (IOException e) {\n    // inspect cause: AccessControlException / FileNotFoundException / RPC timeout\n    // transient FS errors -> restart job; commit retries from checkpoint are safe (rename is the atomic step)\n}","preventionTips":["Pre-create sink directories and verify permissions in deployment checks","Monitor NameNode health; rename failures spike during NN instability","Prefer store-native connectors (S3 Sink) over rename-based commit on object stores"],"tags":["flink","hadoop","filesystem","rename","commit","io"],"backgroundTag":null,"analyzedSha":"2f3c205e9266cb30240eb7f4fdab15cad629a70f","analyzedAt":"2026-08-14T08:48:24.518Z","schemaVersion":2},"datasetVersion":"2026-08-14T10:17:34.591Z"}