{"record":{"id":"299c03cccd60bd23","repo":"apache/flink","slug":"output-directory-could-not-be-created","errorCode":null,"errorMessage":"Output directory could not be created.","messagePattern":"Output directory could not be created\\.","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"error","filePath":"flink-core/src/main/java/org/apache/flink/api/common/io/FileOutputFormat.java","lineNumber":294,"sourceCode":"        if (fs.isDistributedFS()) {\n\n            final WriteMode writeMode = getWriteMode();\n            final OutputDirectoryMode outDirMode = getOutputDirectoryMode();\n\n            if (parallelism == 1 && outDirMode == OutputDirectoryMode.PARONLY) {\n                // output is not written in parallel and should be written to a single file.\n                // prepare distributed output path\n                if (!fs.initOutPathDistFS(path, writeMode, false)) {\n                    // output preparation failed! Cancel task.\n                    throw new IOException(\"Output path could not be initialized.\");\n                }\n\n            } else {\n                // output should be written to a directory\n\n                // only distributed file systems can be initialized at start-up time.\n                if (!fs.initOutPathDistFS(path, writeMode, true)) {\n                    throw new IOException(\"Output directory could not be created.\");\n                }\n            }\n        }\n    }\n\n    @Override\n    public void tryCleanupOnError() {\n        if (this.fileCreated) {\n            this.fileCreated = false;\n\n            try {\n                close();\n            } catch (IOException e) {\n                LOG.error(\"Could not properly close FileOutputFormat.\", e);\n            }\n\n            try {\n                FileSystem.get(this.actualFilePath.toUri()).delete(actualFilePath, false);","sourceCodeStart":276,"sourceCodeEnd":312,"githubUrl":"https://github.com/apache/flink/blob/2f3c205e9266cb30240eb7f4fdab15cad629a70f/flink-core/src/main/java/org/apache/flink/api/common/io/FileOutputFormat.java#L276-L312","documentation":"Thrown by FileOutputFormat on a DISTRIBUTED filesystem when output is parallel (parallelism>1) or OutputDirectoryMode.ALWAYS, and fs.initOutPathDistFS(path, writeMode, true) returns false. The true flag requests directory creation on the distributed FS; failure means the directory could not be created/cleared — a file exists at the path, permissions are missing, or parent does not exist.","triggerScenarios":"A regular file exists at the target path so directory creation fails; missing write permission on the distributed FS; parent directory missing and not auto-created; quota exceeded.","commonSituations":"Parallel sink where a prior non-parallel run left a single file at the path; HDFS quota/permissions; S3 object key collision.","solutions":["Clear any file at the target path or use WriteMode.OVERWRITE.","Ensure the parent directory exists and is writable.","Switch the path to a fresh directory per run if collisions recur."],"exampleFix":"// before: parallel run after single-file run, same path\nenv.setParallelism(8);\nFileOutputFormat out = new FileOutputFormat(new Path(\"hdfs:///out/result\"));\n\n// after\nout.setWriteMode(WriteMode.OVERWRITE);","handlingStrategy":"validation","validationCode":"// Pre-check for parallel distributed directory write\nPath out = new Path(\"hdfs:///out/result\");\nFileSystem fs = out.getFileSystem();\nif (fs.exists(out) && !fs.getFileStatus(out).isDir()) {\n    fs.delete(out, false); // remove stray file blocking dir creation\n}\nformat.setWriteMode(WriteMode.OVERWRITE);\nfs.mkdirs(out.getParent());","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Use WriteMode.OVERWRITE for parallel distributed sinks.","Remove any file occupying the directory path.","Verify parent directory creation and quotas on the distributed FS."],"tags":["flink","file-output","distributed-fs","hdfs","directory","write-mode"],"backgroundTag":null,"analyzedSha":"2f3c205e9266cb30240eb7f4fdab15cad629a70f","analyzedAt":"2026-08-14T08:48:24.518Z","schemaVersion":2},"datasetVersion":"2026-08-14T10:17:34.591Z"}