apache/flink · error · IOException

Output directory '{}' could not be created. Canceling task..

Error message

Output directory '{}' could not be created. Canceling task...

What it means

Thrown by FileOutputFormat.open on a LOCAL filesystem when output is parallel (numTasks>1) or OutputDirectoryMode.ALWAYS, and fs.initOutPathLocalFS(path, writeMode, true) returns false. The true flag requests directory creation; failure means the output directory could not be created or cleared (existing non-directory target, permissions, parent missing).

Source

Thrown at flink-core/src/main/java/org/apache/flink/api/common/io/FileOutputFormat.java:231

            if (numTasks == 1 && outputDirectoryMode == OutputDirectoryMode.PARONLY) {
                // output should go to a single file

                // prepare local output path. checks for write mode and removes existing files in
                // case of OVERWRITE mode
                if (!fs.initOutPathLocalFS(p, writeMode, false)) {
                    // output preparation failed! Cancel task.
                    throw new IOException(
                            "Output path '"
                                    + p.toString()
                                    + "' could not be initialized. Canceling task...");
                }
            } else {
                // numTasks > 1 || outDirMode == OutputDirectoryMode.ALWAYS

                if (!fs.initOutPathLocalFS(p, writeMode, true)) {
                    // output preparation failed! Cancel task.
                    throw new IOException(
                            "Output directory '"
                                    + p.toString()
                                    + "' could not be created. Canceling task...");
                }
            }
        }

        // Suffix the path with the parallel instance index, if needed
        this.actualFilePath =
                (numTasks > 1 || outputDirectoryMode == OutputDirectoryMode.ALWAYS)
                        ? p.suffix("/" + getDirectoryFileName(taskNumber))
                        : p;

        // create output file
        this.stream = fs.create(this.actualFilePath, writeMode);

        // at this point, the file creation must have succeeded, or an exception has been thrown
        this.fileCreated = true;

View on GitHub (pinned to 2f3c205e92)

Solutions

1. Remove any existing file at the target path, or use WriteMode.OVERWRITE.
2. Ensure the parent directory exists and is writable, or let the FS auto-create it with proper config.
3. If a single file already exists there, switch OutputDirectoryMode or clear it first.

Example fix

// before: parallel sink, prior file at path
FileOutputFormat out = new FileOutputFormat(new Path("file:///tmp/out"));
env.setParallelism(4);

// after
out.setWriteMode(WriteMode.OVERWRITE);

// Ensure the target is clear for a parallel local directory write
Path dir = new Path("file:///tmp/out");
FileSystem fs = dir.getFileSystem();
if (fs.exists(dir) && !fs.getFileStatus(dir).isDir()) {
    fs.delete(dir, false); // remove stray file
}
format.setWriteMode(WriteMode.OVERWRITE);

When it happens

Trigger: A file (not a directory) exists at the target path so directory creation fails; parent directories are missing and not auto-created; local permissions deny directory creation; NO_OVERWRITE against an existing directory with conflicting contents.

Common situations: Parallel local sink where a previous run left a single file at the directory path; pointing the parallel sink at a path occupied by a regular file; restricted local filesystem permissions in a container.

Related errors

• Output path '{}' could not be initialized. Canceling task...
• Output directory could not be created.
• Output path could not be initialized.
• Output file path may not be null.
• TaskNumber: {}, numTasks: {}

AI-assisted analysis of apache/flink@2f3c205e92 (2026-08-14). Data as JSON: /api/errors/a65cd5048f119b5d. Report an issue: GitHub.