apache/flink · error · IOException

Output directory could not be created.

Error message

Output directory could not be created.

What it means

Thrown by FileOutputFormat on a DISTRIBUTED filesystem when output is parallel (parallelism>1) or OutputDirectoryMode.ALWAYS, and fs.initOutPathDistFS(path, writeMode, true) returns false. The true flag requests directory creation on the distributed FS; failure means the directory could not be created/cleared — a file exists at the path, permissions are missing, or parent does not exist.

Source

Thrown at flink-core/src/main/java/org/apache/flink/api/common/io/FileOutputFormat.java:294

        if (fs.isDistributedFS()) {

            final WriteMode writeMode = getWriteMode();
            final OutputDirectoryMode outDirMode = getOutputDirectoryMode();

            if (parallelism == 1 && outDirMode == OutputDirectoryMode.PARONLY) {
                // output is not written in parallel and should be written to a single file.
                // prepare distributed output path
                if (!fs.initOutPathDistFS(path, writeMode, false)) {
                    // output preparation failed! Cancel task.
                    throw new IOException("Output path could not be initialized.");
                }

            } else {
                // output should be written to a directory

                // only distributed file systems can be initialized at start-up time.
                if (!fs.initOutPathDistFS(path, writeMode, true)) {
                    throw new IOException("Output directory could not be created.");
                }
            }
        }
    }

    @Override
    public void tryCleanupOnError() {
        if (this.fileCreated) {
            this.fileCreated = false;

            try {
                close();
            } catch (IOException e) {
                LOG.error("Could not properly close FileOutputFormat.", e);
            }

            try {
                FileSystem.get(this.actualFilePath.toUri()).delete(actualFilePath, false);

View on GitHub (pinned to 2f3c205e92)

Solutions

1. Clear any file at the target path or use WriteMode.OVERWRITE.
2. Ensure the parent directory exists and is writable.
3. Switch the path to a fresh directory per run if collisions recur.

Example fix

// before: parallel run after single-file run, same path
env.setParallelism(8);
FileOutputFormat out = new FileOutputFormat(new Path("hdfs:///out/result"));

// after
out.setWriteMode(WriteMode.OVERWRITE);

// Pre-check for parallel distributed directory write
Path out = new Path("hdfs:///out/result");
FileSystem fs = out.getFileSystem();
if (fs.exists(out) && !fs.getFileStatus(out).isDir()) {
    fs.delete(out, false); // remove stray file blocking dir creation
}
format.setWriteMode(WriteMode.OVERWRITE);
fs.mkdirs(out.getParent());

When it happens

Trigger: A regular file exists at the target path so directory creation fails; missing write permission on the distributed FS; parent directory missing and not auto-created; quota exceeded.

Common situations: Parallel sink where a prior non-parallel run left a single file at the path; HDFS quota/permissions; S3 object key collision.

Related errors

• Output path could not be initialized.
• Output directory '{}' could not be created. Canceling task..
• Output path '{}' could not be initialized. Canceling task...
• The given offset is not contained in the any block.
• Input opening request timed out. Opener was {} alive. Stack

AI-assisted analysis of apache/flink@2f3c205e92 (2026-08-14). Data as JSON: /api/errors/299c03cccd60bd23. Report an issue: GitHub.