What causes "Failed to create the parent directory: {}" in apache/flink?

Verify the output path's parent directory exists and is writable: ensure the base directory is created before the job starts or pre-create it in your deployment.

apache/flink · error · IOException

Failed to create the parent directory: {}

Error message

Failed to create the parent directory: {}

What it means

Thrown by LocalRecoverableWriter.open when the parent directory of the staging temp file cannot be created. The writer calls parent.mkdirs() and, if that returns false AND the parent still does not exist, throws this IOException. This protects against writing to a path whose parent is missing and cannot be auto-created (e.g. permission denied, read-only filesystem, or an intermediate path component is a file).

Source

Thrown at flink-core/src/main/java/org/apache/flink/core/fs/local/LocalRecoverableWriter.java:54

/** A {@link RecoverableWriter} for the {@link LocalFileSystem}. */
@Internal
public class LocalRecoverableWriter implements RecoverableWriter {

    private final LocalFileSystem fs;

    public LocalRecoverableWriter(LocalFileSystem fs) {
        this.fs = checkNotNull(fs);
    }

    @Override
    public RecoverableFsDataOutputStream open(Path filePath) throws IOException {
        final File targetFile = fs.pathToFile(filePath);
        final File tempFile = generateStagingTempFilePath(targetFile);

        // try to create the parent
        final File parent = tempFile.getParentFile();
        if (parent != null && !parent.mkdirs() && !parent.exists()) {
            throw new IOException("Failed to create the parent directory: " + parent);
        }

        return new LocalRecoverableFsDataOutputStream(targetFile, tempFile);
    }

    @Override
    public RecoverableFsDataOutputStream recover(ResumeRecoverable recoverable) throws IOException {
        if (recoverable instanceof LocalRecoverable) {
            return new LocalRecoverableFsDataOutputStream((LocalRecoverable) recoverable);
        } else {
            throw new IllegalArgumentException(
                    "LocalFileSystem cannot recover recoverable for other file system: "
                            + recoverable);
        }
    }

    @Override
    public boolean requiresCleanupOfRecoverableState() {

View on GitHub (pinned to 2f3c205e92)

Solutions

Verify the output path's parent directory exists and is writable: ensure the base directory is created before the job starts or pre-create it in your deployment.
Check filesystem permissions for the user running the TaskManager process.
If running in a container, confirm the volume is mounted read-write and the mount point exists.
Avoid paths that collide with existing files for any intermediate component.

Example fix

// before
Path out = new Path("/data/sink/output.parquet");
writer.open(out);

// after — pre-create parent
new File(out.getPath()).getParentFile().mkdirs();
writer.open(out);

Defensive patterns

Strategy: validation

Validate before calling

File parent = new File(outputPath.getPath()).getParentFile();
if (parent != null && !parent.exists() && !parent.mkdirs()) {
    throw new IOException("Cannot create output parent directory: " + parent);
}

Try / catch

try {
    writer.open(filePath);
} catch (IOException e) {
    if (e.getMessage().contains("parent directory")) {
        // check permissions, mount type, path validity
    }
    throw e;
}

Prevention

Pre-create the output base directory in your deployment scripts.
Verify the TaskManager process user has write permission on the output path.
Confirm container volume mounts are read-write before starting the job.

When it happens

Trigger: Opening a LocalRecoverableWriter for an output Path whose parent directory does not exist and cannot be created; the parent path is occupied by a regular file; insufficient filesystem permissions to create directories; the filesystem is mounted read-only.

Common situations: Configuring a file sink with an output path whose parent directory was never created; running in a container/sandbox where the output mount is read-only; path typos pointing to a non-existent base directory; concurrent cleanup that removed the parent between checks.

Related errors

AI-assisted analysis of apache/flink@2f3c205e92 (2026-08-14). Data as JSON: /api/errors/1efe049a9c207b06. Report an issue: GitHub.