apache/dolphinscheduler · error · IOException

Failed to create parent directory for destination file

Error message

Failed to create parent directory for destination file

What it means

HdfsStorageOperator.download copies a file from HDFS to a local destination. Before copying it ensures the destination's parent directory exists; if the parent is missing and mkdirs() fails it throws IOException, aborting the download.

Source

Thrown at dolphinscheduler-storage-plugin/dolphinscheduler-storage-hdfs/src/main/java/org/apache/dolphinscheduler/plugin/storage/hdfs/HdfsStorageOperator.java:188

    @SneakyThrows
    @Override
    public void download(String srcHdfsFilePath, String dstFile, boolean overwrite) {
        Path srcPath = new Path(srcHdfsFilePath);
        File dstPath = new File(dstFile);

        if (dstPath.exists()) {
            if (dstPath.isFile()) {
                if (overwrite) {
                    Files.delete(dstPath.toPath());
                }
            } else {
                log.error("destination file must be a file");
            }
        }

        if (!dstPath.getParentFile().exists() && !dstPath.getParentFile().mkdirs()) {
            throw new IOException("Failed to create parent directory for destination file");
        }

        FileUtil.copy(fs, srcPath, dstPath, false, fs.getConf());
    }

    @SneakyThrows
    @Override
    public void copy(String srcPath, String dstPath, boolean deleteSource, boolean overwrite) {
        FileUtil.copy(fs, new Path(srcPath), fs, new Path(dstPath), deleteSource, overwrite, fs.getConf());
    }

    @SneakyThrows
    @Override
    public void upload(String srcAbsoluteFilePath,
                       String dstAbsoluteFilePath,
                       boolean deleteSource,
                       boolean overwrite) {
        Path srcPath = new Path(srcAbsoluteFilePath);

View on GitHub (pinned to 02eac45a1b)

Solutions

1. Pre-create the parent directory manually (mkdir -p) with correct permissions and retry
2. Check that no regular file exists at a path component of the destination parent
3. Verify write permission and disk space on the destination filesystem

Example fix

// before
hdfs.download(hdfsPath, "/data/locked/out/result.csv");
// after
Files.createDirectories(Paths.get("/data/out"));
hdfs.download(hdfsPath, "/data/out/result.csv");

File parent = new File(dstLocalPath).getParentFile();
if (!parent.exists() && !parent.mkdirs()) throw new IOException("Cannot create " + parent);
if (parent.exists() && !parent.canWrite()) throw new IOException("No write permission: " + parent);

try {
    hdfs.download(src, dst);
} catch (IOException e) {
    if (e.getMessage().contains("Failed to create parent directory")) {
        Files.createDirectories(Paths.get(dst).getParent());
        hdfs.download(src, dst); // retry once
    } else throw e;
}

When it happens

Trigger: Calling download(srcHdfsPath, dstLocalPath) where dstLocalPath's parent directory does not exist and File.mkdirs() fails — typically due to filesystem permissions, a read-only filesystem, or a parent path component being an existing regular file.

Common situations: Downloading into a temp dir whose permissions changed; destination parent path conflicting with an existing file; disk mounted read-only or full.

Understand the failure class

Background: mkdir permission denied (EACCES): failed to create directory errors explained — this error's family across 32 libraries.

Related errors

• Create xlsx directory error
• Close HadoopUtils instance failed
• generate excel error
• Update the resource file from content: {fileAbsolutePath} fa
• Download the resource file: {fileAbsolutePath} failed

AI-assisted analysis of apache/dolphinscheduler@02eac45a1b (2026-09-06). Data as JSON: /api/errors/9a2678c91318b7a4. Report an issue: GitHub.